From the basics of the C language we know that in the following code:
y = fn3(fn2(fn1(x)));
...the fn1() is executed first, the fn2() is executed second and the fn3() is executed last.
What order of matrix transformations is built by the following C code ?:
ctm = fz_pre_translate(fz_pre_rotate(fz_scale(sx, sy), r), tx, ty);
Case A or Case B ?:
The documentation of the muPDF Library API is available at this link and it states the following on Page 61:
Alternatively, operations can be specifically applied to existing matrices. Because of the non-commutative nature of matrix operations, it matters whether the new operation is applied before or after the existing matrix. For example, if you have a matrix that performs a rotation, and you wish to combine that with a translation, you must decide whether you want the translation to occur before the rotation (‘pre’) or afterwards (‘post’).
MuPDF has various API functions for such operations:
To me the statement above suggests that the order of transformations, being built by these functions, is not the same as the order of nested function evaluations in C (and their invocations) ...but I just can't be sure.

In mathematical terms, case A, which is a translation, followed by a rotation, followed by scaling could be expressed as
x' = S · (R · (T · x)) = S · R · T · x = ((S · R) · T ) · x
So we want to apply the translation before the other transformations combined, while the scaling should only be applied after.
M = S · R · T = MS, R · T = S · MR, T
I'd say that a richer API, like the one exposed by this library, let the user choose between different ways of expressing their intent, but it doesn't (of course) violate the rules of C nested functions calls.
There could be cases in which particular algorithms may benefit from one approach or the other.
Note that, despite the order in which the nested functions are called, this line can be read exactly as the first statement of this answer (first translation, then rotation, then scale), while the mathematical notation is basically backwards.
Whether this generates confusion or not in the reader, I'm afraid is a matter of opinion and personal backgrounds.
To my knowledge, though, having a small as possible public API is considered less error prone and easier to mantain.