prevent function interposing in the library that is linked to the library which does the interposing

Question

Well I couldn't find very nice short phrase for my question title, but here is what I mean:

I have a library that interposes some syscalls like open(2). I use this method for interposing. I add it to DYLD_INSERT_LIBRARIES in order to achieve my goal.

The library that contains the interposing code is a.dylib.

I need to link a library b.dylib to a.dylib because it contains some functions that a.dylib needs.

The issue is a.dylib interposes functions in b.dylib too which I don't want.

I am not an expert with linking. Is there any way to prevent this behaviour and what's the reason behind this happening?

UPDATE:

This is how b.dylib is built:

clang -dynamiclib -fPIC -Wextra -Wall -pedantic -Wl,-single_module <object files> -o b.dylib -install_name <path>

This is how a.dylib is built:

clang -dynamiclib -fPIC -Wextra -Wall -pedantic -Wl,-single_module <object files> -o a.dylib -install_name <path> b.dylib

Only a.dylib is added to the end of DYLD_INSERT_LIBRARIES.

Answer 1

You say a.dylib depends on b.dylib but b.dylib does not depend on a.dylib and that you want no calls in b.dylib to ever be interposed. You gave open as an example of an interposed function.

The example I gave was that a.dylib needs to call processFile that is defined in b.dylib and it opens files. You don't want the open call in b.dylib to be interposed ever (whether a.dylib is loaded or not).

The only way I know to pull this off is for processFile to dynamically load the C library instead of letting ld do it automatically.

There are two problems with this approach. The first is that each function that needs to call open will need one additional if statement to execute the code to resolve the symbol.

The other is that you need to know ahead of time the filename of the C library to load. And if there's a problem resolving the symbol, then you have to deal with a situation that most programs don't have to deal with; at least at a later time.

The code will look something like this:

void processFile (char * filename) {
    FILE *fp;
    // c_open can have file scope if thread safety isn't needed
    static int (*c_open)() = NULL;

    if (c_open == NULL) {
        // factor this if more than one function needs open
        void *dlh = dlopen(C_LIBRARY_FILENAME, RTLD_LAZY);
        if (dlh == NULL) {
           perror("cannot load C library");
           exit(1);
        }
        c_open = dlsym(dlh, "open");
        if (c_open == NULL) {
           perror("cannot resolve open in the C library");
           exit(1);
        }
    }
    c_open(filename, O_RDONLY);
    ...
}

---

UPDATE

If you do not specify the full pathname to the C library, dlopen could open an unexpected file.