dlsym causing segfault when dynamically loading shm_open

Question

I am trying to figure out to load the function shm_open dynamically at runtime and have hit a wall.

My current code is as such:

CMakeLists.txt

project(dlsysm_stack_overflow_example)
cmake_minimum_required(VERSION 3.16)

add_executable(${PROJECT_NAME}
    main.cpp
    custom_mman.cpp    
)

target_link_libraries(${PROJECT_NAME}
    dl
)

custom_mman.cpp

#include <iostream>

#define _GNU_SOURCE 
#include <dlfcn.h>

extern "C"
{

int shm_open(const char* name, int oflag, mode_t mode)
{
    using ShmOpenFuncPtr_t = int (*)(const char*, int, mode_t);
    static ShmOpenFuncPtr_t real_shm_open = nullptr;

    std::cout << "custom shm_open\n";

    if (nullptr == real_shm_open)
    {
        real_shm_open = reinterpret_cast<ShmOpenFuncPtr_t>(dlsym(RTLD_NEXT, "shm_open"));
        char* error = dlerror();
        if (error != nullptr) 
        {
            std::cout << "could not load shm_open at runtime! error: " << error << "\n";
            return -1;
        }
    }

    return real_shm_open(name, oflag, mode);
}

}

main.cpp

#include <sys/mman.h>
#include <fcntl.h>
#include <iostream>

int main(int argc, char** argv)
{
    int return_value = shm_open("/shm_name", O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
    std::cout << "return_value: " << return_value << "\n";
    return 0;
}

At the moment, because I give the executable the custom_mann.cpp source file, the linker finds my definition of shm_open before the standard library definition and calls that instead. My intention is then to do some logic before calling the real shm_open function.

However, the call to dlsym is returning a null pointer so when real_shm_open is called, a segfault occurs. Strangely, dlerror does not set the error code so the function doesn't return early as I would expect if dlsym is not returning the proper pointer.

Any ideas why this could be happening is much appreciated?

Answer 1

shm_open is supplied by librt.so. Per the Linux shm_open() man page (bolding mine):

NAME

shm_open, shm_unlink - create/open or unlink POSIX shared memory objects

SYNOPSIS

   #include <sys/mman.h>
   #include <sys/stat.h>        /* For mode constants */
   #include <fcntl.h>           /* For O_* constants */

   int shm_open(const char *name, int oflag, mode_t mode);
   int shm_unlink(const char *name);

Link with -lrt.

You need to add rt to your target_link_libraries.

You should be checking the value returned from dlsym() directly for errors. Per the dlsym man page:

RETURN VALUE

On success, these functions return the address associated with symbol. On failure, they return NULL; the cause of the error can be diagnosed using dlerror(3).

Additionally, your code is not thread-safe:

static ShmOpenFuncPtr_t real_shm_open = nullptr;

std::cout << "custom shm_open\n";

if (nullptr == real_shm_open)
{
    real_shm_open = reinterpret_cast<ShmOpenFuncPtr_t>(dlsym(RTLD_NEXT, "shm_open"));
    char* error = dlerror();
    if (error != nullptr) 
    {
        std::cout << "could not load shm_open at runtime! error: " << error << "\n";
        return -1;
    }
}

Two or more threads can be executing the code in parallel, resulting in a corrupted real_shm_open that is neither NULL nor correct.