Linux: how can I force my process to free unloaded libraries?

Question

I develop both the process and the '.so' dynamic library.
I have the following scenario:

1. Using dlopen() my process is loading my '.so' library,
2. I load symbols and use the library,
3. After some time I stop using the the library and unload it using dlclose(),
4. Now I edit my library-code, fixing bugs, and then rebuild it,
5. At later point, while process is still active, I want to reload the new version of my library.
   I call dlopen() once more, hoping the new image will be loaded.

The problem is that even though last dlclose() did succeed (returned 0), and even though now I successfully reload my library and symbols, I can see that the library was not really reloaded, and the old code is running.

I understand that dlclose() keeps the library in the process memory for some time and does not necessarily unloads it immediately.

Can I force my library to be unloaded, or be replaced with the new dlopen()?