LIBUMEM says NO memory leaks, but PRSTAT on Solaris shows leaking?

Question

I have an application I have been trying to get "memory leak free", I have been through solid testing on Linux using Totalview's MemoryScape and no leaks found. I have ported the application to Solaris (SPARC) and there is a leak I am trying to find...

I have used "LIBUMEM" on Solaris and it seems to me like it als picks up NO leaks...

Here is my startup command:

LD_PRELOAD=libumem.so UMEM_DEBUG=audit ./link_outbound config.ini

Then I immediatly checked the PRSTAT on Solaris to see what the startup memory usage was:

PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  9471 root       44M   25M sleep   59    0   0:00:00 1.1% link_outbou/3

Then I started to send thousands of messages to the application...and over time the PRSTAT grew..

PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  9471 root       48M   29M sleep   59    0   0:00:36 3.5% link_outbou/3

And just before I eventually stopped it:

PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  9471 root       48M   48M sleep   59    0   0:01:05 5.3% link_outbou/3

Now the interesting part is when I use LIBUMEM on this application that it showing 48 MB memory, like follows:

pgrep link
9471
# gcore 9471
gcore: core.9471 dumped
# mdb core.9471
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> ::findleaks
BYTES             LEAKED         VMEM_SEG CALLER
131072                 7 ffffffff79f00000 MMAP
57344                  1 ffffffff7d672000 MMAP
24576                  1 ffffffff7acf0000 MMAP
458752                 1 ffffffff7ac80000 MMAP
24576                  1 ffffffff7a320000 MMAP
131072                 1 ffffffff7a300000 MMAP
24576                  1 ffffffff79f20000 MMAP
------------------------------------------------------------------------
           Total       7 oversized leaks, 851968 bytes

CACHE             LEAKED           BUFCTL CALLER
----------------------------------------------------------------------
           Total       0 buffers, 0 bytes
> 

The "7 oversized leaks, 851968 bytes" never changes if I send 10 messages through the application or 10000 messages...it is always "7 oversized leaks, 851968 bytes". Does that mean that the application is not leaking according to "libumem"?

What is so frustrating is that on Linux the memory stays constant, never changes....yet on Solaris I see this slow, but steady growth.

Any idea what this means? Am I using libumem correctly? What could be causing the PRSTAT to be showing memory growth here?

Any help on this would be greatly appreciated....thanks a million.

Answer 1

If the SIZE column doesn't grow, you're not leaking.

RSS (resident set size) is how much of that memory you are actively using, it's normal that that value changes over time. If you were leaking, SIZE would grow over time (and RSS could stay constant, or even shrink).

Answer 2

1. check out this page. the preferred option is UMEM_DEBUG=default, UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1. that is the options that I use for debugging solaris memory leak problem, and it works fine for me.
2. based on my experience with RedHat REL version 5 and solaris SunOS 5.9/5.10, linux process memory footprint doesn't increase gradually, instead it seems it grabs a large chunk memory when it needs extra memory and use them for a long run. (purely based on observation, haven't done any research on its memory allocation mechanism). so you should send a lot more data (10K messages are not big).
3. you can try dtrace tool to check memory problem at solaris.

Jack