why is there difference in statistics shown by unix and Java RunTime

Question

I am having some memory issues with my application and need help understanding these statistics.

Unix 'top' shows these stats for my process-

VSZ: 37.4g
RSS: 20.0g 

So, this means 20g is currently swapped in for the process and in use.

However, when I print stats from within my application using Runtime class, I get this:

Runtime.totalMemory() : 9.8G
Runtime.freeMemory()  : 3.6G
Runtime.maxMemory()  : 14.3G

Why doesn't [Runtime.totalMemory() - Runtime.freeMemory()] match RSS? This is the memory currently in use by the process. There is a huge difference between the two numbers.

Also, does the runtime give back the unused memory (Runtime.freeMemory()) back to OS for use by other processes?

Note that my applications are running in a peer to peer GemFire caching system set up with shared and replicated caches. I need to optimize the application to reduce memory footprint.

Answer 1

Runtime.totalMemory shows currently available memory. Java allocates memory lazily.

Also, does the runtime give back the unused memory (Runtime.freeMemory()) back to OS for use by other processes?

No. If Java allocated memory (totalMemory) it is in java process now.

RSS: 20.0g 14.3G

As already mentioned Java uses memory besides heap. Also Gemfire uses off heap memory (check this). Try to look at them in VisualVM-Buffer Monitor.

What is your infrastracture (OS, VMs)?

If you can't use standard tools, you probably should write own serviceability agent using JMX (for example)

UPDATE

As per doc, by default Gemfire uses JVM Heap.

Ok

Why is RSS consistently showing 20G for a process that is using only ~10G of heap memory.

If you still ask, I would provide more details. What are java memory usage?

• Heap memory (Xmx);
• Stack memory (each thread has own stack ThreadStackSize, VMThreadStackSize);
• MaxPermSize or MaxMetaspaceSize (MaxPermGenSize, MaxMetaspaceSize);
• Direct byte buffers (MaxDirectMemorySize);
• Memory Pools (Par Eden Space, Par Survivor Space, CMS Old Gen, Metaspace/PermGen, Code Cache, Compressed Class Space (CompressedClassSpaceSize));
• String table (where all String.intern() will be, StringTableSize);
• PerfDataMemorySize;
• MarkStackSize;
• CompilerThreadStackSize;
• Constant Pool;
• I likely missed something;

To see default values, you can run:

 java -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

To see which options for your particular process, you can run:

 jps -lvm

Is there an command line tool in unix to determine the rest usage?

For some of them yes. For some of them no.

For direct memory you could try sun.misc.SharedSecrets.getJavaNioAccess().getDirectBufferPool().getMemoryUsed().

With cmd, for example:

 jcmd <pid> VM.native_memory baseline

But it depends on running settings (NativeMemoryTracking). You could read how to enable Native Memory tracking here.

In linux you also could use:

 pmap -x <pid>

In conclusion this is probably no matter, because your task:

Note that my applications are running in a peer to peer GemFire caching system set up with shared and replicated caches. I need to optimize the application to reduce memory footprint.

And you can not impact on native memory usage. I suppose look at jmap util, which can show you class histogram. You should check what has big size in GemFire and review those objects, probably you store in cache data, which are not should be there. I mean in my practice for optimizing cache I review object and fields and see what fields are really frequent, what are not. Another approach is check your serialization mechanism and object layouts.

As I mentioned you can use serviceability agents:

import com.sun.tools.attach.VirtualMachine;
import sun.tools.attach.HotSpotVirtualMachine;

import java.io.InputStream;

public class JMemoryMain {

    public static void main(String[] args) throws Exception {
        final int pid = JMemoryMainUtils.getPid(args);
        final HotSpotVirtualMachine vm = (HotSpotVirtualMachine) VirtualMachine.attach(String.valueOf(pid));
        final byte[] buffer = new byte[1024];
        try (InputStream is = vm.heapHisto()) {
            for (int read; (read = is.read(buffer)) > 0;) {
                System.out.write(buffer, 0, read);
            }
        }
        vm.detach();
    }

}

You need tools.jar from JDK in dependencies to run this. It could print you class histogram too, but sometimes works, when jmap does not. Also, when you should wait a lot of time, while histogram is calculated, you could use VM.getSystemDictionary() and find only your classes. And, it would be usefull, if you can not enable NMT, because of overhead.