Java memory leak in non-heap space

Question

I have a Java application (running under OpenJDK 1.8.0_181) that has memory issues. Specifically after some time all physical memory appears consumed and the app fails to allocate more memory. The difficult part is that it's not the heap memory that causes problems. I'm struggling to understand how to investigate it further as it's not the heap memory.

Here are types of memory crashes I got:

Exception in thread "eXistThread-22" java.lang.OutOfMemoryError: unable to create new native thread

OR

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000773200000, 142606336, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 142606336 bytes for committing reserved memory.

In all such cases the physical memory of the server is completely consumed. Then the app is unable to allocate memory or extend the heap so it crashes.

To start with here are the results of the top command:

KiB Mem :  7867456 total,   454608 free,  6696500 used,   716348 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   933392 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   18 root      20   0 8827048 6.180g  28028 S   0.0 82.4  24:41.10 java

The Java app is running with the following memory settings: java -Xms512m -Xmx3072m

So the app can use max 3GB of the heap space. However you can see that the java process (RES memory) already uses over 6GB RAM.

At this point it's clear I need to look at non-heap space. There is so-called Metaspace which I thought faces memory leaks. I've done the following checks:

# jcmd 18 GC.heap_info
18:
 PSYoungGen      total 984576K, used 647154K [0x0000000780000000, 0x00000007c0000000, 0x00000007c0000000)
  eden space 925696K, 63% used [0x0000000780000000,0x00000007a3e83970,0x00000007b8800000)
  from space 58880K, 99% used [0x00000007b8800000,0x00000007bc179030,0x00000007bc180000)
  to   space 64000K, 0% used [0x00000007bc180000,0x00000007bc180000,0x00000007c0000000)
 ParOldGen       total 1671168K, used 1048981K [0x0000000700000000, 0x0000000766000000, 0x0000000780000000)
  object space 1671168K, 62% used [0x0000000700000000,0x0000000740065488,0x0000000766000000)
 Metaspace       used 2235431K, capacity 2246406K, committed 2252780K, reserved 2981888K
  class space    used 317765K, capacity 320038K, committed 321024K, reserved 1048576K

Here I see that the used memory is 647154K + 1048981K of heap and 2235431K of metaspace (non-heap). The total is 3931566K = 3840M = 3.75 GB. But what is the rest of memory consumed by the Java process then? It's a lot: 6.18 - 3.75 = 2.43 GB of RAM.

I've also checked more stats with jstat:

# jstat -gccapacity 18
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
174592.0 1048576.0 1048576.0 64000.0 58880.0 925696.0   349696.0  2097152.0  1671168.0  1671168.0      0.0 2981888.0 2252780.0      0.0 1048576.0 321024.0    289    13

Here I calculate the entire Java memory using this formula:

NGCMN + S0C + S1C + EC + OGCMN + MC + CCSC = 174592 + 64000 + 58880 + 925696 + 349696 + 2252780 + 321024 = 4146668K = 3.95 GB

It's slightly more than 3.75 GB calculated from GC.heap_info but still much less that 6.18 GB consumed by the Java process.

So I'm a bit lost and I need an advice:

• How can I identify which type of Java memory is taking 2.43 GB?
• How can I limit that memory or make sure it's cleared timely to avoid OutOfMemory crashes?
• Should I set extra parameters like -XX:MaxMetaspaceSize -XX:MaxMetaspaceFreeRatio? If so, what will be the reasonable values for those?