memory management problems with openj9 and k8s

Question

we are running a k8s-cluster with some java applications. As container-runtime we use cr-io and as jvm adoptopenjdk-16 openj9 (should have good container support).

Recently we have an increased workload on our cluster and we noticed that the containers do not release unused heap memory.

I've tried some gc options to achieve that, but without success:

-Xtune:virtualized

-XX:+IdleTuningGcOnIdle -XX:IdleTuningMinIdleWaitTime=30 

-Xgc:concurrentScavenge

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/repository

What am I missing?

Answer 1

Take a look at OpenJ9 user documentation > Memory management > Heap sizing > Expansion and contraction

This explains that the OpenJ9 JVM sizes the heap by keeping the ratio of the allocated space free space (after a GC) in a range determined by the -xminf and -xmaxf options. The JVM will expand "eagerly" if the heap is too small, but it is "reluctant" about contracting the heap.

So your observed behavior (the heap not shrinking) could just be normal behavior given the default settings for -xminf and -xmaxf. (You can find out what they are using the -XX+PrintFlagsFinal option.)

Note that HotSpot JVMs have roughly similar mechanisms for resizing the heap, though the command line options are different.