Firstly my setup details.
I have a remote linux based server where I am running my containerised docker application which has java on jdk 11 and have photon os as the base image(https://vmware.github.io/photon/assets/files/html/3.0/Introduction.html).
Queries
- Now while performing a memory footprint scan using docker stats command I see the following values
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
5f683918eab7 some-random-agent 0.31% 474.2MiB / 476.8MiB 99.44% 71.3MB / 2.51MB 45.4MB / 197kB 58
Here we can see a memory consumption of 99.4% aprox. So my first question is what does this 99.4% memory signifies, from my understanding this all memory has been consumed by all the tenants which are running in our container, this is a sum up of the memory consumption of all the applications running in container. Am I correct ?
To gain further insights, I decided to take a heap dump of my Java application. However, the results are a bit perplexing
So from the above screenshots I see the memory footprints are around 70Mbs I am not able to understand 474.2Mb - 70Mb (I understand this 474Mb is complete container but even inside the container java application is the top memory consuming process with 470Mb memory). Who is taking this much of space. In the above screenshots too the major contributors are system class loaders, I searched for my components in dominator tree they are even less than 10Mb in all. So is this a expected behaviour in java.
One of my guess is that JDK somehow kicks in some process in the container as soon as the java application triggers which ends up taking this much of space. Is there someway do analysis of that part. Note that I already tried looking into system proc files there also I see some random addresses with some huge memory consumptions but I wasn't able to understand what they are and where do they belong as they weren't annotated properly. There sum up was also almost 474Mbs
On the other hand with the same base image I had few of my other golang build application files which I ran(consider it to be same business logic as the java ap) and they were falling in range of 20-30mb(docker stats memory footrprint). So I am kind of very confused on these things that who is victim here. All the stars align up to say java is victim here but, what part is victim in java that I am not able to understand.
So any help on the above would be much appreciable