Terracotta EhCache - concurrent modification creates huge byte array during serialization

Question

If my heapdump Dominator Tree view looks like below, can I assume the major contributor (~1GB) to my heap is the Thread instance created by Weblogic? And in that Thread a reference to ByteArrayOutputStream is the reason for 1GB?

Follow up question: Can I also assume it is due to object serialization? Maybe triggered by Terracotta ehcahce (from the third line. Not because it is next to the ByeArrayOS, because in our code that is the only place serialization can happen)?

Answer 1

Ehcache does indeed rely on Java Serialization the moment your cache goes beyond heap.

However, it will only ever serialize what you put in the cache. So is it possible that some cached mappings have such huge value or even key?

Answer 2

The Dominator tree is saying that you have a Weblogic thread holding a ByteArrayOutputStream (and a SerializerObjectOutputStream). The Weblogic thread is a classical worker thread currently processing a request. And it is currently stuck on something.

So, this is the equivalent of

ByteArrayOutputStream in = new ByteArrayOutputStream();
Thread.wait();

The thread is holding a ByteArrayOutputStream and it can't be garbage collected since the thread isn't done with it.

Seeing the serializer makes me think that you are currently deserializing from Ehcache disk or offheap.

Is it possible that you are putting pretty huge objects in your cache? As @louis-jacomet mentioned noticed.