Java - Strange Heap Space Error keeps getting thrown

Question

people. This is a very common error, but no matter what I do, I just can't seem to find a way around it. I use IDEA IntelliJ, and for quite a while now, I've been trying to implement a working Semantic Role Labeling (SRL) System, and finally, I decided to use the PathLSTM model (https://github.com/microth/PathLSTM), to do the same.

Problem is, this algorithm depends on model files which are as big as 2.7G, for it's parsing operations.(https://drive.google.com/uc?id=0B5aLxfs6OvZBYUk2b0hLZjNqY3c&export=download)

This throws me an insufficient heap space - Out of memory error

---

Loading pipeline from 
C:\Users\Vyso\Downloads\NLP\SRL\SEMAFOR\absSemafor\LTH\wttv\PathLSTM-pre-
illinois-built\srl-ACL2016-eng.model
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newInstance(Array.java:75)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1883)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1919)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1919)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)

Process finished with exit code 1

---

So naturally, I changed the heap space of both the JVM on my system, and my IDE, where I could change the vmoptions, as follows. (It was -Xms128m and -Xmx512m, by default).

---

# custom IntelliJ IDEA VM options

-Xms2048m
-Xmx4000m
-XX:ReservedCodeCacheSize=240m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow

---

But even after assigning around 4G as Max Heap space, I get the error. Funny thing is, in the memory management toolbar of my IDE, I can see that the code uses a max of just 500m, during runtime, so I really don't know how this heap space error is still getting thrown.

Maybe it's just a beginner level mistake from my side, but I've been trying to get out out of this problem for quite a few days now, and I'd really appreciate it if someone could instruct me on how I could get rid of this error.

Thank You.

Answer 1

There are two different JVM instances at play here:

• The JVM which is running IDEA
• The JVM which is running PathLSTM

I suspect the issue here is with the JVM which is running PathLSTM. To grant this non default values for -Xms, -Xmx you need to go to Run > Edit Configurations ... and then choose the run configuration for your PathLSTM calls and then add -XmsNNNm -XmxYYYm to the VM options input.

You can use JVisualVM (you'll find this in the bin directory of your JDK) to see how much memory is being used when running PathLSTM. Start IDEA then start JVisualVM and you'll see something (likely named "Idea") under the Local node in the Applications tree view then start whatever runs PathLSTM and you'll see another entry appear under the Local node, click on that and then choose the Monitor tab. By default, the top right hand graph displayed there shows memory usage of the selected application. You'll likely see heap used grow until that process gets an OOM, then change the -Xmx in Run > Edit Configurations ... and retry until you find the required value. Or alternatively, choose a very large value and then let JVisualVM show you the actual maximum value used and then edit your -Xmx to match that maximum.

The key points here are:

• Any changes you make to heap size configuration must be made to the JVM instance which is actually running PathLSTM, you configure that JVM through Run > Edit Configurations ...
• You can use JVisualVM to monitor the running of that JVM
• For > 2GB of heap you'll need to be running a 64bit JVM