How to increase heap size in Corb2 on MarkLogic

Question

I am trying to run a deletion framework that I created which has a very huge amount of data. While trying to run Corb2, I get the below warning and sometimes an error because of insufficient memory as well.

WARNING: Slow receive! Consider increasing max heap size and using -XX:+UseConcMarkSweepGC

Error:

Sep 23, 2021 1:06:23 PM com.marklogic.developer.corb.QueryUrisLoader closeRequestAndSession
INFO: closing uris session
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
        at java.lang.StringBuilder.append(StringBuilder.java:190)
        at com.marklogic.io.IOHelper.literalStringFromReader(IOHelper.java:50)
        at com.marklogic.io.IOHelper.literalStringFromStream(IOHelper.java:66)
        at com.marklogic.http.MultipartBuffer.getBodyAsString(MultipartBuffer.java:75)
        at com.marklogic.xcc.impl.AbstractResultSequence.instantiateResultItem(AbstractResultSequence.java:138)
        at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:147)
        at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:166)
        at com.marklogic.developer.corb.QueryUrisLoader.populateQueue(QueryUrisLoader.java:212)
        at com.marklogic.developer.corb.QueryUrisLoader.createAndPopulateQueue(QueryUrisLoader.java:201)
        at com.marklogic.developer.corb.QueryUrisLoader.open(QueryUrisLoader.java:122)
        at com.marklogic.developer.corb.Manager.runUrisLoader(Manager.java:809)
        at com.marklogic.developer.corb.Manager.populateQueue(Manager.java:830)
        at com.marklogic.developer.corb.Manager.run(Manager.java:578)
        at com.marklogic.developer.corb.Manager.main(Manager.java:138)

Has anybody used it (especially with JavaScript) to be able to work on larger volumes of data?

Answer 1

Rather than trying to find the right size of memory to allocate to hold all of the URIs in-memory, enable the DISK-QUEUE option to allow for the URIs queue to spill to disk in a temporary file that is consumed, instead of all being held in-memory.

You can optionally configure the path for the temp files to be written with DISK-QUEUE-TEMP-DIR. If not specified, it will use the TEMP-DIR, or java.io.tmpdir.