Importing 100000+ records from a JSON - slow hit to AT_ENTITY

Question

In the JSON file, there are about 100000 records. I am attempting to write all of them into mantle.product.Product entity.

The procedure starts and at about 35000 records, it starts to deteriorate with warning 'Slow hit to AT_ENTITY:create:mantle.product.Product'. Then it halts definitely with 'java.lang.OutOfMemoryError: GC overhead limit exceeded' error. This behavior is on my PC.

Any hints welcome.

This is the code:

void processJson2(String filePath) {
    //def json = new JsonSlurper().parseText(new BufferedReader(new InputStreamReader(this.getFileIO().openStream(), "UTF-8")))

    //will initialize class manually
    def docReadReference = this.executionContext.resource.getLocationReference(filePath)

    if (docReadReference.isFile()) {
        //inputstream
        InputStream inputFile = docReadReference.openStream()
        TransactionFacade trxFacade = this.executionContext.getTransaction()

        this.executionContext.artifactExecution.disableTarpit()
        this.executionContext.artifactExecution.disableEntityEca()
        this.executionContext.artifactExecution.disableAuthz()

        trxFacade.runRequireNew(50000, "Error loading entity JSON data", {

            try {
                logMachine.info("Opening file ${docReadReference.isFile()}")

                JsonSlurper slurper = new JsonSlurper().setType(JsonParserType.CHARACTER_SOURCE)
                def json = slurper.parse(new BufferedReader(new InputStreamReader(inputFile, "UTF-8")))

                //writer
                Long counter = 1

                json.each {
                    this.executionContext.service.sync().name("create", "mantle.product.Product").parameters([productId: it.sourceFileReference]).call()

                    //display thousands
                    if (counter % 1000 == 0) {
                        logMachine.info("JSON rows processed ${counter} > ${it.sourceFileReference}")
                    }

                    //move counter
                    counter += 1
                }

                //log
                logMachine.info("File processed.")

            } catch (Throwable t) {
                trxFacade.rollback("Error while processing JSON", t);

                //log as warning
                logMachine.warn("Incorrectly handled JSON parsing ${t.message}.")
            } finally {
                if (trxFacade.isTransactionInPlace()) trxFacade.commit();

                inputFile.close()

                this.executionContext.artifactExecution.enableTarpit()
                this.executionContext.artifactExecution.enableEntityEca()
                this.executionContext.artifactExecution.enableAuthz()
            }
        })
    }
}

Answer 1

This seems to work fine, so if anyone has similar troubles, it may help.

1. since, I was using MoquiDevConf, first thing to solve the slow-hit problem is to remove the for type AT_ENTITY,
2. next thing, the BufferedReader is not the most effective solution to read the data, I used the InputStream to initialize the json ArrayList.

This is the result:

InputStream inputFile = docReadReference.openStream()
        TransactionFacade trxFacade = this.executionContext.getTransaction()


        JsonSlurper slurper = new JsonSlurper().setType(JsonParserType.INDEX_OVERLAY)
        //ArrayList<Object> json = slurper.parse(new BufferedReader(new InputStreamReader(inputFile, "UTF-8")))
        ArrayList<Object> json = slurper.parse(inputFile)