Why and how to find where JVM off-heap memory leaks occur

Question

I have a Java project which shows a memory leak in the product environment. We can see its memory expand and grow bigger and bigger using 'top' or 'cat /proc/PID/status|grep VmRSS', but its JVM heap remains in a good state, which means the heap memory is in control by '-Xmx' and other JVM options. I had used jmap/dump/jstack and any other command I know to analyze what's wrong, but in vain.

Finally, we found the problem by testing line by line. It's a problem in the JVM JavaScript engine. I extracted the code here which can reproduce the problem:

package com.unionpay.cqp.arch.js;


import javax.script.*;


import static java.lang.Thread.sleep;


public class JsEngineMain {
    private static final ScriptEngine SCRIPT_ENGINE = new ScriptEngineManager().getEngineByName("JavaScript");
    private static final String SCRIPT_FUNC_1 = "function max_num(a, b){return (a>b)?a:b;} ";


    public static void main(String[] args) {
        if (args == null || args.length < 1) {
            System.out.println("wrong args, use like:");
            System.out.println("java -jar xxx.jar SLEEP_TIME_MILLIES");
            System.out.println("java -cp xxx.jar com.unionpay.cqp.arch.js.JsEngineMainBak SLEEP_TIME_MILLIES");
            return;
        }
        long sleepTime = Long.parseLong(args[0]);
        try {
            while (true) {
                sleep(sleepTime);
                testLoopFuncString();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


    private static void testLoopFuncString() {
        try {
            Compilable compilable = (Compilable) SCRIPT_ENGINE;
            CompiledScript compiledScript = compilable.compile(SCRIPT_FUNC_1);
            compiledScript.eval();
            Invocable invocable = (Invocable) SCRIPT_ENGINE;
            Object res = invocable.invokeFunction("max_num", 6, 1);
            showRes(res);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


    private static void showRes(Object res) {
        if (System.currentTimeMillis() % 20000 == 1) {
            System.out.println(res);
        }
    }
}

You can run it with the command:

java -Xmx 128M -XX:MaxMetaspaceSize=64M -XX:MaxDirectMemorySize=64M -cp js-engine-test.jar com.unionpay.cqp.arch.js.JsEngineMain 2

and monitor the memory usage with:

while true; do cat /proc/PID/status|grep VmRSS;sleep 5;done

Then you will see the memory usage grow over time.

Well, my question is, how to find the problem if we don't know where the problem is? And how do we find the incorrect code by memory analysis or command tools?

I use pmap and jcmd to see where the memory is occupied, I can get some info from jcmd tools and I know it's a off-heap problem. but I can not connect it with codes, so I can not find which line of my code is incorrect.