JMH tests with a javaagent

Question

I'm trying to measure the impact of a JVM agent on performance, to make sure it won't invalidate the tests we're trying to run (and maybe make a case for taking some samples from prod). This case is a set of BTrace scripts which will run during automated load testing, but the issue might be generic to any agent.

To run the benchmarks, I've set up a small JMH project and am attaching the agent as:

java -javaagent:/home/ssube/btrace/build/btrace-agent.jar=scriptdir=/home/ssube/btrace/scripts/,port=0 -jar benchmarks.jar

Doing so causes the following error to appear each time the JVM is forked by JMH:

# Run progress: 0.00% complete, ETA 00:02:00
# Fork: 1 of 1
Exception in thread "main" java.lang.IllegalArgumentException: org.openjdk.jmh.runner.options.CommandLineOptions; local class incompatible: stream classdesc serialVersionUID = 8906142321598115825, local class serialVersionUID = 7529911323947566771
    at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:72)
<binary link had failed, forked VM corrupted the stream? Use EXTRA verbose to print exception>
<forked VM failed with exit code 1>
<stdout last='10 lines'>
</stdout>
<stderr last='10 lines'>
Exception in thread "main" java.lang.IllegalArgumentException: org.openjdk.jmh.runner.options.CommandLineOptions; local class incompatible: stream classdesc serialVersionUID = 8906142321598115825, local class serialVersionUID = 7529911323947566771
    at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:72)
</stderr>

# VM invoker: /usr/java/jdk1.8.0_11/jre/bin/java
# VM options: -javaagent:/home/ssube/btrace/build/btrace-agent.jar=scriptdir=/home/ssube/btrace/scripts/,port=0
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.stackoverflow.questions.ShaderBench.testProcessProc

None of my classes are serializable or have a serialVersionUID. The JMH benchmark works without the BTrace agent attached, and the agent and scripts work on the code without JMH.

(How) can you attach a javaagent to a set of JMH benchmarks and capture performance differences caused by the agent?

Answer 1

JMH maintainer here. From maintainer's perspective, it feels like the JMH fault for not supplying SUIDs for its own internal classes, so it can be resistant to benign instrumentation. This is fixed now with 3d44d68e45be:

changeset:   960:3d44d68e45be
tag:         tip
user:        shade
date:        Sat Aug 16 15:00:18 2014 +0400
summary:     Apply SUIDs for all Serializable classes: this protects from the benign instrumentation.

Please try with the bleeding edge JMH, and see if it solves the issue for you. Note that providing SUIDs does not protect us from blatant changes in the serialization form the Java agent can make (e.g. injecting the non-transient field), so if BTrace is (mis)behaving like that, we are in trouble.

If the JMH change above does not work, please send some minimal reproducible scenario to jmh-dev mailing list, we would see what can be done to mitigate the issue.

Otherwise, look for excluding org.openjdk.jmh.* classes from instrumentation.

Answer 2

I was running into the same issue recently. I assume that it is related to the JMH's forking. As you can read from your benchmark's output, JMH creates a fork 1 of 1 where JMH starts a fresh JVM instance for each test row. These JVM's do not longer have your agent attached.

This affects the classes you redefine using your agent which are loaded without the agent on the forked JVMs. Therefore, you get the serialization issues. As you do not define an explicit UID for your classes, these UIDs are calculated implicitly. The forked JVMs will therefore recognize that the original JVM's classes differ to the forked JVM's classes what results in your error. You could avoid these serialization issues by explicitly defining a UID but your classes would still not be instrumented.

Instead, try to annotate the benchmarks with @Fork(0) what disables the forking altogether. You should however be careful if this distorts your results too much. This is the case when your code can be optimized heavily by profiling. This is especially true if you share code between your benchmarks where the profile from the first benchmarks will influence your other benchmarks, usually to the worse.

Another solution is to also apply the agent to the forked JVM. For this purpose, the @Fork annotation offers several parameters. For your example, you could define:

@Fork(jvmArgs = "-javaagent:/home/ssube/btrace/build/btrace-agent.jar=scriptdir=/home/ssube/btrace/scripts/,port=0")

Note however that this makes your build dependent on your file system.