Printing additional output from jmh benchmarks

Question

I currently have a JMH benchmark to measure the performance of various data structures that all implement the same interface. The benchmark works fine, but I would like to print some additional information for each benchmark which describes the state of my data structure before and after the trials.

Currently, I'm doing something like the below code

@Param({"impl0", "impl1", "impl2"})
String dataStructureImplementation;

DataStructureInterface dataStructure;

List<String> inputData;

@Setup(Level.Trial)
public void setUp() {
    switch (dataStructureImplementation) {
        case "impl0":
            dataStructure = new DataStructureImplementation0();
            break;
        case "impl1":
            dataStructure = new DataStructureImplementation1();
            break;
        case "impl2":
            dataStructure = new DataStructureImplementation2();
            break;
        default:
            throw new IllegalArgumentException();
    }
    setUpDataStructure();
    inputData = readInputDataFromFile();
    System.out.println(dataStructure.getStateRepresentation());
}

@TearDown(Level.Trial)
public void tearDown() {
    System.out.println(dataStructure.getStateRepresentation());
}

@Benchmark
@OperationsPerInvocation(1000000)
public void queryDataStructure(Blackhole bh) {
    for (String datum : inputData)
        bh.consume(dataStructure.query(datum));
}

The problem is the output from System.out.println gets garbled with the output from JMH. Is there a way to avoid that and have my print statement run after JMH completes its report for the trial? The alternative is that write this to a file and consolidate afterwards, but would be nice to get a single report that I don't need to manually consolidate.

Answer 1

There is no out-of-the-box way to accumulate some custom-formatted output from @Setup and @Teardown methods and append it to the final report.

One way you could approach it, is to append your desired output in a file, and print its content after all your benchmark tests have been completed.

You need to use Runner class to run your tests though.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
public class BenchmarkTestWithMoreOutput {

    public static void main(String[] args) throws RunnerException {

        // This is just a very basic setup just to get an idea
        Options opt = new OptionsBuilder()
                .include(BenchmarkTestWithMoreOutput.class.getSimpleName())
                .forks(2)
                .build();

        Collection<RunResult> results = new Runner(opt).run();

        // Now print your file content. 
        // It will be printed after the default JMH report
   }

    @Benchmark
    public int benchmarkMethod() {
        // code to benchmark
    }
}

RunResult object contains information from an iteration. You can mix it up with your custom output and print some nice report.