jmh benchmark methods not run while setup method is

Question

I am trying to run jmh benchmark in a grade project. ./gradlew :sub-project:jmh

I can see the setup method got executed twice from stdout. but the two @Benchmark methods don't seem to be executed. the jmh report file (.../build/reports/jmh/results/index.html) is also empty.

here is the simplified code snippet.

@Fork(1)
@State(Scope.Benchmark)
@Warmup(iterations = 3)
@Measurement(iterations = 5)
@BenchmarkMode(Mode.SingleShotTime)
public class MyBenchmark {

  @Setup
  public void setupBenchmark() {
    System.out.println(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>");
    // ...
  }

  @TearDown
  public void tearDownBenchmark() throws IOException {}

  @Benchmark
  @Threads(1)
  public void testFoo(Blackhole blackhole) {
    System.out.println("foooooooooooooooooooooooo");
    blackhole.consume(1);
  }

  @Benchmark
  @Threads(1)
  public void testRealBenchmark(Blackhole blackhole) {
    System.out.println("==============================");
    // ...
    blackhole.consume(results);
  }
}

expect the two @Benchmark methods executed with stdout and jmh report.

Gradle setup should be fine, as other jmh runs for for other sub projects. Also the @Setup method was executed based on stdout.

Answer 1

it turns out to be a bug. setup method failed silently and exception wasn't shown in stdout from grade run. that is why the benchmark method wasn't executed.