How to use Caliper benchmark beta snapshot without maven?

Question

I have been asked to use Google's Caliper project to create a few microbenchmarks. I would very much like to use the annotation features of the newest beta snapshot, but aside from a few small examples I am having trouble finding good documentation on how to actually run the thing... There is a video tutorial up which instructs users on the new maven integration feature, which I was also asked NOT to use.

Right now I just have a small example stripped from one of theirs, modified with some other information I gleaned from another SO question:

public class Benchmarks {

    public class Test {
        @Param int size; // set automatically by framework

        private int[] array; // set by us, in setUp()

        @BeforeExperiment void setUp() {
          // @Param values are guaranteed to have been injected by now
          array = new int[size];
        }

        @Benchmark int timeArrayIteration(int reps) {
            int dummy = 0;
            for (int i = 0; i < reps; i++) {
                for (int doNotIgnoreMe : array) {
                    dummy += doNotIgnoreMe;
                }
            }
          return dummy;
        }

    }

    //(Questionable practice here?)
    public static void main (String args[]) {
        CaliperMain.main(Test.class, args); 
    }

}

Running it gives me the message that I did not set a default value for size. I am having trouble tracking down where I should be putting it.

Removing "size" entirely by commenting out the @Param line and giving a hard value to the array declaration in setUp just leads to it deciding that there are "No Experiments to be done," which makes sense, I suppose.

If there are any up-to-date resources or tutorials that could point out what I am doing wrong (probably a whole lot, honestly) I would be very appreciative.

EDIT:

I have updated to this as per some advice:

public class Benchmarks {
      @Param({"1", "10", "1000"}) int size; // set automatically by framework

  private int[] array; // set by us, in setUp()

  @BeforeExperiment void setUp() {
    // @Param values are guaranteed to have been injected by now
    array = new int[size];
  }

  @Benchmark int timeArrayIteration(int reps) {
    int dummy = 0;
    for (int i = 0; i < reps; i++) {
      for (int doNotIgnoreMe : array) {
        dummy += doNotIgnoreMe;
      }
    }
    return dummy;
  }
}

I am running through the beta snapshot and passing in the Benchmarks class as an argument. I receive the following:

Experiment selection: 
  Instruments:   []
  User parameters:   {size=[1, 10, 1000]}
  Virtual machines:  [default]
  Selection type:    Full cartesian product

There were no experiments to be performed for the class Benchmarks using the instruments [allocation, runtime]

It doesn't seem to be detecting any Instruments. I am not passing any in, as it's mentioned in the documentation that it simply uses default allocation, runtime (which is fine for my purposes).

DOUBLE EDIT: Found that problem, stupid mistake. Will do a quick write-up once I confirm it.

Answer 1

Running it gives me the message that I did not set a default value for size.

Parameters are set either from default values:

@Param({"1", "10", "1000"}) int size;

Or by passing values via the the -D flag. E.g.: -Dsize=1,10,1000. Enums and booleans get special treatment in that it uses all possible values without having to list them in the annotation.

---

Removing "size" entirely by commenting out the @Param line and giving a hard value to the array declaration in setUp just leads to it deciding that there are "No Experiments to be done," which makes sense, I suppose.

The issue is likely that that your benchmark is an inner class and needs a reference to the enclosing class (though this should have been an error). Either make your benchmark class a top-level class (recommended) or make it static.

---

Also, there is no particular need to include the main method. Invoking CaliperMain with your benchmark class as the first parameter is equivalent.

Answer 2

Running it gives me the message that I did not set a default value for size.

That's very simple:

@Param({"1", "10", "1000"}) int size;

---

Removing "size" entirely by commenting out the @Param line and giving a hard value to the array declaration in setUp just leads to it deciding that there are "No Experiments to be done," which makes sense, I suppose.

No, it doesn't. Without any params, each benchmark method is to be run exactly once. See the other answer for the solution.

---

There's a quite some Javadoc, e.g., on @Param. Actually, not so much has changed. Annotations have replaced conventions (now you don't need the time prefix), params stayed the same, setup uses an annotation instead of inheritance.