Using Randoop To Generate Test Cases (Based on Pre- and Post- Conditions)

Question

I am trying to use Randoop (by following Randoop Manual) to generate test cases based on pre- and post- conditions specification stored in a JSON file.

Target program is the following (buggy) Java method.

package com.example.math;

public class Math {
    /*Expected Behavior:
          Given upperBound >= 0, the method returns
               1 + 2 + ... + upperBound                 
      But This method is buggy and works only on
      inputs with odd value, e.g. for upperBound == 4,
      the method returns 1 + 2 + 3 + 4 + 1 instead of
      1 + 2 + 3 + 4                                   */
    public static int sum(int upperBound) {
        int s = 0;
        for (int i = 0; i <= upperBound; i++) {
            s += i;
        }
        if (upperBound % 2 == 0) {// <--------- BUG!
            s++;                  // <--------- BUG!
        }                         // <--------- BUG!
        return s;
    }
}

And I use the following JSON file to specify the desired behavior of the method:

[
  {
    "operation": {
      "classname": "com.example.math.Math",
      "name": "sum",
      "parameterTypes": [ "int" ]
    },
    "identifiers": {
      "parameters": [ "upperBound" ],
      "returnName": "res"
    },
    "post": [
      {
        "property": {
          "condition": "res == upperBound * (upperBound + 1) / 2",
          "description": ""
        },
        "description": "",
        "guard": {
          "condition": "true",
          "description": ""
        }
      }
    ],
    "pre": [
      {
        "description": "upperBound must be non-negative",
        "guard": {
          "condition": "upperBound >= 0",
          "description": "upperBound must be non-negative"
        }
      }
    ]
  }
]

I compile the program, and run the following command to apply Randoop so as to generate test cases based on the correctness specification:

java -cp my-classpath:$RANDOOP_JAR randoop.main.Main gentests --testclass=com.example.math.Math --output-limit=200 --specifications=spec.json

Where spec.json is the JSON file containing the above specification for method contracts. I have got two questions:

1. Why does not changing --output-limit change the number of generated test cases? For sufficiently large numbers, it seems that I always get only 8 regression test cases two of which checking the method getClass does not return null value (even though that is not part of my specification). Please kindly let me know how I can generate more regression test cases. Am I missing a command-line option?
2. It seems that Randoop does not consult the specification inside spec.json when it tries to generate error-revealing test cases. Can we make Randoop generate error-revealing test cases on every input that violates the provided post-condition?

Thank you.

Answer 1

1. Why does not changing --output-limit change the number of generated test cases?

Randoop generates tests, then outputs a subset of them. For example, Randoop does not output subsumed tests, which appear as a subsequence of some longer test.

This is mentioned obliquely in the documentation for --output-limit.

two of which checking the method getClass does not return null value (even though that is not part of my specification)

getClass() is a method in Math (the class under test), so Randoop calls getClass(). At test generation time, the return value was not null, so Randoop made an assertion to that effect.

There is nothing special about getClass(); Randoop would create similar regression tests for other methods.

2. It seems that Randoop does not consult the specification inside spec.json

There was a bug in Randoop's handling of postcondition specifications for static methods. The bug has been fixed.

To report a bug, it is better to use Randoop's issue tracker, as noted in the Randoop manual. The options for getting help also include mailing lists. Unlike Stack Overflow, the issue tracker and mailing list permit discussions and tracking current status. Thanks!