Is Cucumber-jvm thread safe?

Question

I want to run the same Cucumber tests in multiple threads. More specifically, I have a set of features, and running these features in one thread works fine. I use the JSON formatter to record running time of each step. Now I want to do load test. I care more about the running time of each feature/step in a multi-thread environment. So I create multiple threads, and each thread runs on the same feature set. Each thread has its own JSON report. Is this possible in theory?

For some project setup reason I cannot use the JUnit runner. So I have to resort to the CLI-way:

        long threadId = Thread.currentThread().getId();
        String jsonFilename = String.format("json:run/cucumber%d.json", threadId);

            String argv[] = new String[]{
                "--glue",
                "com.some.package",
                "--format",
                jsonFilename,
                "d:\\features"};

        // Do not call Main.run() directly. It has a System.exit() call at the end.             
        // Main.run(argv, Thread.currentThread().getContextClassLoader());

        // Copied the same code from Main.run(). 
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        RuntimeOptions runtimeOptions = new RuntimeOptions(new Env("cucumber-jvm"), argv);
        ResourceLoader resourceLoader = new MultiLoader(classLoader);
        ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
        Runtime runtime = new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
        runtime.writeStepdefsJson();
        runtime.run();      

I tried to create a seperate thread for each Cucumber run. The problem is, only one of the thread has a valid JSON report. All the other threads just create empty JSON files. Is this by design in Cucumber or is there something I missed?

Answer 1

Well if you can find a way for cucumber to output scenario location ( I.e. feature_file_path:line_nunber_in_feature_file) for all the scenarios you want run based on given tag, then you can use gpars and gradle to run scenarios in parallel. Step 1: In the first gradle task, we’ll use above solution to generate a text file (say scenarios.txt) containing locations for all the scenarios that we want to execute Step 2: Next, extract contents of scenarios.txt generated in step 1 into a groovy list say scenariosList Step 3: create one more task (javaExec), here we’ll use gpars withPool in combination with scenariosList.eachParallel, and use cucumber main class and other cucumberOptions to run these scenarios in parallel. PS: here we will provide a scenario location as the value of the option “features” so that cucumber will run only this scenario. Also no need to provide any tag name as we already have a list of scenarios that we need to execute.

Note: You need to use a machine with high configuration like a Linux sever because a new jvm instance is created per scenario, and probably use a cloud service like Saucelabs to execute scenarios. This way you don’t have to worry about the infrastructure.

Step4: This is the last step. Every scenario bran in step 3 will generate an json output file. You have to collate the output based on the feature names so as to generate one json file per feature file.

This solution sounds a bit complex, but with right efforts can yield significant results.

Answer 2

Supposedly you can run your Cucumber-JVM tests in parallel by using this Maven POM configuration from here: https://opencredo.com/running-cucumber-jvm-tests-in-parallel/

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.14</version>
    <executions>
        <execution>
            <id>acceptance-test</id>
            <phase>integration-test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <forkCount>${surefire.fork.count}</forkCount>
                <refuseForks>false</reuseForks>
                <argLine>-Duser.language=en</argLine>
                <argLine>-Xmx1024m</argLine>
                <argLine>-XX:MaxPermSize=256m</argLine>
                <argLine>-Dfile.encoding=UTF-8</argLine>
                <useFile>false</useFile>
                <includes>
                    <include>**/*AT.class</include>
                </includes>
                <testFailureIgnore>true</testFailureIgnore>
            </configuration>
        </execution>
    </executions>
</plugin>

In the above snippet, you can see that the maven-surefire-plugin is used to run our acceptance tests – any classes that end in *AT will be run as a JUnit test class. Thanks to JUnit, making the tests run in parallel is now a simple case of setting the forkCount configuration option. In the example project, this is set to 5, meaning that we can run up to 5 threads (ie, 5 runner classes) at a time.

Answer 3

Not currently -- here is the issue you observe. I haven't found any way to parallelize by scenario.

Here's a nice write up on poor-man's concurrency. Just run multiple commands each selecting a different subset of your tests -- by feature or tag. I would fork a new JVM (as a JUnit driver would) rather than trying to thread it since cucumber was not designed for that. You have to balance them yourself, then figure out how to combine the reports. (But at least the problem is combining reports not corrupt reports.)

Answer 4

We have looked into multi-threading cucumber tests under Gradle and Groovy using the excellent GPars library. We have 650 UI tests and counting.

We didn't encounter any obvious problems running cucumber-JVM in multiple threads but the multi-threading also didn't improve performance as much as we hoped.

We ran each feature file in a separate thread. There are a few details to take care of, like splicing together the cucumber reports from different threads and making sure our step code was thread-safe. We sometimes need to store values between steps, so we used a concurrentHashMap keyed to the thread id to store this kind of data:

class ThreadedStorage {
    static private ConcurrentHashMap multiThreadedStorage = [:]

    static private String threadSafeKey(unThreadSafeKey) {
        def threadId = Thread.currentThread().toString()
        "$threadId:$unThreadSafeKey"
    }

    static private void threadSafeStore(key, value) {
        multiThreadedStorage[threadSafeKey(key)] = value
    }

    def static private threadSafeRetrieve(key) {
        multiThreadedStorage[threadSafeKey(key)]
    }


}

And here's the gist of the Gradle task code that runs the tests multi-threaded using GPars:

def group = new DefaultPGroup(maxSimultaneousThreads())
def workUnits = features.collect { File featureFile ->
    group.task {
        try {
            javaexec {
                main = "cucumber.api.cli.Main"
                ...
                args = [
                     ...
                     '--plugin', "json:$unitReportDir/${featureFile.name}.json",
                             ...
                             '--glue', 'src/test/groovy/steps',
                             "path/to/$featureFile"
                    ]
            }
        } catch (ExecException e) {
                ++noOfErrors
                stackTraces << [featureFile, e.getStackTrace()]
        }
    }
}
// ensure all tests have run before reporting and finishing gradle task
workUnits*.join()

We found we needed to present the feature files in reverse order of execution time for best results.

The results were a 30% improvement on an i5 CPU, degrading above 4 simultaneous threads, which was a little disappointing.

I think the threads were too heavy for multi-threading on our hardware. Above a certain number of threads there were too many CPU cache misses.

Running concurrently on different instances using a thread-safe work queue like Amazon SQS now seems a good way forward, especially since it is not going to suffer from thread-safety issues (at least not on the test framework side).

It is non-trivial for us to test this multi-threading method on i7 hardware due to security constraints in our workplace, but I would be very interested to hear how an i7 with a larger CPU cache and more physical cores compares.