Parameterized method runs twice

Question

I am running this code, and realized that getAllParameters() method runs twice for some reason. Because the static field enumMap is initialized outside that method, it gets populated twice, which results in duplicate elements and fails the test I'm running.

I figured that initializing enumMap inside the method fixes the problem, as the map does get reset when the method runs the 2nd time.

Even though this fixes the problem, I am wondering why this happens when running Maven Test? I played around with the number of parameters, thinking that might possibly affect how many times the method runs, but it only seems to be running twice.

@RunWith(Parameterized.class)
public class MyTest {

    private static Map<String, List<Class<? extends LocalizedJsonEnum>>> enumMap = new HashMap<>();

    @Parameter
    @SuppressWarnings({"WeakerAccess", "unused"})
    public Class<? extends LocalizedJsonEnum> currentEnum;
    @Parameter(value = 1)
    @SuppressWarnings({"WeakerAccess", "unused"})
    public  String currentClassName;

    /**
     * Generate a list of all the errors to run our test against.
     *
     * @return the list
     */
    @Parameters(name = "{1}.class")
    public static Collection<Object[]> getAllParameters() throws Exception {
        Collection<Object[]> parameters = new LinkedList<>();
        Reflections reflections = new Reflections("com.class.path");
        Set<Class<? extends LocalizedJsonEnum>> JsonEnums = reflections.getSubTypesOf(LocalizedJsonEnum.class);

        //workaround: initialize here
        //enumMap = new HashMap<>();

        //some code that inserts elements into the enumMap and parameters 
        return parameters;
    }


@Test
public void testEnumIdentifierIsNotDuplicated() throws Exception {

    String enumId;
    if (currentEnum.isAnnotationPresent(Identifier.class)) {
        enumId = currentEnum.getAnnotation(Identifier.class).value();
    } else {
        enumId = currentEnum.getSimpleName();
    }
    List<Class<? extends LocalizedJsonEnum>> enumList = enumMap.get(enumId);

    if (enumList.size() > 1) {

        StringBuilder sb = new StringBuilder("Enum or identifier [" + enumId + "] has been duplicated in the following classes:\n");
        for (int listIndex = 0; listIndex < enumList.size(); listIndex++) {
            Class<? extends LocalizedJsonEnum> enumDuplicate = enumList.get(listIndex);
            sb.append("   [").append(listIndex).append("] Enum Class:[").append(enumDuplicate.getName()).append("]\n");
        }
        fail(sb.toString());
    }
}

Answer 1

You've got a parameterized test function, hence, the system will run it twice - once for each parameter.

There are different reasonable ways as to how it does this.

One simple solution is to do the most obvious thing. From the point of view of the test framework:

1. Create an instance of MyTest once.
2. Call the test method once for the first parameter.
3. Call it again, for the second parameter.

Of course, this results in 'reuse' of fields and calls - the object in the second test run is 'tainted'. Hence, that's not actually how any test framework works. Instead, they do this:

1. Create an instance of MyTest.
2. Call the test method once for the first parameter.
3. Toss away the previous instance and make a new one, to start with a clean slate as much as possible.
4. Call it again, for the second parameter, using the new instance.

But, because you are using static this still isn't good enough. One new instance for each test invoke doesn't do anything to 'reset' static stuff. So, the test framework could instead do this vastly more convoluted setup:

1. Create a custom classloader instance.
2. Use it to load the test class.
3. Create a new instance.
4. Run the first test, using the first parameter.
5. Toss away the instance, and the entire classloader.
6. Start back at item 1 for the second run.

This would do what you want: Every 'test invoke' (here you have 1 test method and 2 invokes; one for each parameter option) gets a completely blank slate.

However this is going to do quite a number on performance. Class loading is relatively expensive; you need to open a file on disk, and even if you rely on the OS to cache (as every test opens the same section of the jar file or whatnot every time), turning a sack o' bytes into a class definition, running the verifier on it, that's not exactly cheap. The property of 'running the test suite takes a long time' is a productivity drain that nobody wants to pay, but, having 'clean slate' tests where the order in which tests run has no effect on the results is also something everybody wants. So where do we cut this particular cake? Do we give people the 'test independence' they prefer, at the cost of slow-as-molasses test suite running? Or do we give them as speedy a test run as possible, but toss away the notion of test independence like yesterday's newspaper?

The real answer is somewhere in the middle with a sprinkling of configurability.

This is what I strongly suggest you do:

1. Avoid static in your test code unless you intentionally want some aspect of the test to be reused between multiple invokes.
2. There where you explicitly want to do a task once and then 'reuse' it for more than a single test invoke, use the tools for this: @Setup, @Teardown, @BeforeEach,@BeforeAll and @BeforeClass should be used for this stuff.

Now you have full control over the very problem you ran into:

You would put the code that 'makes' the enumList in a separate method that is not itself a @Test. And isn't static.

You mark this method as @BeforeAll if you want it to run once. You mark it @BeforeEach if you prefer the slower, but more 'independent' concept instead. The code should, of course, 'reset' (it shouldn't just add stuff, it should clear / create a new list and fill that).