Getting a "no coverage data has been collected" message using EclEmma and Eclipse

Question

I recently had this very annoying problem come out of nowhere. Running my unit tests with EclEmma coverage enabled causes the following dialog window to pop up from Eclipse:

For the search engines, it reads:

No coverage data has been collected during this coverage Session.
Please do not terminate the Java process manually from Eclipse.

No coverage information is provided for any of the classes in my project. Needless to say I am not terminating the Java process manually. To try and fix it I: reimported my project, upgraded Java, reinstalled Emma, restarted my Macbook Pro, made sure that temp filesystem space looked good, and 20 other things I'm forgetting right now.

I finally noticed that it was only a couple of my open source projects generating this error and decided to whittle down one of my tests. Here's the minimum test that reproduces the problem.

Test class I'm trying to get coverage on:

public class Foo {
    public void method() {
        System.out.println("hello");
    }
}

Here's the junit class which drives it:

public class EclEmmaFailureTest {
    @Test(timeout = 100000) // if you remove the timeout it works
    public void testStuff() {
        // this should cover Foo 100%
        new Foo().method();
        // if you comment this out stuff works
        org.apache.commons.logging.LogFactory.getLog(getClass());
    }
}

The commons-log Log reference in the test seems to break the coverage collection. I've posted a working repo at: https://github.com/j256/eclemma-failure

If you do any of the following, the problem goes away:

• Comment out the LogFactory.getLog(getClass()) call.
• Remove the junit @Test timeout field.
• Downgrade Junit from 4.13.1 to 4.12.

I'm running Emma version 3.1.3 and my test program depends on commons-logging version 1.2.

I have a downgrade path that resolves this problem and gets me working again but Junit 4.12 has security issues. I'm curious if someone knows of the specific issue with junit or emma that is causing this.

Jacoco is also affected which isn't surprising. Here's my coverage report before upgrading to Junit 4.13.1 showing 80% coverage and here it is afterwards with no coverage information available showing 0%.

Thanks.

Answer 1

No coverage data has been collected during this coverage Session. Please do not terminate the Java process manually from Eclipse.

This seems to be a Junit problem around threadgroups that was introduced in version 4.13. See this github discussion and see this pull request. When a timeout was set, it seems that Junit had been destroying the threadgroup to handle stuck threads issues which was not giving Eclemma a chance to write out the coverage information. They seem to have changed the thread-group to be daemon threads as a better way to handle this issue.

Seems like the only solution is going to have to wait for 4.13.2 to be released since versions before 4.13.1 suffer from a security issue.

Looks like shutdown hooks are not executed the same way in 4.13 as it used to happen in 4.12 and thus the coverage failure.

Thanks to the Eclemma mailing list for their help on this.

Answer 2

EMMA is not used here, even if the name EclEmma might imply it. In fact, EclEmma started in 2006 as an Eclipse integration of EMMA. But more than 9 years ago, since EclEmma 2.0, EMMA has been replaced by JaCoCo, a free code coverage library for Java created by the EclEmma team.

Since a code change in the application and/or in the test makes the problem go away, it is very unlikely that the coverage data is collected but not displayed. Therefore, the only likely remaining reason is that something is interfering with JaCoCo collecting the data. The FAQ of JaCoCo names what that might be:

Why does a class show as not covered although it has been executed?

First make sure execution data has been collected. For this select the Sessions link on the top right corner of the HTML report and check whether the class in question is listed. If it is listed but not linked the class at execution time is a different class file. Make sure you're using the exact same class file at runtime as for report generation. Note that some tools (e.g. EJB containers, mocking frameworks) might modify your class files at runtime. Please see the chapter about class ids for a detailed discussion.

To make sure it's not a caching problem, try if also a minor code change makes the problem go away as well.

The things you list that make the problem go away are very different, but all might affect the timing, which would indicate a concurrency issue. You might try to change the order of the tests and/or add Thread.sleep() at some places to see if that changes anything.

However, in your case the root cause is unclear without having minimal reproducible example (that might be difficult to provide, if it is a concurrency issue).

Update:

As Evgeny Mandrikov pointed out, the root problem is indeed a concurrency issue of JUnit 4.13 and 4.13.1 (including all 4.13-beta-* and 4.13-rc-* versions, but previous versions of JUnit are not affected):

JUnit 4 issue #1652: Timeout ThreadGroups should not be destroyed

The issue has already been fixed for the upcoming JUnit 4.13.2 release.

The following can be used as a workaround to prevent the thread group from being destroyed and thus JaCoCo loosing its collected data (by adding a dummy thread into that group):

/** Workaround for already fixed JUnit 4 issue #1652 (<https://github.com/junit-team/junit4/issues/1652>) */
public static void workaround_for_junit4_issue_1652() {
    String version = junit.runner.Version.id();
    if (!"4.13.1".equals(version) && !"4.13".equals(version))
        fail("Workaround for JUnit 4 issue #1652 required for JUnit 4.13 and 4.13.1 only; actual version: "
           + version);
    Thread thread = Thread.currentThread();
    if (!"Time-limited test".equals(thread.getName())) fail("Workaround only required for a test with a timeout.");
    new Thread(thread.getThreadGroup(), new Runnable() {
        @Override
        public void run() {
            try {
                while (!thread.isInterrupted()) Thread.sleep(100);
            } catch (InterruptedException e) {}
        }
    }).start();
}

@Test(timeout = 42)
public void test() {
    workaround_for_junit4_issue_1652();
    // ...
}