Tools to find code coverage intersections?

Question

Our company is investigating the idea of using cobertura code coverage during manual regression tests to find out where we have 'adjacent functionality'. The general idea would be if regression test A hit method businessLogicFoo(), and regression test B also hit that method we could say that regression test A and B have 'adjacent functionality'.

We're particularly interested in efficiently determining which regression tests have 'adjacent functionality' so that we can schedule better regression runs (we have vastly more tests that time to test - so we always end up testing a subset of all regression worthy tests).

Has anyone attempted something like this before? With cobertura or some other code coverage library?

My first guess is that we write a groovy script (my preferred scripting language) to export the cobertura reports as XML, then parse out the classes/methods covered - filter out any superfluous classes - and then find method/class intersections between the two reports. Ideally all controlle in maven. But I'm just guessing.

Answer 1

Your guess sounds about right (though I'd use Ruby myself :)). But hen you've determined which of your tests are potentially duplicated, you will then of course have to determine whether the tests are really superfluous (they may just partially overlap, in which case combining them would be the answer, not scrapping one in favour of another). I don't know any way except manually going over all identified hits to do that.

Answer 2

Our Test Coverage tools will compute intersections, unions, complements, set differences, of test coverage vectors easily, making your intersections trivial to compute.

You can associate an individual test with its own test coverage vector, and compute intersections of that with other tests, telling you how much overlap two tests have. If they have small overlap, then they are "adjacent" by your characterization. Often they will exercised some shared "core functionality" which might confuse the issue, but you can compute this by intersection of a number of tests which exercise that core functionality, and subtract it away, to see what they do not share.

Even better, if you modify some code, the tool will tell you which test coverage sets (e.g., which tests) were affected, and therefore which ones you need to run. More importantly, this tells which ones you don't need to run. If your tests are manual, this should be really helpful in minimizing re-testing effort.