Failed automated tests: how to distinguish known and newly introduced bugs?

Question

Use case: Fitnesse is used for automated testing of the web site.

SUT (software under test) contains a known bug. Say, we expect that web page contains "Changes saved successfully" string but this string is missing because of the bug. So in Fitnesse, this test case is marked as red.

Suppose, in another test case we expect that web page contains "A user created successfully" string. And it worked just fine until the last test execution. So, now this test case is also marked as red.

So, now we have red light for two test cases: a well known bug and a newly found bug. The problem is they are both marked as red. So when I have a look at test results I can't distinguish which of them are known and new.

Of course, I can compare test history and see the difference between two runnings (with and without a newly created bug).

Or I may not execute a test case with a known bug.

Or I can tune it so that this test case has always been green and change it when the bug is fixed.

But all this is very inconvenient. What I want is to differ two kinds of bugs (a well known bug and new bug) so that:

1. By looking at test results, I could easily say: this is a new bug and those are old. For example: No bugs - green, Already known bugs - yellow, New bugs - red.

2. It is easy to change test case when the bug is fixed.

What are the best strategies for acceptance tests, in general, and Fitnesse, in particular?

Answer 1

There's a subtle distinction here: you're talking about tracking test state, not just whether it's a known bug. Good CI systems can help you track a test's state via history, and let you know when it changes state. (Passed yesterday, fails today.) Good CI systems can also resolve false failures so they don't muddy up your history. (I'm thinking specifically of Team City which I've done this in.)

Having a bug against a failing test is another issue. Naming conventions can help, as Barry mentioned. I've also used test framework metadata to help identify existing bugs by marking descriptions in test attributes or properties.

Answer 2

To the best of my knowledge, there is no easy way to distinguish between a 'new' bug and an expected bug, other than to have a list of known issues handy. If you use a naming convention where you can quickly list tests that are known that they're going to fail, you can then quickly make a scan of which bugs are not on that list - I.E. they're new bugs that need to be looked at.

Answer 3

This is actually multiple questions. I'm going to focus on what I consider a good practice for running FitNesse tests.

I suggest that you run your tests in a CI system like Hudson. CI Systems are good at tracking what happens from run to run.

To do this you will need results in a format Hudson can track. You can use XSL to transform the results of FitNesse Suite runs into Junit style reports and then leverage Hudson's ability to track Junit results. This gives you trends and graphs. It also lets you see the age of any failure. Here is my way of doing that: http://whotestedthis.squarespace.com/journal/2012/1/26/transforming-fitnesse-results-to-junit.html

Answer 4

When we log a confirmed bug, we add a #warning at the failing statement in source code of the test to indicate the bug number and a (very) short description of the failing behavior. If we don't expect the bug to be fixed soon (e.g., within a few days), we quarantine the test into a group of tests that are expected to fail. We review the history those tests periodically, to ensure that they haven't developed new failure modes, and we move them out of quarantine when the bug is fixed.

As Jim Holmes says, a good CI system (we also use JetBrains' TeamCity) will maintain the history in a manner that makes this kind of post mortem analysis easy.