When is it OK to group similar unit tests?

Question

I'm writing unit tests for a simple IsBoolean(x) function to test if a value is boolean. There's 16 different values I want to test.

Will I be burnt in hell, or mocked ruthlessly by the .NET programming community (which would be worse?), if I don't break them up into individual unit tests, and run them together as follows:

    [TestMethod]
    public void IsBoolean_VariousValues_ReturnsCorrectly()
    {

        //These should all be considered Boolean values
        Assert.IsTrue(General.IsBoolean(true));
        Assert.IsTrue(General.IsBoolean(false));
        Assert.IsTrue(General.IsBoolean("true"));
        Assert.IsTrue(General.IsBoolean("false"));
        Assert.IsTrue(General.IsBoolean("tRuE"));
        Assert.IsTrue(General.IsBoolean("fAlSe")); 
        Assert.IsTrue(General.IsBoolean(1));
        Assert.IsTrue(General.IsBoolean(0));
        Assert.IsTrue(General.IsBoolean(-1));

        //These should all be considered NOT boolean values
        Assert.IsFalse(General.IsBoolean(null));
        Assert.IsFalse(General.IsBoolean(""));
        Assert.IsFalse(General.IsBoolean("asdf"));
        Assert.IsFalse(General.IsBoolean(DateTime.MaxValue));
        Assert.IsFalse(General.IsBoolean(2));
        Assert.IsFalse(General.IsBoolean(-2));
        Assert.IsFalse(General.IsBoolean(int.MaxValue));
    }

I ask this because "best practice" I keep reading about would demand I do the following:

    [TestMethod]
    public void IsBoolean_TrueValue_ReturnsTrue()
    {
        //Arrange
        var value = true;

        //Act
        var returnValue = General.IsBoolean(value);

        //Assert
        Assert.IsTrue(returnValue);

    }

    [TestMethod]
    public void IsBoolean_FalseValue_ReturnsTrue()
    {
        //Arrange
        var value = false;

        //Act
        var returnValue = General.IsBoolean(value);

        //Assert
        Assert.IsTrue(returnValue);

    }

    //Fell asleep at this point

For the 50+ functions and 500+ values I'll be testing against this seems like a total waste of time.... but it's best practice!!!!!

-Brendan

Answer 1

It's best practice to split each of the values you want to test into separate unit tests. Each unit test should be named specifically to the value you're passing and the expected result. If you were changing code and broke just one of your tests, then that test alone would fail and the other 15 would pass. This buys you the ability to instantly know what you broke without then having to debug the one unit test and find out which of the Asserts failed.

Hope this helps.

Answer 2

I would not worry about it. This sort of thing isn't the point. JB Rainsberger talked about this briefly in his talk Integration Tests are a Scam. He said something like, "If you have never forced yourself to use one assert per test, I recommend you try it for a month. It will give you a new perspective on test, and teach you when it matters to have one assert per test, and when it doesn't". IMO, this falls into the doesn't matter category.

Incidentally, if you use nunit, you can use the TestCaseAttribute, which is a little nicer:

[TestCase(true)]
[TestCase("tRuE")]
[TestCase(false)]
public void IsBoolean_ValidBoolRepresentations_ReturnsTrue(object candidate)
{
    Assert.That(BooleanService.IsBoolean(candidate), Is.True);
}

[TestCase("-3.14")]
[TestCase("something else")]
[TestCase(7)]
public void IsBoolean_InvalidBoolRepresentations_ReturnsFalse(object candidate)
{
    Assert.That(BooleanService.IsBoolean(candidate), Is.False);
}

EDIT: wrote the tests in a slightly different way, that I think communicates intent a little better.

Answer 3

I can't comment on "Best Practice" because there is no such thing.

I agree with what Ayende Rahien says in his blog:

At the end, it boils down to the fact that I don’t consider tests to be, by themselves, a value to the product. Their only value is their binary ability to tell me whatever the product is okay or not. Spending a lot of extra time on the tests distract from creating real value, shippable software.

If you put them all in one test and this test fails "somewhere", then what do you do? Either your test framework will tell you exactly which line it failed on, or, failing that, you step through it with a debugger. The extra effort required because it's all in one function is negligible.

The extra value of knowing exactly which subset of tests failed in this particular instance is small, and overshadowed by the ponderous amount of code you had to write and maintain.

Answer 4

Although I agree it's best practice to separate the values in order to more easily identify the error. I think one still has to use their own common sense and follow such rules as guidelines and not as an absolute. You want to minimize assertion counts in a unit test, but what's generally most important is to insure a single concept per test.

In your specific case, given the simplicity of the function, I think that the one unit test you provided is fine. It's easy to read, simple, and clear. It also tests the function thoroughly and if ever it were to break somewhere down the line, you would be able to quickly identify the source and debug it.

As an extra note, in order to maintain good unit tests, you'll want to always keep them up to date and treat them with the same care as you do the actual production code. That's in many ways the greatest challenge. Probably the best reason to do Test Driven Development is how it actually allows you to program faster in the long run because you stop worrying about breaking the code that exists.

Answer 5

Think for a minute the reasons for breaking them up into individual tests. It's to isolate different functionality and to accurately identify all the things that went wrong when a test breaks. It looks like you might be testing two things: Boolean and Not Boolean, so consider two tests if your code follows two different paths. The bigger point, though, is that if none of the tests break, there are no errors to pinpoint.

If you keep running them, and later have one of these tests fail, that would be the time to refactor them into individual tests, and leave them that way.