Unable to run TestCaseSource tests when the type under test overrides ToString

Question

First, let me present the test setup in F# (using FsUnit on top of NUnit):

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString()

[<TestFixture>]
type ``Simple Test Cases``() =     
    static member SimpleDataSource =
        [|
            [|{ A = 1; B = 2} ,3|]
        |]

    [<TestCaseSource("SimpleDataSource")>]
    member x.``SimpleTest`` (testData: SimpleRecord * int) =
        let data, expected = testData
        data.A + data.B
        |> should equal expected

This test will run and pass as expected. However, changing the ToString override to include a call to Guid.ToString() will prevent the test from being run:

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString() + Guid.NewGuid().ToString()

With the above change, the test still appears in Test Explorer, but it will not be run. Even right-clicking on it and selecting Run Selected Tests will not execute the test. No build errors are reported.

I also experimented with using DateTime.ToString() instead of Guid.ToString(), but that also refuses to run:

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString() + DateTime.Now.ToString()

Why would calling Guid.ToString() or DateTime.ToString() within the ToString override on the type being tested result in the test not running?

Answer 1

Charlie Poole provided an explanation of the root cause:

When tests are run using the NUnit console or gui runners, they are first loaded (discovered) and then executed. When running under the adapter, the tests are loaded twice, once in the discovery phase and again at the beginning of the execution phase. This is a result of the way the VS Test Window works: discovery and execution are run in separate processes created by VS.

This doesn't cause a problem in most cases. The tests are loaded twice, which means the code for creating any test data is run twice. Usually, the same data is generated both times.

However, when using random data, it's known that different data will be generated in each phase. Tests that are shown in the initial discovery seem to disappear while "unknown" tests are run.

In your case, the guid is generated twice. The first one is used as part of the name of the test, consequently, that test disappears in the execution phase.

Here's an example that demonstrates the workaround:

type ProblemRecord = { A: int; B: int } 
                     override x.ToString() = Guid.NewGuid().ToString()

[<TestFixture>]
type ``Test Cases``() =     
    // Using TestCaseData and explicitly setting a name will resolve the problem
    member x.SolutionDataSource =
        [
            TestCaseData(({ A = 1; B = 2} : ProblemRecord), 3)
                .SetName("Workaround")
        ]

    // This test case will be run by Test Explorer
    [<TestCaseSource("SolutionDataSource")>]
    member x.``SolutionTest`` (data: ProblemRecord, expected: int) =
        data.A + data.B
        |> should equal expected