FsCheck with Setup and Teardown

Question

Summary

Are there any events that can be run before every property case so that I can run setup and teardown for each run of a property?

Full version

I want to be able to test paired behaviors like "I can always fetch written records" or "output of readAllLines equals input to writeAllLines" with properties. I also want the property to not care how the operation sets are implemented (i.e. if any resources need to be cleaned up).

Each run of the property should

• be independent from other runs
• maintain state between calls of the operations within this single run
• not know about how how the operations maintain state
• not be a resource leak

I'm using FsCheck and Expecto. Examples will be in Expecto, but the problem isn't specific to the framework.

It is super easy to write this sort of setup and teardown with example-based tests. They take a predictable argument set, so I can run them in a wrapper that adds before and after events.

let testWithEnv setup cleanup name test = 
    let testWrap () = 
        let (api, env) = setup ()
        test api
        cleanup env 
    testCase name testWrap

The same can't be done with property tests. They have an unknown number of arguments that will largely be filled with random data.

I can apply the set of paired behaviors pretty easily, but any created resources like streams are left undisposed.

let testPropertyWithEnv setup cleanup name test = 
    let testWrap () = 
        let (api, env) = setup () // this is actually run once, but immutable so the individual runs don't leak state
        test api // have to return this to pass along unapplied parameters
    testProperty name testWrap

I've looked into

Runner events

Looking at how to run FsCheck tests the closest hooks appear to be

• OnStartFixture which is only run once per test class
• OnArguments is run after every pass and would potentially work to run cleanup

Model-based features

There is also the experimental Model-based testing features that could work. However, it seems really heavy considering I'm only concerned with the external consistency of operations. I do not want access to the backing state.

Giving up and inlining

I can always write

testProperty "name" (fun arg1 arg2 ->
    let (api,env) = setup ()
    //test code here
    cleanup env
)

but I'd like to avoid the boilerplate in every property and the exposure of the backing state.

IDisposables

Disposable objects also don't address the lack of setup hook.

More hands-on run loop

I looked into ways of running the property tests in my wrapper, but the smallest runner Check.one is for a single property, with no hooks between runs of the property.

Lazy wrapper

Making the wrapper lazy also doesn't work testProperty name lazy(testWithSetup)

Answer 1

There isn't really anything in FsCheck to help you, and even if there was I don't think it'd be straightforwardly exposed in Expecto. I also don't think it's that straightforward to add on the FsCheck side - that said happy to discuss that further if you open an issue in the FsCheck repo.

In any case, with some clever use of partial application and at the cost of some slight boilerplate, it is in fact possible to wrap "variadic" functions, which I think is essentially what you're asking.

Behold, the code:

// these types are here to make the signatures look nicer
type Api = Api
type Env = Env

let testProperty k = 
    // call the property with "random" arguments
    for i in 0..2 do
        k i (char i) (string i)

let setup() = 
    printfn "setup ran"
    (Api, Env)

let teardown Env = 
    printfn "teardown ran"

let test0 Api arg1 = 
    printfn "test0 %A" arg1

let test1 Api (arg1:int) (arg2:char) = 
    printfn "test1 %A %A" arg1 arg2

let test2 Api arg1 arg2 arg3 =
    printfn "testFun %A %A %A" arg1 arg2 arg3

let testWithEnv (setup:unit -> Api*Env) (teardown: Env -> unit) (test: Api -> 'a) (k: 'a -> unit) :unit =
    let (api, env) = setup()
    k (test api)
    teardown env

let (<*>) (f,k) arg  =
    f, (fun c -> k c arg)

let (<!>) f arg =
    f, (fun k -> k arg)

let run (f, k) = f k

testProperty (fun arg1 arg2 arg3 ->
    testWithEnv setup teardown test2 <!> arg1 <*> arg2 <*> arg3 |> run
)

The idea here is that you use the operators <!> and <*> to string together any number of arguments of any type, pass that to the testWithEnv function and then call run on the result. The operators and the run are basically necessary to build up and then apply the variadic list of arguments.

It's all type safe i.e. if you forget to pass an argument or it's of the wrong type, you'll get a type error, though admittedly it might not be as clear as normal function application.

I recommend pasting this in an IDE and checking out the types, that will help greatly with understanding what is going on.

There are a few other styles you can write this in. E.g. with a slightly different definition and a the function you can write something like

let (<+>) k arg  =
    fun c -> k c arg

let the arg =
    fun k -> k arg

testWithEnv setup teardown test2 (the arg1 <+> arg2 <+> arg3)

This replaces the run function with the and needs only one operator <+>. There's probably other ways to cut it too, pick your poison.