How to report failing test cases

Question

I am using Haskell Test Framework through Stack to evaluate QuickCheck properties. When I run stack test, failing properties are reported in the form of Gave up! Passed only 95 tests. The many examples of property testing I've found report failures in the form of Falsifiable, after 48 tests followed by the arguments that failed. These examples, however, seem to be running QuickCheck directly instead of through Stack and HTF.

How can I configure my environment to report the arguments generated by QuickCheck that failed to satisfy the property under test? As pointed out in Testing with HTF, documentation is already sparse and poor for some of these tools alone, let alone for combining them together.

Answer 1

"Gave up!" means a different kind of failure than "Falsifiable".

QuickCheck has a way of discarding test cases you consider "improper", counting neither towards actual success or failure. A typical source of such discards comes from using the implication operator (==>), where tests cases which do not satisfy the precondition are discarded: "success" is only counted when the precondition is satisfied, to give you a better idea of the extent to which you are testing the postcondition to the right (which is likely the part that actually matters to you as a user). Explicit use of the discard property is also possible, with a different meaning from an actual failure such as returning False.

Discarded tests thus do not falsify the property as a whole (an implication with a false precondition is logically true), but too many discarded tests may result in insufficient coverage, which is signaled through the failure you observed, and there is no counterexample to print. To resolve this failure, find where the discards are coming from, possible outcomes include:

• use a better generator (avoiding discards);
• raise the discard threshold, @stefanwehr shows how to do this in HTF in the other answer;
• these discards should actually be failures.

Answer 2

@Li-yao Xia is right by saying that your generator generates to many discardable test cases. To raise the discard threshold with HTF, you would write your property like this:

prop_somePropertyWithRaisedDiscardThreshold =
    withQCArgs (\args -> args { maxDiscardRatioy = 1000 })
    somePredicateOrProperty

The args variable has type Args, directly from the quickcheck package.