Property-based testing
- •Generate correct test cases, don't filter down to correct test cases. Example: don't generate all lists and filter to non-empty ones; instead, generate an element and a list and concatenate them.
- •When writing a test that has nontrivial requirements on the input distribution, add instrumentation to it and assert the actual observed distribution. Example: if the last line of the test is
someSet.Contains elt |> shouldEqual (referenceImplementation.Contains elt), we want to make sure we do hit the positive case sometimes; so additionally increment mutable variablespositiveCountandnegativeCount, and after the call to the testing framework to assert the property, check that the counts are in a reasonable ratio to each other. Assert this, rather than merely printing it; it's a bug in the test if the test isn't exploring the space. - •Fuzz over the distribution itself where appropriate. You can cheaply explore very different regimes by biasing the generator according to some parameters and fuzzing over the bias parameters themselves.
- •The tests give more interpretable output if the property throws on failure, rather than merely failing an equality check. In FsCheck, use
b |> shouldEqual a(aunit-returning property) rather thana = b(abool-returning one).