Propcheck options like numtests not considered until performing mix propcheck.clean

[turion]

Short description

Adding the numtests: some_number option to a property has no effect until MIX_ENV=test mix propcheck.clean is run.

To reproduce

1. Write a test that fails when run multiple times (e.g. using a resource that is tainted after use)
2. Add numtests: 1
3. Be surprised it still fails

Long description with MWE

Consider this test module:

defmodule FooTest do
  use ExUnit.Case
  use PropCheck

  setup do
    Application.ensure_all_started(:propcheck)
    {:ok, resource} = Agent.start_link(fn -> :pristine end, name: :AgentFoo)
    [resource: resource]
  end

  property "foo", [:verbose], ctx do
    forall _ <- PropCheck.BasicTypes.integer() do
      agent_state = Agent.get(ctx.resource, & &1)
      Agent.update(ctx.resource, fn _ -> :tainted end)
      equals(agent_state, :pristine)
    end
  end
end

It contains a test setup that launches an Agent which we suppose necessary for testing. When performing a test, this resource is tainted, and cannot be used anymore:

$ mix test
OK: The input passed the test.
!
Failed: After 1 test(s).
1
:tainted != :pristine

Shrinking .(1 time(s))
0
:tainted != :pristine


  1) property foo (FooTest)
     test/foo_test.exs:11
     Property Elixir.FooTest.property foo() failed. Counter-Example is:
     [0]
     
     Counter example stored.
     
     code: nil
     stacktrace:
       (propcheck) lib/properties.ex:206: PropCheck.Properties.handle_check_results/4
       test/foo_test.exs:11: (test)



Finished in 0.06 seconds
1 property, 1 failure

Randomized with seed 417964

To work around this issue, we want to limit the number of tests to 1. We change the property line to:

  property "foo", [:verbose, numtests: 1], ctx do

Weirdly, this has no effect. The test is still executed several times and of course still fails:

$ mix test
OK: The input passed the test.
!
Failed: After 1 test(s).
0
:tainted != :pristine

Shrinking (0 time(s))
0
:tainted != :pristine

[...]

Strange. Because it has become muscle memory when something in propcheck doesn't work, let's clean generators:

$ MIX_ENV=test mix propcheck.clean

Now it works miraculously:

$ mix test
.
OK: Passed 1 test(s).
.

Finished in 0.04 seconds
1 property, 0 failures

Randomized with seed 602259

I find this behaviour very surprising.

[alfert]

The behaviour is not surprising at all, it is effectively the same situation we found in #30:

• Propcheck finds a counter examples and stories it automatically
• In the next run, PropCheck simply reruns all failing counter examples, quietly assuming that you changed the offending code, i.e. that you tried to fix the bug.
• this will happen until the counter example passes or you clean the counter examples with MIX_ENV=test mix propcheck.clean. This has nothing to do with cleaning generators.

The problem you face is that any changes to a generator or a property which failed before are not detected by PropCheck. As discussed in #30, it is not easy to detect such changes automatically and reliably, since generators and properties may be defined in different files which requiring a thorough code analysis. On the other hand, this seems to be a documentation issue. We report on the command line that you might need to call mix propcheck.clean since the counter examples persists. But this might not be enough and give not the right clues.

Any PRs that fix this problem are highly welcome - either for detecting changes to properties or for better documentation or better command line output!