Initial gcov work

[cbatson]

Separation of deterministic tests from randomized tests; initial work for gcov build & run. Next step is filling out deterministic tests to trigger branches. All feedback welcome.

[gmaxwell]

Just a quick question I had: Whats the motivation of not running the randomized tests in coverage testing? They're just as deterministic as the rest when run with a static seed.

[cbatson]

I think that could get challenging to maintain in practice. In order to guarantee identical test behavior across changes, it'd be necessary to not only never change the RNG algorithm, but also not change the sequence of calls to the RNG (as could happen with adding, removing, or changing the order of tests, for example). Or, statically seed before every test and make sure the intra-test RNG call pattern doesn't change.

Also, for me personally, I find a fixed sequence of instructions used to evoke a specific condition in the code under test is easier to reason confidently about.

There's philosophical motivations as well, upon which I'm happy to expound if you like. :-)

[cbatson]

P.S. I'm happy to take a different approach if you prefer.

[sipa]

Hi,

sorry for not commenting on this earlier, but there are several changes in the pipeline which will likely conflict with this, in particular #217, #215, and #208. I'd like to see those go in first.

I'm definitely interested in better automatic coverage tests afterwards, though.

[cbatson]

No worries, let me know when you're ready.

[real-or-random]

Closing as obsolete, we have coverage support for a long time now (#426). There's still #954 open though.