Multithreaded testing

[wiredfool]

This is the update to the other half of the multithreading PR.

• It adds environment variables and flags to let nosetests run in parallel
• It changes the way that tempfiles are created to safely create and delete the tempfiles.
• It changes the reporting of temp files that are not cleaned up.

Tempfiles are now created using tempfile.mkstmp, the secure, unaffected by race conditions api, rather than manually creating a well known directory and adding possibly unique names in it. The tempfiles are cleaned up after each individual successful test. If not successful, then the test file is reported as remaining at that time. This allows tests to be more independent, reorderable, and not have any global state that needs cleaned up at the end of a run.

This removes tearDownModule, as it is run each time a module changes, which could be as frequently as every test. Global state in the teardown function is very difficult to handle when using multiple test runners.

FWIW, this appears to improve the testing time from ~1:20 to ~30 seconds wall time for the testing phase on travis.

[coveralls]

Changes Unknown when pulling 2b4de32 on wiredfool:multithreaded_testing into * on python-pillow:master*.

[hugovk]

Good stuff, shaved off a good 4-5 minutes from the build too: from about 90s to 30s per test run.

But coverage reporting is broken, it's really higher than 34%: we'll need have separate coverage data files and then combine at the end.

[wiredfool]

From http://nose.readthedocs.org/en/latest/doc_tests/test_multiprocess/multiprocess.html:

The multiprocess plugin does not work well with other plugins that expect to wrap or gain control of the test-running process. Examples from nose’s builtin plugins include coverage and profiling: a test run using both multiprocess and either of those is likely to fail in some confusing and spectacular way.

So, it's either fast tests or working coverage for now, until the upstreams fix it.

Or... http://nedbatchelder.com/code/coverage/cmd.html

If you are measuring coverage in a multi-process program, or across a number of machines, you’ll want the --parallel-mode switch to keep the data separate during measurement. See Combining data files below.

https://bitbucket.org/memedough/nose-cov/overview

The nose cov plugin partially works with the nose multiprocess plugin

[coveralls]

Coverage decreased (-39.07%) when pulling 5e750d5 on wiredfool:multithreaded_testing into d79d281 on python-pillow:master.

[coveralls]

Coverage decreased (-39.07%) when pulling 1f7c4fa on wiredfool:multithreaded_testing into d79d281 on python-pillow:master.

[wiredfool]

Well, that didn't do it.

[coveralls]

Coverage decreased (-39.07%) when pulling b690570 on wiredfool:multithreaded_testing into d79d281 on python-pillow:master.

[hugovk]

Working coverage is more useful than faster tests.

It may be worth asking Ned Batchelder directly for tips.

[wiredfool]

Well, that appears to have fixed the coverage generation on travis, it's reporting 71%, not so sure about coveralls reporting, which is reporting a grey box.

[wiredfool]

@hugovk Any ideas on what's wrong here?

[hugovk]

@wiredfool Not really. Clearly Coveralls isn't processing the combined build report properly, or isn't being sent it properly.

It looks like we're combining properly, as coverage report looks fine in the build (except for PyPy but that's happening elsewhere too) so coverage.py seems to be working fine. But then the Coveralls shows grey question marks for all jobs, and then the list of files at the bottom shows all 86 but only 8 have any coverage percentage; the rest are grey question marks.

This is the third-party coveralls tool we're using to send reports from Travis to Coveralls.io:
https://github.com/coagulant/coveralls-python

[wiredfool]

I'm checking this locally, and it's consistent at least. There's nothing obviously wrong that I can see.

[wiredfool]

Coveralls hates me: #748 (comment) #749 (comment)

(Both of these are Changes Unknown, but with a reasonable percentage)

[wiredfool]

see #754, #755