Delay fatality of Travis tests

[pao]

Is there a way we can make a best effort to run all the tests, even if a test fails? Instant fatality means that unrelated tests to a patchset can prevent the patchset from getting appropriate test coverage.

[StefanKarpinski]

I should add this to our test suite's behavior. The Base.Test module needs some work, but not a huge amount.

[pao]

It only matters because people keep insisting on pushing code to master which breaks the tests.

[StefanKarpinski]

That's understandable when they all pass locally but fail only on Travis (how could you have known?) but pushing a build that fails test locally is completely not ok at this point.

[carlobaldassi]

Sorry for the problems I caused, having the code working locally and failing on Travis is precisely what happened (and I'm still fighting to fix all the errors). The suggestion in this issue seems like a good idea.

[StefanKarpinski]

That's bound to happen.

[pao]

It's a good argument for running through a pull request even if you have commit access. Sorry, @carlobaldassi, I came across too sarcastically here; just got kind of annoyed when my pull request suddenly failed tests.

[andreasnoack]

Branches on origin are tested even without a pull request so if you push to a branch before you push to master the problem could be avoided.

[pao]

Oh, that's good to know! Here's the link if anyone is wondering: https://travis-ci.org/JuliaLang/julia/branches

[Keno]

I think we are doing this now?

[ViralBShah]

We are not doing this yet.

[staticfloat]

It only looks like we do this because on Travis we get 8 worker processes, and each worker has a chance to hit a separate test error before dying.

[ViralBShah]

In a particular worker, we could potentially continue all tests and give all errors, rather than just the first one, right?

[amitmurthy]

Upon an error in a particular worker, we should not run more tests in the same worker, since we are not clear what exactly caused the error - for example, could be a memory corruption which will just screw up any additional tests.

We just need to remove err_stop=true from the pmap call in runtests.jl, wrap
Core.include("$name.jl") in a try-catch block in testdefs.jl and return a serialized backtrace to the driver.

[amitmurthy]

Actually just removing err_stop=true should do it.

[IainNZ]

@tkelman you just did the opposite of this right, fail-fast?

[tkelman]

Ref #12387. That was more about the situation where you open a branch or PR and push a bunch of commits to it in rapid succession. It's easier on the Travis and AppVeyor queues to not bother building the redundant builds there when something newer is queued for the same PR/branch.

I think this issue is more about the test framework itself, like your #9398 (edit: and #10835). So if there are multiple problems with a commit, may as well run the tests to completion to find out about all of them at once.

[vtjnash]

I think we can call this resolved by #10835 and testsets (e.g. #13470). Eventually, we may want testfiles to adopt @testset to increase their granularity, but that can be done incrementally and may require performance fixes first (#15346).