Calling return from within a testset, makes overall tests to pass even when there are failures

[oxinabox]

If from a testset you call a function that contains a testset that contains a return, then any test failures after the point the functions are called will not actually cause the outer testset to fail.

Encountered this in the wild (fix is in invenia/Models.jl#16)
where someone thought they could return a value from a testset to use in later tests.
This ended up resulting in ony of our packages passing CI but actually failing tons of tests, and noone noticed for months because they trusted CI and just looked at the end of the tests to see if there was a failure message.

julia> function check_foo()
           @testset "Test Foo" begin
               @test true
               return rand()
           end
       end
check_foo (generic function with 1 method)

julia> k = @testset "main" begin
           check_foo()
           @test false
       end
Test Foo: Test Failed at REPL[7]:3
  Expression: false
Stacktrace:
 [1] top-level scope at REPL[7]:3
 [2] top-level scope at /usr/local/src/julia/julia-1.5/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1114
 [3] top-level scope at REPL[7]:2
Test.DefaultTestSet("main", Any[Test.DefaultTestSet(#= circular reference @-2 =#)], 0, false)

julia> k.anynonpass
false

Probably related to #32937

[DilumAluthge]

This is really bad.

It means that your CI might be green even when your test suite is failing.

It also means that NewPkgEval might say that a package is "passing" even though it is actually failing. @maleadt

[KristofferC]

Probably related to #32937

It's the same, no?

[tkf]

#36046 fixes it