Tests can break Test

[timholy]

If you dev the brand-new https://github.com/JuliaMath/CheckedArithmetic.jl and then do pkg> test CheckedArithmetic, all seems well. But then insert error("oops") here, and you'll find to your dismay that the tests still pass.

If you comment out the "check" tests, then it fails as expected.

EDIT: package has been fixed. It was due to a habit of writing return val at the end of a block, and mistakenly doing that in the @check macro definition.

[fredrikekre]

This is also funny when putting it on e.g. L42:

@checked: Error During Test at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:12
  Got exception outside of a @test
  fsdfds
  Stacktrace:
   [1] error(::String) at ./error.jl:33
   [2] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:42
   [3] top-level scope at /home/fredrik/julia13/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [4] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:13
   [5] top-level scope at /home/fredrik/julia13/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [6] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:12
   [7] include at ./boot.jl:328 [inlined]
   [8] include_relative(::Module, ::String) at ./loading.jl:1105
   [9] include(::Module, ::String) at ./Base.jl:31
   [10] include(::String) at ./client.jl:424
   [11] top-level scope at none:6
   [12] eval(::Module, ::Any) at ./boot.jl:330
   [13] exec_options(::Base.JLOptions) at ./client.jl:263
   [14] _start() at ./client.jl:460
  
   Testing CheckedArithmetic tests passed

You get an error with a stacktrace, but the process exits with 0.

[rfourquet]

The error seems to come from the @check macro, e.g. @test @check(3+5) == 12 will succeed. I think it comes from your @check macro, it shouldn't return in the returned quoted expression. I.e. we observe the same behavior here if you put a return 0 within a @testset.

Very cool package by the way!

[timholy]

Yes, I mistakenly added a return statement to the returned expression. Perhaps a duplicate of #32937?

[timholy]

Off-topic, but is this an oversight?

julia> Ref(3) == Ref(3)
false

[fredrikekre]

#31885

[rfourquet]

And #31813 !

[tkf]

Perhaps a duplicate of #32937?

FYI, if so, #36046 might have fixed this.