Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_interface breaks testsets #15

Closed
sdl1 opened this issue May 26, 2020 · 1 comment · Fixed by #16
Closed

test_interface breaks testsets #15

sdl1 opened this issue May 26, 2020 · 1 comment · Fixed by #16

Comments

@sdl1
Copy link
Member

sdl1 commented May 26, 2020

E.g. on Julia 1.4:

julia> using Models, Models.TestUtils, Test

# Normal behaviour: test summary is printed, and nothing is assigned to results1
julia> results1 = @testset "test" begin
           @test false
       end
test: Test Failed at REPL[2]:2
  Expression: false
Stacktrace:
 [1] top-level scope at REPL[2]:2
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at REPL[2]:2
Test Summary: | Fail  Total
test          |    1      1
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

julia> results1.anynonpass
ERROR: UndefVarError: results1 not defined


# Incorrect behaviour: no test summary printed, and returns a DefaultTestSet
# indicating that tests were passed (anynonpass = false)
julia> results2 = @testset "test" begin
           test_interface(FakeTemplate{PointEstimate, SingleOutput}())
           @test false
       end
Models API Interface Test: FakeTemplate: Test Failed at REPL[5]:3
  Expression: false
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at REPL[5]:2
Test.DefaultTestSet("test", Any[Test.DefaultTestSet(#= circular reference @-2 =#)], 0, false)

julia> results2.anynonpass
false

Maybe related to JuliaLang/julia#32937?

Seems to be solved by removing return from this line

@nickrobinson251
Copy link
Contributor

that is an excellent bit of detective work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants