Behavior changes when flag --code-coverage=true is used in 1.0.3

[tkoolen]

I found in tkoolen/PlanarConvexHulls.jl#1 that running tests with code coverage on resulted in a test failure that was not present when code coverage is off. This seems to have been fixed in 1.1.0.

I've tried to reduce the example a bit in the PR branch (to a single 94 line file: https://github.com/tkoolen/PlanarConvexHulls.jl/blob/821015de83c223accb9fd93fd98c4c5a4a65212f/test/runtests.jl), but it's a bit finicky. Removing or adding statements can cause tests to pass. Unfortunately, it still uses StaticArrays and LinearAlgebra, as my attempt to remove either one caused the example no longer reproduce the behavior. Similarly, adding an @show statement here: https://github.com/tkoolen/PlanarConvexHulls.jl/blob/821015de83c223accb9fd93fd98c4c5a4a65212f/test/runtests.jl#L36 resulted in different behavior.

To reproduce, add https://github.com/tkoolen/PlanarConvexHulls.jl.git#tk/debug-travis, run

using Pkg; Pkg.test("PlanarConvexHulls", coverage=true)

and compare to coverage=false.

[KristofferC]

Bounds check elision can change the results.

[tkoolen]

Testing locally, that didn't matter.

[KristofferC]

How did you test locally? Did you recompile the package with the different flag?

[tkoolen]

First, whether you run

Pkg.test("PlanarConvexHulls", coverage=true)

or

Pkg.test("PlanarConvexHulls", coverage=false)

tests will be run in a new process with check-bounds=yes, so as I understand it that can't be the cause for the difference in behavior.

For more evidence, you can dev the package and just include("runtests.jl") after starting the REPL with --check-bounds={yes|no} and --code-coverage={none|user|all}, which shows that --code-coverage={user|all} results in different behavior than --code-coverage=none, regardless of the value of the --check-bounds.

[mdashti]

We also have a similar issue in our private repository at RelationalAI, but couldn't create a small snippet to reproduce the issue, yet.

[KristofferC]

So to be clear, this has not really anything to do with Pkg but difference in behavior in running code depending on the --code-coverage setting? In that case, perhaps we can update the issue title.

[mdashti]

@KristofferC Yeah, that's correct. At least in our case, just running Julia with julia --code-coverage=user reproduces the issue.

An update on the issue on our side, it's solved after we've removed an explicit type declaration on a local variable, which is storing the value returned by tryparse(::Type{Float64}, ::WeakRefString), e.g.:

parsed_val::Union{Nothing, Float64} = tryparse(Float64, my_weak_ref_string_var)

to

parsed_val = tryparse(Float64, my_weak_ref_string_var)

[tkoolen]

I just ran into this again in a different package (see referenced PR above).

[vtjnash]

Apparently fixed already