Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix invalidations in open from Static.jl #46551

Merged
merged 1 commit into from
Aug 31, 2022
Merged

Conversation

ranocha
Copy link
Member

@ranocha ranocha commented Aug 30, 2022

This is based on the following code:

julia> using Pkg; Pkg.activate(temp=true); Pkg.add("Static")

julia> using SnoopCompileCore; invalidations = @snoopr(using Static); using SnoopCompile

julia> trees = invalidation_trees(invalidations)
inserting !(::False) in Static at ~/.julia/packages/Static/sVI3g/src/Static.jl:427 invalidated:
...
                 15: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.PlatformEngines.var"#26#28"{String}, ::Cmd) (0 children)
...
                 27: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, _A, String} where _A, ::Cmd) (0 children)
                 28: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, Tar.var"#1#2", String}, ::Cmd) (0 children)
...
                 65: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Function, ::Cmd) (20 children)
...
                 67: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.Registry.var"#11#14"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd) (28 children)
   15 mt_cache
...

I suggest the labels latency and backport-1.8.

@PallHaraldsson
Copy link
Contributor

PallHaraldsson commented Aug 30, 2022

It's unclear why this is needed. It seems eof returns Bool, documented as such (does it needs to be enforced there, rather than at the call site?):

eof(stream) -> Bool

@ranocha
Copy link
Member Author

ranocha commented Aug 30, 2022

The compiler doesn't read the docs:

julia> ascend(root)
Choose a call for analysis (q to quit):
 >   #open#734(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.Registry.va
       open(::Pkg.Registry.var"#11#14"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd)
         uncompress_registry(::String)
           Pkg.Registry.RegistryInstance(::String)
             #reachable_registries#17(::Vector{String}, ::typeof(Pkg.Registry.reachable_registries))
               reachable_registries()
                 find_installed_registries(::IO, ::Vector{Pkg.Registry.RegistryInstance})
                   #update#55(::IO, ::Bool, ::typeof(Pkg.Registry.update), ::Vector{Pkg.Registry.RegistrySpec})
                 find_installed_registries(::IO, ::Vector{Pkg.Registry.RegistrySpec})
v                  #update#55(::IO, ::Bool, ::typeof(Pkg.Registry.update), ::Vector{Pkg.Registry.RegistrySpec})
var"#open#734"(kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(open), f::Function, cmds::Base.AbstractCmd, args...) in Base at process.jl:414
Variables
  #open#734::Core.Const(Base.var"#open#734")
  kwargs::Core.Const(Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}())
  @_3::Core.Const(open)
  f::Pkg.Registry.var"#11#14"{IOBuffer, Vector{UInt8}, Dict{String, String}}
  cmds::Cmd
  args::Tuple{}
  ret::Dict{String, Any}
  waitkill::Base.var"#waitkill#736"
  P::Base.Process
  @_10::Base.Process
  @_11::Dict{String, Any}

Body::Dict{String, Any}
     @ process.jl:415 within `#open#734`
1 ──       nothing
│          Core.NewvarNode(:(ret))
│          Core.NewvarNode(:(waitkill))
│          Core.NewvarNode(:(P))
│    %5  = Base.NamedTuple()::Core.Const(NamedTuple())
│    %6  = Base.merge(%5, kwargs)::Core.Const(NamedTuple())
│    %7  = Base.isempty(%6)::Core.Const(true)
│          Core.typeassert(%7, Core.Bool)
│    %9  = Core.tuple(cmds)::Tuple{Cmd}
│          (@_10 = Core._apply_iterate(Base.iterate, Base.open, %9, args))
└───       goto #2
2 ──       (P = @_10)
│    @ process.jl:416 within `#open#734`
└───       (waitkill = %new(Base.:(var"#waitkill#736")))
     @ process.jl:426 within `#open#734`
3 ──       $(Expr(:enter, #6))
     @ process.jl:427 within `#open#734`
4 ──       (@_11 = (f)(P))
└───       $(Expr(:leave, 1))
5 ──       goto #8
6 ┄─       $(Expr(:leave, 1))
     @ process.jl:429 within `#open#734`
7 ──       (waitkill)(P)
│    @ process.jl:430 within `#open#734`
└───       (@_11 = Base.rethrow())
8 ┄─       (ret = @_11)
│    @ process.jl:432 within `#open#734`%22 = Base.getproperty(P, :in)::IO
│          Base.close(%22)
│    @ process.jl:433 within `#open#734`%24 = Base.getproperty(P, :out)::IO%25 = Base.eof(%24)::Any%26 = !%25::Any
└───       goto #10 if not %26
     @ process.jl:434 within `#open#734`
9 ──       (waitkill)(P)
│    @ process.jl:435 within `#open#734`%29 = Base._UVError("open(do)", Base.UV_EPIPE)::Any
└───       Base.throw(%29)
     @ process.jl:437 within `#open#734`
10%31 = Base.success(P)::Bool
└───       goto #12 if not %31
11 ─       goto #13
12 ─       Base.pipeline_error(P)
     @ process.jl:438 within `#open#734`
13return ret

Note the line Base.eof(%24)::Any.

@PallHaraldsson
Copy link
Contributor

The compiler doesn't read the docs:

Right. I'm fine with the PR if it fixes. I was just curious if it could be fixed at the (non-doc) source, in case that's it, but Bool is there:

eof(io::AbstractPipe) = eof(pipe_reader(io)::IO)::Bool

Maybe it's about that !

@ranocha
Copy link
Member Author

ranocha commented Aug 30, 2022

From #46491 (comment)

inference stops checking whether all possible callees return Bool if the number of methods exceeds 3. Thus in many cases, just adding method annotations does nothing to fix invalidations.

@timholy
Copy link
Member

timholy commented Aug 31, 2022

If it's not inconvenient, you can bundle several of these fixes in a single PR. Example: #44500

@timholy timholy merged commit 7537cb3 into JuliaLang:master Aug 31, 2022
@ranocha ranocha deleted the patch-7 branch August 31, 2022 12:17
@ranocha
Copy link
Member Author

ranocha commented Aug 31, 2022

Thanks!

If it's not inconvenient, you can bundle several of these fixes in a single PR. Example: #44500

Sure, I will try to do so. I just wasn't sure what's better - to combine edits in several places into a single PR or to create several smaller PRs that are easier to review since they concentrate on a single functional unit.

Could we the label backport-1.8 to get this into Julia v1.8.1?

@giordano giordano added compiler:latency Compiler latency backport 1.8 Change should be backported to release-1.8 labels Aug 31, 2022
ranocha added a commit to ranocha/julia that referenced this pull request Sep 7, 2022
This fixes invalidations from loading Static.jl
KristofferC pushed a commit that referenced this pull request Sep 7, 2022
This fixes invalidations from loading Static.jl

(cherry picked from commit 7537cb3)
@KristofferC KristofferC mentioned this pull request Sep 7, 2022
28 tasks
ranocha added a commit to ranocha/julia that referenced this pull request Sep 7, 2022
This fixes invalidations from loading Static.jl
@KristofferC KristofferC removed the backport 1.8 Change should be backported to release-1.8 label Sep 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:latency Compiler latency
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants