Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression in 1.9.0-rc2 compared to 1.9.0-rc1 (weak dependencies not loaded correctly at precompilation?) #49250

Closed
ranocha opened this issue Apr 4, 2023 · 6 comments · Fixed by #49257
Milestone

Comments

@ranocha
Copy link
Member

ranocha commented Apr 4, 2023

I can install the package SummationByPartsOperators.jl without problems on Julia 1.9.0-rc1
julia> using Pkg; Pkg.activate(temp = true); Pkg.add("SummationByPartsOperators")
  Activating new project at `/tmp/jl_flVwL0`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/jl_flVwL0/Project.toml`
  [9f78cca6] + SummationByPartsOperators v0.5.31
    Updating `/tmp/jl_flVwL0/Manifest.toml`
  [621f4979] + AbstractFFTs v1.3.1
  [79e6a3ab] + Adapt v3.6.1
  [dce04be8] + ArgCheck v2.3.0
  [4fba245c] + ArrayInterface v7.4.3
  [30b0a656] + ArrayInterfaceCore v0.1.29
  [15f4f7f2] + AutoHashEquals v0.2.0
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.5
  [2a0fbf3d] + CPUSummary v0.2.2
  [fb6a15b2] + CloseOpenIntervals v0.1.12
  [38540f10] + CommonSolve v0.2.3
  [34da2185] + Compat v4.6.1
  [187b0558] + ConstructionBase v1.5.1
  [adafc99b] + CpuId v0.3.1
  [9a962f9c] + DataAPI v1.14.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [ffbed154] + DocStringExtensions v0.9.3
  [4e289a0a] + EnumX v1.0.4
  [e2ba6199] + ExprTools v0.1.9
  [7a1cc6ca] + FFTW v1.6.0
  [442a2c76] + FastGaussQuadrature v0.5.0
  [069b7b12] + FunctionWrappers v1.1.3
  [77dc65aa] + FunctionWrappersWrappers v0.1.3
  [46192b85] + GPUArraysCore v0.1.4
  [3e5b6fbb] + HostCPUFeatures v0.1.14
  [615f187c] + IfElse v0.1.1
  [92d709cd] + IrrationalConstants v0.2.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.4.1
  [10f19ff3] + LayoutPointers v0.1.14
  [50d2b5c4] + Lazy v0.15.1
  [2ab3a3ac] + LogExpFunctions v0.3.23
  [bdcacae8] + LoopVectorization v0.12.155
  [1914dd2f] + MacroTools v0.5.10
  [d125e4d3] + ManualMemory v0.1.8
  [46d2c3a1] + MuladdMacro v0.2.4
  [6fe1bfb0] + OffsetArrays v1.12.9
  [bac558e1] + OrderedCollections v1.6.0
  [1d0040c9] + PolyesterWeave v0.2.1
  [c74db56a] + PolynomialBases v0.4.17
  [21216c6a] + Preferences v1.3.0
  [3cdcf5f2] + RecipesBase v1.3.3
  [731186ca] + RecursiveArrayTools v2.38.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [7e49a35a] + RuntimeGeneratedFunctions v0.5.6
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.38
  [0bca4576] + SciMLBase v1.91.6
  [c0aeaf25] + SciMLOperators v0.2.0
  [efcf1570] + Setfield v1.1.1
  [ce78b400] + SimpleUnPack v1.1.0
  [66db9d55] + SnoopPrecompile v1.0.3
  [276daf66] + SpecialFunctions v2.2.0
  [aedffcd0] + Static v0.8.6
  [0d7ed370] + StaticArrayInterface v1.3.0
  [90137ffa] + StaticArrays v1.5.19
  [1e83bf80] + StaticArraysCore v1.4.0
  [9f78cca6] + SummationByPartsOperators v0.5.31
  [2efcf032] + SymbolicIndexingInterface v0.2.2
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.10.1
  [8290d209] + ThreadingUtilities v0.5.1
  [410a4b4d] + Tricks v0.1.7
  [781d530d] + TruncatedStacktraces v1.3.0
  [3a884ed6] + UnPack v1.0.2
  [9602ed7d] + Unrolled v0.1.4
  [3d5dd08c] + VectorizationBase v0.21.64
  [f5851436] + FFTW_jll v3.3.10+0
  [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2
  [856f044c] + MKL_jll v2022.2.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics v1.9.0
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [05823500] + OpenLibm_jll v0.8.1+0
  [bea87d4a] + SuiteSparse_jll v5.10.1+6
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850b90] + libblastrampoline_jll v5.4.0+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
Precompiling environment...
  79 dependencies successfully precompiled in 48 seconds. 7 already precompiled.
  1 dependency had warnings during precompilation:
┌ SciMLOperators [c0aeaf25-5076-4817-a8d5-81caf7dfa961]
│  ┌ Warning: Replacing docs for `SciMLOperators.AbstractSciMLOperator :: Union{}` in module `SciMLOperators`
│  └ @ Base.Docs docs/Docs.jl:243
└  

julia> import Pkg; Pkg.precompile()

julia> using SummationByPartsOperators
However, the package cannot be precompiled on Julia 1.9.0-rc2. Full output:
julia> using Pkg; Pkg.activate(temp = true); Pkg.add("SummationByPartsOperators")
  Activating new project at `/tmp/jl_EOdZr8`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/jl_EOdZr8/Project.toml`
  [9f78cca6] + SummationByPartsOperators v0.5.31
    Updating `/tmp/jl_EOdZr8/Manifest.toml`
  [621f4979] + AbstractFFTs v1.3.1
  [79e6a3ab] + Adapt v3.6.1
  [dce04be8] + ArgCheck v2.3.0
  [4fba245c] + ArrayInterface v7.4.3
  [30b0a656] + ArrayInterfaceCore v0.1.29
  [15f4f7f2] + AutoHashEquals v0.2.0
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.5
  [2a0fbf3d] + CPUSummary v0.2.2
  [fb6a15b2] + CloseOpenIntervals v0.1.12
  [38540f10] + CommonSolve v0.2.3
  [34da2185] + Compat v4.6.1
  [187b0558] + ConstructionBase v1.5.1
  [adafc99b] + CpuId v0.3.1
  [9a962f9c] + DataAPI v1.14.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [ffbed154] + DocStringExtensions v0.9.3
  [4e289a0a] + EnumX v1.0.4
  [e2ba6199] + ExprTools v0.1.9
  [7a1cc6ca] + FFTW v1.6.0
  [442a2c76] + FastGaussQuadrature v0.5.0
  [069b7b12] + FunctionWrappers v1.1.3
  [77dc65aa] + FunctionWrappersWrappers v0.1.3
  [46192b85] + GPUArraysCore v0.1.4
  [3e5b6fbb] + HostCPUFeatures v0.1.14
  [615f187c] + IfElse v0.1.1
  [92d709cd] + IrrationalConstants v0.2.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.4.1
  [10f19ff3] + LayoutPointers v0.1.14
  [50d2b5c4] + Lazy v0.15.1
  [2ab3a3ac] + LogExpFunctions v0.3.23
  [bdcacae8] + LoopVectorization v0.12.155
  [1914dd2f] + MacroTools v0.5.10
  [d125e4d3] + ManualMemory v0.1.8
  [46d2c3a1] + MuladdMacro v0.2.4
  [6fe1bfb0] + OffsetArrays v1.12.9
  [bac558e1] + OrderedCollections v1.6.0
  [1d0040c9] + PolyesterWeave v0.2.1
  [c74db56a] + PolynomialBases v0.4.17
  [21216c6a] + Preferences v1.3.0
  [3cdcf5f2] + RecipesBase v1.3.3
  [731186ca] + RecursiveArrayTools v2.38.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [7e49a35a] + RuntimeGeneratedFunctions v0.5.6
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.38
  [0bca4576] + SciMLBase v1.91.6
  [c0aeaf25] + SciMLOperators v0.2.0
  [efcf1570] + Setfield v1.1.1
  [ce78b400] + SimpleUnPack v1.1.0
  [66db9d55] + SnoopPrecompile v1.0.3
  [276daf66] + SpecialFunctions v2.2.0
  [aedffcd0] + Static v0.8.6
  [0d7ed370] + StaticArrayInterface v1.3.0
  [90137ffa] + StaticArrays v1.5.19
  [1e83bf80] + StaticArraysCore v1.4.0
  [9f78cca6] + SummationByPartsOperators v0.5.31
  [2efcf032] + SymbolicIndexingInterface v0.2.2
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.10.1
  [8290d209] + ThreadingUtilities v0.5.1
  [410a4b4d] + Tricks v0.1.7
  [781d530d] + TruncatedStacktraces v1.3.0
  [3a884ed6] + UnPack v1.0.2
  [9602ed7d] + Unrolled v0.1.4
  [3d5dd08c] + VectorizationBase v0.21.64
  [f5851436] + FFTW_jll v3.3.10+0
  [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2
  [856f044c] + MKL_jll v2022.2.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics v1.9.0
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [05823500] + OpenLibm_jll v0.8.1+0
  [bea87d4a] + SuiteSparse_jll v5.10.1+6
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850b90] + libblastrampoline_jll v5.4.0+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
Precompiling environment...
  ✗ SummationByPartsOperators
  78 dependencies successfully precompiled in 50 seconds. 7 already precompiled.
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
  1 dependency had warnings during precompilation:
┌ SciMLOperators [c0aeaf25-5076-4817-a8d5-81caf7dfa961]
│  ┌ Warning: Replacing docs for `SciMLOperators.AbstractSciMLOperator :: Union{}` in module `SciMLOperators`
│  └ @ Base.Docs docs/Docs.jl:243
└  

julia> import Pkg; Pkg.precompile()
Precompiling environment...
  ✗ SummationByPartsOperators
  0 dependencies successfully precompiled in 8 seconds. 85 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

SummationByPartsOperators [9f78cca6-572e-554e-b819-917d2f1cf240]

Failed to precompile SummationByPartsOperators [9f78cca6-572e-554e-b819-917d2f1cf240] to "/home/hendrik/.julia/compiled/v1.9/SummationByPartsOperators/jl_0lvrD8".
ERROR: LoadError: MethodError: no method matching convolve_interior_coefficients!(::Vector{Float64}, ::StaticArraysCore.SVector{1, Float64}, ::Float64, ::StaticArraysCore.SVector{1, Float64}, ::Vector{Float64}, ::Float64, ::Int64, ::Int64, ::SummationByPartsOperators.FastMode)

Closest candidates are:
  convolve_interior_coefficients!(::AbstractVector, ::StaticArraysCore.SVector{LowerOffset}, ::Any, ::StaticArraysCore.SVector{UpperOffset}, ::AbstractVector, ::Any, ::Any, ::Static.StaticInt{left_boundary_width}, ::Static.StaticInt{right_boundary_width}, ::Any) where {LowerOffset, UpperOffset, left_boundary_width, right_boundary_width}
   @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/SBP_operators.jl:232
  convolve_interior_coefficients!(::AbstractVector, ::StaticArraysCore.SVector{LowerOffset}, ::Any, ::StaticArraysCore.SVector{UpperOffset}, ::AbstractVector, ::Any, ::Static.StaticInt{left_boundary_width}, ::Static.StaticInt{right_boundary_width}, ::Any) where {LowerOffset, UpperOffset, left_boundary_width, right_boundary_width}
   @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/SBP_operators.jl:274
  convolve_interior_coefficients!(::AbstractVector{<:StaticArraysCore.StaticArray{Tuple{N}, T1, 1}}, ::StaticArraysCore.SVector{LowerOffset}, ::Any, ::StaticArraysCore.SVector{UpperOffset}, ::AbstractVector{<:StaticArraysCore.StaticArray{Tuple{N}, T2, 1}}, ::Any, ::Any, ::Static.StaticInt{left_boundary_width}, ::Static.StaticInt{right_boundary_width}, ::Any) where {LowerOffset, UpperOffset, N, T1, T2, left_boundary_width, right_boundary_width}
   @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/SBP_operators.jl:315
  ...

Stacktrace:
  [1] mul!(dest::Vector{Float64}, coefficients::SummationByPartsOperators.PeriodicDerivativeCoefficients{Float64, 1, 1, SummationByPartsOperators.FastMode, SummationByPartsOperators.Fornberg1998}, u::Vector{Float64}, α::Float64)
    @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/periodic_operators.jl:57
  [2] mul!
    @ ~/.julia/packages/SummationByPartsOperators/Suff3/src/periodic_operators.jl:596 [inlined]
  [3] mul!
    @ ~/.julia/packages/SummationByPartsOperators/Suff3/src/general_operators.jl:138 [inlined]
  [4] *(D::SummationByPartsOperators.PeriodicDerivativeOperator{Float64, 1, 1, SummationByPartsOperators.FastMode, SummationByPartsOperators.Fornberg1998, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, u::Vector{Float64})
    @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/general_operators.jl:147
  [5] macro expansion
    @ ~/.julia/packages/SummationByPartsOperators/Suff3/src/precompile.jl:7 [inlined]
  [6] top-level scope
    @ ~/.julia/packages/SnoopPrecompile/1XXT1/src/SnoopPrecompile.jl:62
  [7] include(mod::Module, _path::String)
    @ Base ./Base.jl:457
  [8] include(x::String)
    @ SummationByPartsOperators ~/.julia/packages/SummationByPartsOperators/Suff3/src/SummationByPartsOperators.jl:44
  [9] top-level scope
    @ ~/.julia/packages/SummationByPartsOperators/Suff3/src/SummationByPartsOperators.jl:172
 [10] include
    @ ./Base.jl:457 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2012
 [12] top-level scope
    @ stdin:2
in expression starting at /home/hendrik/.julia/packages/SummationByPartsOperators/Suff3/src/precompile.jl:3
in expression starting at /home/hendrik/.julia/packages/SummationByPartsOperators/Suff3/src/SummationByPartsOperators.jl:2
in expression starting at stdin:2
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/API.jl:1556
 [3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [4] precompile(pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [5] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/API.jl:171
 [6] precompile()
   @ Pkg.API ~/Software/julia-1.9.0-rc2/share/julia/stdlib/v1.9/Pkg/src/API.jl:162
 [7] top-level scope
   @ REPL[2]:1

The stacktrace shows that the calls to static_length from StaticArrayInterface.jl in line
https://github.com/ranocha/SummationByPartsOperators.jl/blob/f0b37d29e1fd9ae27b7acc97b38adc85a12960cd/src/periodic_operators.jl#LL57C87-L57C112
return an Int instead of the desired StaticInt when given an SVector from StaticArrays.jl as argument.

StaticArrayInterface.jl uses a weak dependency to load the required code when StaticArrays.jl is present, see
https://github.com/JuliaArrays/StaticArrayInterface.jl/blob/7a8988436b82e9cdff2d2e59d5e75d0204d118a9/Project.toml#L27
SummationByPartsOperators.jl requires this code to be loaded to work correctly. In particular, it needs to be loaded at precompile time to precompile workloads and benefit from the massive improvements in Julia v1.9. Based on the stacktrace above, it looks like the weak dependency of StaticArrayInterface.jl is not loaded correctly in Julia 1.9.0-rc2 while everything is fine with Julia 1.9.0-rc1.

I have seen something similar before when using Requires.jl on Julia v1.8, see https://discourse.julialang.org/t/how-to-load-requires-jl-code-of-dependency-while-precompiling/95962.

My best guess is that this change of behavior was introduced by #48674, see

julia/base/loading.jl

Lines 1672 to 1675 in 93df7e2

# Load extensions when not precompiling and not in a nested package load
if JLOptions().incremental == 0 && isempty(package_locks)
run_extension_callbacks()
end

@vtjnash
Copy link
Member

vtjnash commented Apr 4, 2023

The weakdep must be an explicit dependency, or you cannot depend on when it will eventually be loaded

@staticfloat
Copy link
Member

@vtjnash
Copy link
Member

vtjnash commented Apr 4, 2023

The package it depends on is StaticArrayInterfaceStaticArraysExt

@staticfloat
Copy link
Member

But you can't depend directly on an extension package can you?

@ranocha
Copy link
Member Author

ranocha commented Apr 4, 2023

So SummationByPartsOperators.jl needs to depend on the internal implementation detail StaticArrayInterfaceStaticArraysExt included in StaticArrayInterface.jl? How can I do that? (Is this even a good idea given that this is an implementation detail of StaticArrayInterface.jl?)

@KristofferC
Copy link
Member

We stopped loading extensions during precompilation. Putting that back fixes this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants