Test using Julia v1.13 in Yggdrasil (v2)

[IanButterworth]

I can't push to #10524 so opening this

[giordano]

I can't push to #10524

You've done it before.

[IanButterworth]

Odd. I was getting a github perm issue acquiring a push lock or something.

[IanButterworth]

@giordano it seems we're getting build errors that I don't think we saw before. I guess there are other julia issues with nightly getting in the way of testing Pkg out?

[fingolfin]

Why is this changed?

[IanButterworth]

JuliaLang/Pkg.jl#4156

v"3.24.3+0" exists, but v"3.24.3" doesn't. We need to give this as a string to signify its a package spec rather than a fixed version.

[giordano]

@IanButterworth something is failing still.

[IanButterworth]

Yeah a resolver issue on 1.7 and 1.8.

[giordano]

Libunwind for riscv64 in v1.13 can't be found, do we have the wrong version in the recipe or Pkg is resolving it incorrectly?

[IanButterworth]

Libunwind for riscv64 in v1.13 can't be found, do we have the wrong version in the recipe or Pkg is resolving it incorrectly?

HistoricalStdlibVersions probably is out of date

[IanButterworth]

Related issue JuliaLang/Pkg.jl#4335

The libjulia script specifies a stdlib versions specifically, which is basically invalid unless we're resolving with ctx.julia_version=nothing, which we don't seem to be doing.. and it resulted in a different version being installed because HistoricalStdlibVersions was out of date (now updated).

[IanButterworth]

1.7.0 is erroring with

ERROR: LoadError: Unsatisfiable requirements detected for package GMP_jll [781609d7]:
 GMP_jll [781609d7] log:
 ├─possible versions are: 6.2.1 or uninstalled
 ├─restricted to versions 6.2.0 by MPFR_jll [3a97d323] — no versions left
 │ └─MPFR_jll [3a97d323] log:
 │   ├─possible versions are: 4.1.1 or uninstalled
 │   └─MPFR_jll [3a97d323] is fixed to version 4.1.1+1
 └─GMP_jll [781609d7] is fixed to version 6.2.1+1

• The version of the GMP_jll stdlib in 1.7.0 is 6.2.1
• The version of the MPFR_jll stdlib in 1.7.0 is 4.1.1+1, and it doesn't set any compat on GMP_jll https://github.com/JuliaLang/julia/blob/3bf9d1773144bc4943232dc2ffaac307a700853d/stdlib/MPFR_jll/Project.toml
• BUT the MPFR_jll repo (which BB is fetching) for v4.1.1+1 does set compat on GMP_jll of =6.2.0 https://github.com/JuliaBinaryWrappers/MPFR_jll.jl/blob/ba64d135673f0864f125f5990abaadb01a513a41/Project.toml#L15

Basically 1.7 (and 1.8) are invalid in ways that are probably too complicated to fix.

@benlorenz is going to look into whether we can drop 1.7 and 1.8 builds for LibJulia

[IanButterworth]

This is basically complete. The only thing left to test is nightly once JuliaLang/julia#59093 has nightlies out, which will error if Pkg isn't going to install the stdlib version you're actually requesting

[benlorenz]

Packages depending on libjulia:

I had a look at packages including libjulia/common.jl that were touched since 1.10 was announced as LTS version (Oct 2024):

Packages which don't support 1.7 or 1.8 anyway:

• legate_jl_wrapper: >= 1.10
• LEMON: >= 1.9
• slope: >= 1.10
• Pythia8_cxxwrap: >= 1.10
• Geant4_julia: >= 1.10
• OpenFHE: >= 1.10
• blockSQP: >= 1.10
• ITKIOWrapper: >= 1.10
• jinja2cppwrapper: >= 1.9
• libpolymake_julia: >= 1.10

Binaries which do support 1.7 or 1.8 but the corresponding julia package doesn't:

• DACE: >= 1.6 -- DACE.jl >= 1.10
• gecko: >= 1.6 -- Gecko.jl >= 1.10
• Minuit2_Julia_Wrapper: >= 1.6 -- Minuit2.jl >= 1.10
• SDPA: >= 1.6 -- SDPA.jl >= 1.10
• OpenCV: >= 1.6 -- OpenCV.jl >= 1.10
• GAP: >= 1.6 -- GAP.jl dev version: >= 1.10
• libsingular_julia: >= 1.6 (Singular.jl allows 1.6 but Oscar.jl switched to 1.10)

Binaries which do still support 1.7 or 1.8:

• z3: >= 1.6
• ROOT_julia: >= 1.6
• XDiag: >= 1.7
• Spot_julia: >= 1.6
• quickfix: >= 1.8
• openPMD_api: >= 1.7
• Powsybl: >= 1.6
• AlphaMolWrapper: >= 1.7
• algoimWrapper: >= 1.6
• XRootD_cxxwrap_jll: >= 1.6
• libcxxwrap_julia: >= 1.6

These are quite a few more than I expected (and I might have missed some?).

MPFR Compat

Regarding the broken compat for MPFR / GMP, that entry was in fact retroactively removed in the registry to fix some build issues here on Yggdrasil:
JuliaRegistries/General#102652
But the compat entries in the repo itself were until now considered irrelevant.

I checked the most recent libjulia_jll update to figure out how it worked previously, and in fact it didn't: The build log shows that the build for julia 1.7.0 used MPFR_jll 4.2.2+0:
https://buildkite.com/julialang/yggdrasil/builds/21932/steps/canvas?sid=019807bd-b5df-4fd7-8f14-2edb490d1930#019807bd-b6be-4e2b-8d34-cdd312196475/695-2827
For whatever reason this does not cause any issues ... Maybe because most downstream packages that depend on MPFR_jll do specify 4.1.1.

Workaround

Similar to another (bad) workaround that we currently use we could trick Pkg into allowing us to install a slightly different MPFR_jll version for the 1.7 build (and probably 1.8 as well). With something like this:

        # we need to trick Pkg into allowing us to install a slightly different mpfr version due
        # to a broken compat entry in the git repo for MPFR_jll 4.1.1+1
        filter!(p -> BinaryBuilder.getname(p) != "MPFR_jll", dependencies),
        delete!(Pkg.Types.get_last_stdlibs(v"1.7.0"), Base.UUID("3a97d323-0669-5f0c-9066-3539efd106a3"))
        push!(dependencies, BuildDependency(get_addable_spec("MPFR_jll", v"4.1.1+2")))

Dropping old versions

In the end I still think we should drop these old versions, there was a previous discussion regarding this here: #9788 (comment)
Julia 1.12 is almost ready and maybe this is a "real pressing reason"?

And if someone still needs an older version for some rebuild, these libjulia_jll versions are still available. The only thing that will not be possible anymore at some point is to use the latest 1.13 version.

[giordano]

Rebased on top of #12226 to test JuliaLang/Pkg.jl#4403

[giordano]

• good news: building libjulia v1.13 generally works, so Test using Julia v1.12 #12226 (comment) may have indeed be fixed by Make julia compat checking respect julia_version JuliaLang/Pkg.jl#4403
• wrong version of libunwind is still installed: Test using Julia v1.12 #12226 (comment). despite the fact now we're using latest version of HistoricalStdlibVersions:

  Yggdrasil/.ci/Manifest.toml

  Lines 185 to 189 in a4056e8

  	[[deps.HistoricalStdlibVersions]]
  	deps = ["Pkg", "PrecompileTools"]
  	git-tree-sha1 = "3a167a57ca85255f532ae75bd3f0fc2062ae58c4"
  	uuid = "6df8b67a-e8a0-4029-b4b7-ac196fe72102"
  	version = "2.0.5"

  Where's this version coming from then?

[IanButterworth]

Needs a rebase to run CI after #12273

[giordano]

https://buildkite.com/julialang/yggdrasil/builds/24394#019a11cd-2407-4f8f-8357-7206f607aaab/1273-1274

ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("399d05d3-28fc-5cf5-932b-b3ee5b53232d"), "CompatLinearAlgebraExt") not available with flags CacheFlags(; use_pkgimages=true, debug_level=1, check_bounds=0, inline=true, opt_level=2)
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:44
  [2] __require_prelocked(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2702
  [3] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2561
  [4] _require_prelocked
    @ ./loading.jl:2555 [inlined]
  [5] run_extension_callbacks(extid::Base.ExtensionId)
    @ Base ./loading.jl:1607
  [6] run_extension_callbacks(pkgid::Base.PkgId)
    @ Base ./loading.jl:1644
  [7] run_package_callbacks(modkey::Base.PkgId)
    @ Base ./loading.jl:1460
  [8] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2569
  [9] macro expansion
    @ ./loading.jl:2489 [inlined]
 [10] macro expansion
    @ ./lock.jl:376 [inlined]
 [11] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2453
 [12] require
    @ ./loading.jl:2429 [inlined]
 [13] eval_import_path
    @ ./module.jl:36 [inlined]
 [14] eval_import_path_all(at::Module, path::Expr, keyword::String)
    @ Base ./module.jl:60
 [15] _eval_import(::Bool, ::Module, ::Expr, ::Expr, ::Vararg{Expr})
    @ Base ./module.jl:101
 [16] top-level scope
    @ /cache/julia-buildkite-plugin/depots/e2fd9734-29d8-45cd-b0eb-59f7104f3131/packages/Mocking/eug7a/src/Mocking.jl:3
 [17] include(mod::Module, _path::String)
    @ Base ./Base.jl:309
 [18] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:3133
 [19] top-level scope
    @ stdin:5
 [20] eval(m::Module, e::Any)
    @ Core ./boot.jl:489
 [21] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2979
 [22] include_string
    @ ./loading.jl:2989 [inlined]
 [23] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:342
 [24] _start()
    @ Base ./client.jl:577
in expression starting at /cache/julia-buildkite-plugin/depots/e2fd9734-29d8-45cd-b0eb-59f7104f3131/packages/Mocking/eug7a/src/Mocking.jl:1
in expression starting at stdin:5

[giordano]

Probably needs to wait for x86_64-linux-gnu tests in https://buildkite.com/julialang/julia-master/builds/51786 to finish successfully, so that the nightly build which includes the fix for JuliaLang/julia#59956 will actually be available.