Buggy precompilation of init-defined symbols can break CUDA_Driver_jll initialization

[simonbyrne]

Describe the bug

I'm using CUDA.jl on a HPC cluster, configured to use the system CUDA installation via Preferences.jl.

I've then precompiled on one node, but loaded it on another node, I occasionally get the following error:

ERROR: LoadError: InitError: UndefVarError: libcuda not defined
Stacktrace:
  [1] getproperty
    @ ./Base.jl:31 [inlined]
  [2] __init__()
    @ CUDA /central/scratch/esm/slurm-buildkite/climacore-ci/1859/depot/default/packages/CUDA/ZdCxS/src/initialization.jl:42
  [3] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
  [4] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, sourcepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:938
  [5] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base ./loading.jl:1028
  [6] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1315
  [7] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [8] macro expansion
    @ ./loading.jl:1180 [inlined]
  [9] macro expansion
    @ ./lock.jl:223 [inlined]
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
during initialization of module CUDA

e.g. https://buildkite.com/clima/climacore-ci/builds/1859#0186dc49-e196-4714-9bd2-ac2f4ad07ac9/137-139

If I load it at the REPL, I get isdefined(CUDA_Driver_jll, :libcuda) == false.

It goes away if I delete the .ji file for CUDA_Driver_jll and reload.

To reproduce

I'm still not sure exactly how to reproduce it, will see if I can figure it out.

[simonbyrne]

I'm also seeing it without the CUDA_Runtime_jll preferences set.

[vchuravy]

So CUDA_Driver_jll initializes the cuda variable during init:

https://github.com/JuliaBinaryWrappers/CUDA_Driver_jll.jl/blob/787e6222856f1068cf69495445d11251c8123b91/src/wrappers/aarch64-linux-gnu.jl#L25

[vchuravy]

If you can reproduce it with JULIA_DEBUG="CUDA_Driver_jll" that would be great.

[simonbyrne]

When re-running it:

julia> using CUDA
┌ Debug: No system CUDA driver found
└ @ CUDA_Driver_jll /central/scratch/esm/slurm-buildkite/climacore-ci/1862/depot/default/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:54
ERROR: InitError: UndefVarError: libcuda not defined
Stacktrace:
 [1] getproperty
   @ ./Base.jl:31 [inlined]
 [2] __init__()
   @ CUDA /central/scratch/esm/slurm-buildkite/climacore-ci/1862/depot/default/packages/CUDA/ZdCxS/src/initialization.jl:42
 [3] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
   @ Base ./loading.jl:831
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
   @ Base ./loading.jl:1039
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1315
 [6] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [7] macro expansion
   @ ./loading.jl:1180 [inlined]
 [8] macro expansion
   @ ./lock.jl:223 [inlined]
 [9] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
during initialization of module CUDA

Would it be more robust to define it initially to make it always defined? i.e. change the above linked line to

global libcuda = nothing

[maleadt]

There's an isdefined check in there so I'm not sure why the getproperty fails:

CUDA.jl/src/initialization.jl

Lines 38 to 47 in 940d23d

	# check that we have a driver
	global libcuda
	if CUDA_Driver_jll.is_available()
	if isdefined(CUDA_Driver_jll, :libcuda)
	libcuda = CUDA_Driver_jll.libcuda
	else
	_initialization_error[] = "CUDA driver not found"
	return
	end
	else

[simonbyrne]

I've managed to reliably recreate it, using the default configuration (i.e. it is unrelated to using the system CUDA runtime).

1. Instantiate and precompile on a node with a GPU:

┌─[4]──[Tue Mar 14]──[10:07:53]────────────────────────────────────────
│ spjbyrne@hpc-21-18:~/misc/cudax
├ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(cudax) pkg> instantiate
Precompiling project...
  2 dependencies successfully precompiled in 51 seconds. 35 already precompiled.

julia> using CUDA

julia> using CUDA_Driver_jll

julia> CUDA_Driver_jll.is_available()
true

julia> CUDA_Driver_jll.libcuda
"/home/spjbyrne/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so"

2. On a node without a GPU, attempt to load CUDA.jl

┌─[6]──[Tue Mar 14]──[10:16:23]────────────────────────────────────────
│ spjbyrne@hpc-21-14:~/misc/cudax
├ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CUDA
ERROR: InitError: UndefVarError: libcuda not defined
Stacktrace:
 [1] getproperty
   @ ./Base.jl:31 [inlined]
 [2] __init__()
   @ CUDA ~/.julia/packages/CUDA/ZdCxS/src/initialization.jl:42
 [3] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
   @ Base ./loading.jl:831
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
   @ Base ./loading.jl:1039
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1315
 [6] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [7] macro expansion
   @ ./loading.jl:1180 [inlined]
 [8] macro expansion
   @ ./lock.jl:223 [inlined]
 [9] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
during initialization of module CUDA

julia> using CUDA_Driver_jll

julia> CUDA_Driver_jll.is_available()
true

julia> CUDA_Driver_jll.libcuda
ERROR: UndefVarError: libcuda not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:31
 [2] top-level scope
   @ REPL[4]:1

julia> isdefined(CUDA_Driver_jll, :libcuda)
false

My guess is that the isdefined(CUDA_Driver_jll, :libcuda) is incorrectly assumed to be constant during precompilation?

Manifest.toml

```

This file is machine-generated - editing it directly is not advised

julia_version = "1.8.5"
manifest_format = "2.0"
project_hash = "f83b3bde5f4d0bed77c6bdc54f3a7ba41f053ce0"

[[deps.AbstractFFTs]]
deps = ["ChainRulesCore", "LinearAlgebra"]
git-tree-sha1 = "16b6dbc4cf7caee4e1e75c49485ec67b667098a0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.3.1"

[[deps.Adapt]]
deps = ["LinearAlgebra", "Requires"]
git-tree-sha1 = "cc37d689f599e8df4f464b2fa3870ff7db7492ef"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.6.1"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
version = "1.1.1"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.BFloat16s]]
deps = ["LinearAlgebra", "Printf", "Random", "Test"]
git-tree-sha1 = "dbf84058d0a8cbbadee18d25cf606934b22d7c66"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.4.2"

[[deps.Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[deps.CEnum]]
git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.2"

[[deps.CUDA]]
deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CUDA_Driver_jll", "CUDA_Runtime_Discovery", "CUDA_Runtime_jll", "CompilerSupportLibraries_jll", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "Preferences", "Printf", "Random", "Random123", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions"]
git-tree-sha1 = "edff14c60784c8f7191a62a23b15a421185bc8a8"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "4.0.1"

[[deps.CUDA_Driver_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "75d7896d1ec079ef10d3aee8f3668c11354c03a1"
uuid = "4ee394cb-3365-5eb0-8335-949819d2adfc"
version = "0.2.0+0"

[[deps.CUDA_Runtime_Discovery]]
deps = ["Libdl"]
git-tree-sha1 = "58dd8ec29f54f08c04b052d2c2fa6760b4f4b3a4"
uuid = "1af6417a-86b4-443c-805f-a4643ffb695f"
version = "0.1.1"

[[deps.CUDA_Runtime_jll]]
deps = ["Artifacts", "CUDA_Driver_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg", "TOML"]
git-tree-sha1 = "d3e6ccd30f84936c1a3a53d622d85d7d3f9b9486"
uuid = "76a88914-d11a-5bdc-97e0-2f5a05c973a2"
version = "0.2.3+2"

[[deps.ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "c6d890a52d2c4d55d326439580c3b8d0875a77d9"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.15.7"

[[deps.ChangesOfVariables]]
deps = ["ChainRulesCore", "LinearAlgebra", "Test"]
git-tree-sha1 = "485193efd2176b88e6622a39a246f8c5b600e74e"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.6"

[[deps.Compat]]
deps = ["Dates", "LinearAlgebra", "UUIDs"]
git-tree-sha1 = "7a60c856b9fa189eb34f5f8a6f6b5529b7942957"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "4.6.1"

[[deps.CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"
version = "1.0.1+0"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.9.3"

[[deps.Downloads]]
deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
version = "1.6.0"

[[deps.ExprTools]]
git-tree-sha1 = "c1d06d129da9f55715c6c212866f5b1bddc5fa00"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.9"

[[deps.FileWatching]]
uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"

[[deps.GPUArrays]]
deps = ["Adapt", "GPUArraysCore", "LLVM", "LinearAlgebra", "Printf", "Random", "Reexport", "Serialization", "Statistics"]
git-tree-sha1 = "a28f752ffab0ccd6660fc7af5ad1c9ad176f45f7"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "8.6.3"

[[deps.GPUArraysCore]]
deps = ["Adapt"]
git-tree-sha1 = "1cd7f0af1aa58abc02ea1d872953a97359cb87fa"
uuid = "46192b85-c4d5-4398-a991-12ede77f4527"
version = "0.1.4"

[[deps.GPUCompiler]]
deps = ["ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "19d693666a304e8c371798f4900f7435558c7cde"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.17.3"

[[deps.InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[deps.InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "49510dfcb407e572524ba94aeae2fced1f3feb0f"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.8"

[[deps.IrrationalConstants]]
git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.2.2"

[[deps.JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "abc9885a7ca2052a736a600f7fa66209f96506e1"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.4.1"

[[deps.LLVM]]
deps = ["CEnum", "LLVMExtra_jll", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "df115c31f5c163697eede495918d8e85045c8f04"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "4.16.0"

[[deps.LLVMExtra_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg", "TOML"]
git-tree-sha1 = "7718cf44439c676bc0ec66a87099f41015a522d6"
uuid = "dad2f222-ce93-54a1-a47d-0025e8a3acab"
version = "0.0.16+2"

[[deps.LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"
version = "0.6.3"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"
version = "7.84.0+0"

[[deps.LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"
version = "1.10.2+0"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.LinearAlgebra]]
deps = ["Libdl", "libblastrampoline_jll"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[deps.LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "0a1b7c2863e44523180fdb3146534e265a91870b"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.23"

[[deps.Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[deps.Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.28.0+0"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"
version = "2022.2.1"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"
version = "1.2.0"

[[deps.OpenBLAS_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"]
uuid = "4536629a-c528-5b80-bd46-f80d51c5b363"
version = "0.3.20+0"

[[deps.OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"
version = "0.8.1+0"

[[deps.OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.8.0"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.3.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.Random123]]
deps = ["Random", "RandomNumbers"]
git-tree-sha1 = "7a1a306b72cfa60634f03a911405f4e64d1b718b"
uuid = "74087812-796a-5b5d-8853-05524746bad3"
version = "1.6.0"

[[deps.RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "043da614cc7e95c703498a491e2c21f58a2b8111"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.5.3"

[[deps.Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[deps.Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.3.0"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[deps.SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[deps.SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "ef28127915f4229c971eb43f3fc075dd3fe91880"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "2.2.0"

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.0"

[[deps.Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
version = "1.10.1"

[[deps.Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[deps.TimerOutputs]]
deps = ["ExprTools", "Printf"]
git-tree-sha1 = "f2fd3f288dfc6f507b0c3a2eb3bac009251e548b"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.22"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"
version = "1.2.12+3"

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"
version = "5.1.1+0"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"
version = "1.48.0+0"

[[deps.p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
version = "17.4.0+0"

</p>

[maleadt]

My guess is that the isdefined(CUDA_Driver_jll, :libcuda) is incorrectly assumed to be constant during precompilation?

That seems like a serious Julia bug then?

[maleadt]

@staticfloat Have you seen something similar with other JLLs (as I assume isdefined(Whatever_jll, :libwhatever) is a relatively common pattern given how JLLs behave around missing products)? It's really puzzling that basically isdefined(Foo, :bar) && getproperty(Foo, :bar) can throw during getproperty...

[staticfloat]

No, I've never seen that. It does sound like a Julia bug. @simonbyrne can you show the values of best_wrapper and host_platform inside of the JLL?

[maleadt]

Note that we have a fairly complicated __init__ as part of CUDA_Driver_jll that is responsible for setting libcuda, https://github.com/JuliaBinaryWrappers/CUDA_Driver_jll.jl/blob/787e6222856f1068cf69495445d11251c8123b91/src/wrappers/x86_64-linux-gnu.jl#L23-L170, but that shouldn't result in isdefined(CUDA_Driver_jll, :libcuda) erroneously returning true. Or even if it did, AFAIU that shouldn't bake in the result and just have CUDA.jl's __init__ perform a run-time check during its __init__. Of course, __init__ is special in so many ways that my understand may be off here.

[simonbyrne]

I have a simpler reproducer, will open an issue in Julia

[simonbyrne]

@staticfloat you're right, it's a Julia bug (see JuliaLang/julia#48999). Let me know if you have any suggestions for a workaround. As it stands, it breaks our internal CI, so we've had to downgrade to CUDA.jl 3.

[staticfloat]

I'm not sure how to work around this without changing the API. :(

[maleadt]

We could use try/catch instead if if isdefined.

[simonbyrne]

We could use try/catch instead if if isdefined.

That seems to work on Julia 1.8.5, but not 1.9-rc1 (it gives a segfault). It's still an improvement though

[simonbyrne]

As the discussion on JuliaLang/julia#48999 suggests that it won't be fixed any time soon, I propose that we initialize all the globals to nothing:

• [CUDA_Driver] Initialize globals to nothing JuliaPackaging/Yggdrasil#6398
• Initialize globals to nothing #1809

[simonbyrne]

@maleadt can I do anything to help with this?

[Sixzero]

I have a similar issue when precompiling (my MVP):

module X
using CUDA
CUDA.seed!(42)
end

I don't know if other lines of my module affects this, but as for now I won't copy them over.

When I run this line, things work well, it only fails in precompile I think. (Hope these help.)

[maleadt]

This was fixed by #1832

[aasgherali]

I am having the same issue. When I try to update the packages, it breaks. I am using Julia 1.8.5 on Arch through juliaup. I get the following error

ERROR: LoadError: UndefVarError: libcuda not defined

Is there any workaround?

Thanks

[maleadt]

You'll have to provide more details, e.g., a Manifest, full back-trace, etc. There shouldn't be any use of CUDA.jl during precompilation, so it's not clear to me how this can happen while updating packages.

[aasgherali]

_
_       _ _(_)_     |  Documentation: https://docs.julialang.org
(_)     | (_) (_)    |
_ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` |  |
| | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
_/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> update
Updating registry at `~/.julia/registries/General.toml`
ERROR: LoadError: UndefVarError: libcuda not defined
Stacktrace:
[1] cuda_toolkit_tag()
@ Main ~/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg/platform_augmentation.jl:50
[2] augment_platform!(platform::Platform)
@ Main ~/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg/platform_augmentation.jl:137
[3] top-level scope
@ ~/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg/select_artifacts.jl:11
[4] include(fname::String)
@ Base.MainInclude ./client.jl:476
[5] top-level scope
@ none:5
in expression starting at /home/asgher/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg/select_artifacts.jl:11
ERROR: failed process: Process(`/home/asgher/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/bin/julia -Cnative -J/home/asgher/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/asgher/.julia/environments/v1.8/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/asgher/.julia", "/home/asgher/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/local/share/julia", "/home/asgher/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/asgher/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg")
include("/home/asgher/.julia/packages/CUDA_Runtime_jll/Nu66l/.pkg/select_artifacts.jl")
' --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.8.5`, ProcessExited(1)) [1]

Stacktrace:
[1] pipeline_error
@ ./process.jl:565 [inlined]
[2] read(cmd::Cmd)
@ Base ./process.jl:449
[3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
@ Pkg.Operations ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:630
[4] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
@ Pkg.Operations ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:662
[5] up(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, level::Pkg.Types.UpgradeLevel; skip_writing_project::Bool)
@ Pkg.Operations ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1367
[6] up(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, skip_writing_project::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/API.jl:341
[7] up(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
[8] up(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/API.jl:145
[9] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
@ Pkg.REPLMode ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:409
[10] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
@ Pkg.REPLMode ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:387
[11] do_cmd
@ ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
[12] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
@ Pkg.REPLMode ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:551
[13] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[14] invokelatest
@ ./essentials.jl:726 [inlined]
[15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/REPL/src/LineEdit.jl:2510
[16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL ~/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/share/julia/stdlib/v1.8/REPL/src/REPL.jl:1248
[17] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL ./task.jl:484

(@v1.8) pkg>

[maleadt]

Can you also show ]st -m?

[aasgherali]

Here is the outpu

_
_       _ _(_)_     |  Documentation: https://docs.julialang.org
(_)     | (_) (_)    |
_ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` |  |
| | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
_/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> st -m
Status `~/.julia/environments/v1.8/Manifest.toml`
[14f7f29c] AMD v0.5.0
⌅ [c3fe647b] AbstractAlgebra v0.27.10
[621f4979] AbstractFFTs v1.3.1
[398f06c4] AbstractLattices v0.2.1
[1520ce14] AbstractTrees v0.4.4
[7d9f7c33] Accessors v0.1.28
[79e6a3ab] Adapt v3.6.1
[fb37089c] Arblib v0.8.1
[dce04be8] ArgCheck v2.3.0
[ec485272] ArnoldiMethod v0.2.0
⌃ [4fba245c] ArrayInterface v7.4.1
[30b0a656] ArrayInterfaceCore v0.1.29
⌅ [4c555306] ArrayLayouts v0.8.18
[a9b6321e] Atomix v0.1.0
[15f4f7f2] AutoHashEquals v0.2.0
[ab4f0b2a] BFloat16s v0.4.2
⌃ [aae01518] BandedMatrices v0.17.17
[198e06fe] BangBang v0.3.37
[9718e550] Baselet v0.1.1
[6e4b80f9] BenchmarkTools v1.3.2
[e2ed5e7c] Bijections v0.1.4
[d1d4a3ce] BitFlags v0.1.7
[62783981] BitTwiddlingConvenienceFunctions v0.1.5
[764a87c0] BoundaryValueDiffEq v2.11.0
[fa961155] CEnum v0.4.2
[2a0fbf3d] CPUSummary v0.2.2
[00ebfdb7] CSTParser v3.3.6
[336ed68f] CSV v0.10.9
⌃ [052768ef] CUDA v4.1.1
⌅ [1af6417a] CUDA_Runtime_Discovery v0.1.2
[49dc2e85] Calculus v0.5.1
[082447d4] ChainRules v1.48.0
[d360d2e6] ChainRulesCore v1.15.7
[9e997f8a] ChangesOfVariables v0.1.6
[fb6a15b2] CloseOpenIntervals v0.1.12
[523fee87] CodecBzip2 v0.7.2
[944b1d66] CodecZlib v0.7.1
[35d6a980] ColorSchemes v3.20.0
[3da002f7] ColorTypes v0.11.4
[c3611d14] ColorVectorSpace v0.9.10
[5ae59095] Colors v0.12.10
[861a8166] Combinatorics v1.0.2
[3709ef60] CommonEq v0.2.0
⌃ [a80b9123] CommonMark v0.8.10
[38540f10] CommonSolve v0.2.3
[bbf7d656] CommonSubexpressions v0.3.0
[34da2185] Compat v4.6.1
⌃ [b0b7db55] ComponentArrays v0.13.8
[b152e2b5] CompositeTypes v0.1.3
[a33af91c] CompositionsBase v0.1.1
[8f4d0f93] Conda v1.8.0
[5218b696] Configurations v0.17.4
[88cd18e8] ConsoleProgressMonitor v0.1.2
[187b0558] ConstructionBase v1.5.1
[6add18c4] ContextVariablesX v0.1.3
[d38c429a] Contour v0.6.2
[f65535da] Convex v0.15.3
[adafc99b] CpuId v0.3.1
[a8cc5b0e] Crayons v4.1.1
[667455a9] Cubature v1.5.1
[9a962f9c] DataAPI v1.14.0
[a93c6f00] DataFrames v1.5.0
[864edb3b] DataStructures v0.18.13
[e2d170a0] DataValueInterfaces v1.0.0
[244e2a9f] DefineSingletons v0.1.2
[bcd4f6db] DelayDiffEq v5.42.0
[b429d917] DensityInterface v0.4.0
⌃ [2b5f629d] DiffEqBase v6.122.1
[459566f4] DiffEqCallbacks v2.26.1
[77a26b50] DiffEqNoiseProcess v5.16.0
[163ba53b] DiffResults v1.1.0
[b552c78f] DiffRules v1.13.0
[0c46a032] DifferentialEquations v7.7.0
[b4f34e82] Distances v0.10.8
⌃ [31c24e10] Distributions v0.25.86
[ffbed154] DocStringExtensions v0.9.3
⌅ [5b8099bc] DomainSets v0.5.15
[fa6b7ba4] DualNumbers v0.6.8
[7c1d4256] DynamicPolynomials v0.4.6
[fdbdab4c] ElasticArrays v1.2.11
[4e289a0a] EnumX v1.0.4
[d4d017d3] ExponentialUtilities v1.24.0
[e2ba6199] ExprTools v0.1.9
⌅ [55351af7] ExproniconLite v0.7.11
[c87230d0] FFMPEG v0.4.1
[7a1cc6ca] FFTW v1.6.0
[cc61a311] FLoops v0.2.1
[b9860ae5] FLoopsBase v0.1.1
[7034ab61] FastBroadcast v0.2.5
[9aa1b823] FastClosures v0.3.2
[29a986be] FastLapackInterface v1.2.9
[48062228] FilePathsBase v0.9.20
⌅ [1a297f60] FillArrays v0.13.11
[6a86dc24] FiniteDiff v2.19.0
[53c48c17] FixedPointNumbers v0.8.4
[587475ba] Flux v0.13.14
[9c68100b] FoldsThreads v0.1.1
[59287772] Formatting v0.4.2
[f6369f11] ForwardDiff v0.10.35
[069b7b12] FunctionWrappers v1.1.3
[77dc65aa] FunctionWrappersWrappers v0.1.3
[d9f16b24] Functors v0.4.4
[fb4132e2] FuzzyCompletions v0.5.1
⌃ [0c68f7d7] GPUArrays v8.6.5
[46192b85] GPUArraysCore v0.1.4
⌅ [61eb1bfa] GPUCompiler v0.18.0
⌅ [28b8d3ca] GR v0.71.8
[c145ed77] GenericSchur v0.5.3
[c27321d9] Glob v1.3.1
[86223c79] Graphs v1.8.0
[42e2da0e] Grisu v1.0.2
⌅ [0b43b601] Groebner v0.2.11
[d5909c97] GroupsCore v0.4.0
[19dc6840] HCubature v1.5.1
[cd3eb016] HTTP v1.7.4
⌃ [f213a82b] HomotopyContinuation v2.8.1
[3e5b6fbb] HostCPUFeatures v0.1.14
⌃ [34004b35] HypergeometricFunctions v0.3.11
[ac1192a8] HypertextLiteral v0.9.4
[7073ff75] IJulia v1.24.0
[7869d1d1] IRTools v0.4.9
[615f187c] IfElse v0.1.1
[d25df0c9] Inflate v0.1.3
[83e8ac13] IniFile v0.5.1
[22cec73e] InitialValues v0.3.1
[842dd82b] InlineStrings v1.4.0
[18e54dd8] IntegerMathUtils v0.1.0
[de52edbc] Integrals v3.7.0
[c31f79ba] IntegralsCubature v0.2.2
⌅ [8197267c] IntervalSets v0.7.3
[3587e190] InverseFunctions v0.1.8
[41ab1584] InvertedIndices v1.3.0
[92d709cd] IrrationalConstants v0.2.2
[c8e1da08] IterTools v1.4.0
[42fd0dbc] IterativeSolvers v0.9.2
[82899510] IteratorInterfaceExtensions v1.0.0
[1019f520] JLFzf v0.1.5
[692b3bcd] JLLWrappers v1.4.1
⌃ [682c06a0] JSON v0.21.3
[98e50ef6] JuliaFormatter v1.0.25
[b14d175d] JuliaVariables v0.2.4
⌃ [ccbc3e58] JumpProcesses v9.6.0
[ef3ab10e] KLU v0.4.0
⌃ [63c18a36] KernelAbstractions v0.9.1
[ba0b0d4f] Krylov v0.9.0
[0b1a1467] KrylovKit v0.6.0
[40e66cde] LDLFactorizations v0.10.0
⌅ [929cbde3] LLVM v4.17.1
⌃ [8ac3fa9e] LRUCache v1.4.0
[b964fa9f] LaTeXStrings v1.3.0
[2ee39098] LabelledArrays v1.14.0
[984bce1d] LambertW v0.4.6
[23fbe1c1] Latexify v0.15.18
[a5e1c1ea] LatinHypercubeSampling v1.8.0
[73f95e8e] LatticeRules v0.0.1
[10f19ff3] LayoutPointers v0.1.14
[0e77f7df] LazilyInitializedFields v1.2.1
[50d2b5c4] Lazy v0.15.1
[1d6d02ad] LeftChildRightSiblingTrees v0.2.0
[2d8b4e74] LevyArea v1.0.0
[9c8b4983] LightXML v0.9.0
[d3d80556] LineSearches v7.2.0
⌃ [9b3f67b0] LinearAlgebraX v0.1.11
⌃ [7ed4a6bd] LinearSolve v1.41.0
[2ab3a3ac] LogExpFunctions v0.3.23
[e6f89c97] LoggingExtras v1.0.0
⌃ [bdcacae8] LoopVectorization v0.12.155
⌃ [b2108857] Lux v0.4.48
[d0bbae9a] LuxCUDA v0.1.2
[bb33d45b] LuxCore v0.1.3
[82251201] LuxLib v0.1.13
[6c6e2e6c] MIMEs v0.1.4
[d8e11817] MLStyle v0.4.17
[f1d291b0] MLUtils v0.4.1
[1914dd2f] MacroTools v0.5.10
[20f20a25] MakieCore v0.6.3
[d125e4d3] ManualMemory v0.1.8
⌃ [b8f27783] MathOptInterface v1.13.2
[739be429] MbedTLS v1.1.7
[442fdcdd] Measures v0.3.2
⌅ [e9d8d322] Metatheory v1.3.5
[128add7d] MicroCollections v0.1.4
[e1d29d7a] Missings v1.1.0
[291d046c] MixedSubdivisions v1.1.2
⌃ [961ee093] ModelingToolkit v8.46.1
[7475f97c] Mods v1.3.3
⌅ [4886b29c] MonteCarloIntegration v0.0.3
[99f44e22] MsgPack v1.2.0
[46d2c3a1] MuladdMacro v0.2.4
[3b2b4ff1] Multisets v0.4.4
[102ac46a] MultivariatePolynomials v0.4.7
[d8a4904e] MutableArithmetics v1.2.3
[d41bc354] NLSolversBase v7.8.3
[2774e3e8] NLsolve v4.5.1
[872c559c] NNlib v0.8.19
[a00861dc] NNlibCUDA v0.2.7
[77ba4419] NaNMath v1.0.2
[71a1bf82] NameResolution v0.1.5
⌃ [315f7962] NeuralPDE v5.5.0
⌃ [8913a72c] NonlinearSolve v1.5.1
[510215fc] Observables v0.5.4
[6fe1bfb0] OffsetArrays v1.12.9
[0b1bfda6] OneHotArrays v0.2.3
⌃ [4d8831e6] OpenSSL v1.3.3
⌃ [429524aa] Optim v1.7.4
⌃ [3bd65402] Optimisers v0.2.17
⌃ [7f7a1694] Optimization v3.13.0
⌃ [36348300] OptimizationOptimJL v0.1.5
⌃ [bac558e1] OrderedCollections v1.5.0
⌃ [1dea7af3] OrdinaryDiffEq v6.49.4
[90014a1f] PDMats v0.11.17
[d96e819e] Parameters v0.12.3
[69de0a69] Parsers v2.5.8
⌃ [2ae35dd2] Permutations v0.4.15
[b98c9c47] Pipe v1.3.0
[ccf2f8ad] PlotThemes v3.1.0
[995b91a9] PlotUtils v1.3.4
⌃ [91a5bcdd] Plots v1.38.8
⌃ [c3e4b0f8] Pluto v0.19.22
[e409e4f3] PoissonRandom v0.4.4
[f517fe37] Polyester v0.7.3
[1d0040c9] PolyesterWeave v0.2.1
⌃ [f27b6e38] Polynomials v3.2.8
[2dfb63ee] PooledArrays v1.4.2
[85a6dd25] PositiveFactorizations v0.2.4
[d236fae5] PreallocationTools v0.4.12
[91cefc8d] PrecompileSignatures v3.0.3
[21216c6a] Preferences v1.3.0
[8162dcfd] PrettyPrint v0.2.0
[08abe8d2] PrettyTables v2.2.3
[27ebfcd6] Primes v0.5.3
[33c8b6b6] ProgressLogging v0.1.4
[92933f4c] ProgressMeter v1.7.2
[01f381cc] ProjectiveVectors v1.1.3
[438e738f] PyCall v1.95.1
[1fd47b50] QuadGK v2.8.2
[8a4e6c94] QuasiMonteCarlo v0.2.19
[74087812] Random123 v1.6.0
[fb686558] RandomExtensions v0.4.3
[e6cf234a] RandomNumbers v1.5.3
[c1ae055f] RealDot v0.1.0
[3cdcf5f2] RecipesBase v1.3.3
[01d81517] RecipesPipeline v0.6.11
⌃ [731186ca] RecursiveArrayTools v2.38.0
[f2c3362d] RecursiveFactorization v0.2.18
[189a3867] Reexport v1.2.2
[42d2dcc6] Referenceables v0.1.2
[2792f1a3] RegistryInstances v0.1.0
[05181044] RelocatableFolders v1.0.0
[ae029012] Requires v1.3.0
[ae5879a3] ResettableStacks v1.1.1
[37e2e3b7] ReverseDiff v1.14.4
[286e9d63] RingLists v0.2.7
[79098fc4] Rmath v0.7.1
[7e49a35a] RuntimeGeneratedFunctions v0.5.6
[c946c3f1] SCS v1.1.3
[94e857df] SIMDTypes v0.1.0
[476501e8] SLEEFPirates v0.6.38
⌃ [0bca4576] SciMLBase v1.91.3
[e9a6253c] SciMLNLSolve v0.1.4
[c0aeaf25] SciMLOperators v0.2.0
[6c6a2e73] Scratch v1.2.0
[8e049039] SemialgebraicSets v0.2.5
[91c51154] SentinelArrays v1.3.18
[efcf1570] Setfield v1.1.1
[605ecd9f] ShowCases v0.1.0
[992d4aef] Showoff v1.0.3
[777ac1f9] SimpleBufferStream v1.1.0
[55797a34] SimpleGraphs v0.8.4
[727e6d20] SimpleNonlinearSolve v0.1.14
[ec83eff0] SimplePartitions v0.3.0
[cc47b68c] SimplePolynomials v0.2.11
[a6525b86] SimpleRandom v0.3.1
[699a6c99] SimpleTraits v0.9.4
[ce78b400] SimpleUnPack v1.1.0
[66db9d55] SnoopPrecompile v1.0.3
[ed01d8cd] Sobol v1.5.0
[b85f4697] SoftGlobalScope v1.1.0
[a2af1166] SortingAlgorithms v1.1.0
⌅ [47a9eef4] SparseDiffTools v1.31.0
[e56a9233] Sparspak v0.3.9
[276daf66] SpecialFunctions v2.2.0
[171d559e] SplittablesBase v0.1.15
[860ef19b] StableRNGs v1.0.0
[aedffcd0] Static v0.8.6
[0d7ed370] StaticArrayInterface v1.3.0
⌃ [90137ffa] StaticArrays v1.5.19
[1e83bf80] StaticArraysCore v1.4.0
⌃ [82ae8749] StatsAPI v1.5.0
[2913bbd2] StatsBase v0.33.21
[4c63d2b9] StatsFuns v1.3.0
[9672c7b4] SteadyStateDiffEq v1.14.0
[789caeaf] StochasticDiffEq v6.60.0
⌃ [7792a7ef] StrideArraysCore v0.4.9
[892a3eda] StringManipulation v0.3.0
[09ab397b] StructArrays v0.6.15
[c3572dad] Sundials v4.16.0
[24249f21] SymPy v1.1.8
[2efcf032] SymbolicIndexingInterface v0.2.2
⌅ [d1185830] SymbolicUtils v0.19.11
⌅ [0c5d862f] Symbolics v4.14.0
[3783bdb8] TableTraits v1.0.1
[bd369af6] Tables v1.10.1
[62fd8b95] TensorCore v0.1.1
⌅ [8ea1fca8] TermInterface v0.2.3
[5d786b92] TerminalLoggers v0.1.6
[8290d209] ThreadingUtilities v0.5.1
[ac1d9e8a] ThreadsX v0.1.11
[a759f4b9] TimerOutputs v0.5.22
[0796e94c] Tokenize v0.5.25
⌃ [9f7883ad] Tracker v0.2.20
⌃ [3bb67fe8] TranscodingStreams v0.9.11
[28d57a85] Transducers v0.4.75
[a2a6695c] TreeViews v0.3.0
[d5829a12] TriangularSolve v0.1.19
⌃ [410a4b4d] Tricks v0.1.6
[781d530d] TruncatedStacktraces v1.3.0
[5c2747f8] URIs v1.4.2
[3a884ed6] UnPack v1.0.2
[1cfade01] UnicodeFun v0.4.1
⌃ [1986cc42] Unitful v1.12.4
[013be700] UnsafeAtomics v0.2.1
⌃ [d80eeb9a] UnsafeAtomicsLLVM v0.1.1
[41fe7b60] Unzip v0.2.0
⌃ [3d5dd08c] VectorizationBase v0.21.63
[81def892] VersionParsing v1.3.0
[19fa3120] VertexSafeGraphs v0.2.0
[ea10d353] WeakRefStrings v1.4.2
[76eceee3] WorkerUtilities v1.6.1
[c2297ded] ZMQ v1.2.2
⌃ [e88e6eb3] Zygote v0.6.59
[700de1a5] ZygoteRules v0.2.3
[02a925ec] cuDNN v1.0.2
[d9960996] Arb_jll v200.2300.0+0
[6e34b625] Bzip2_jll v1.0.8+0
⌅ [4ee394cb] CUDA_Driver_jll v0.4.0+2
⌅ [76a88914] CUDA_Runtime_jll v0.4.0+2
[62b44479] CUDNN_jll v8.8.1+0
[83423d85] Cairo_jll v1.16.1+1
[7bc98958] Cubature_jll v1.0.5+0
[2e619515] Expat_jll v2.4.8+0
[b22a6f82] FFMPEG_jll v4.4.2+2
[f5851436] FFTW_jll v3.3.10+0
[e134572f] FLINT_jll v200.900.5+0
[a3f928ae] Fontconfig_jll v2.13.93+0
[d7e528f0] FreeType2_jll v2.10.4+0
[559328eb] FriBidi_jll v1.0.10+0
[0656b61e] GLFW_jll v3.3.8+0
⌅ [d2c73de3] GR_jll v0.71.8+0
[78b55507] Gettext_jll v0.21.0+0
[7746bdde] Glib_jll v2.74.0+2
[3b182d85] Graphite2_jll v1.3.14+0
[2e76f6c2] HarfBuzz_jll v2.8.1+1
[1d5cc7b8] IntelOpenMP_jll v2018.0.3+2
[aacddb02] JpegTurbo_jll v2.1.91+0
[c1c5ebd0] LAME_jll v3.100.1+0
[88015f11] LERC_jll v3.0.0+1
⌅ [dad2f222] LLVMExtra_jll v0.0.18+0
[dd4b983a] LZO_jll v2.10.1+0
⌅ [e9f186c6] Libffi_jll v3.2.2+1
[d4300ac3] Libgcrypt_jll v1.8.7+0
[7e76a0d4] Libglvnd_jll v1.6.0+0
[7add5ba3] Libgpg_error_jll v1.42.0+0
[94ce4f54] Libiconv_jll v1.16.1+2
[4b2f31a3] Libmount_jll v2.35.0+0
[89763e89] Libtiff_jll v4.4.0+0
[38a345b3] Libuuid_jll v2.36.0+0
[856f044c] MKL_jll v2022.2.0+0
[2ce0c516] MPC_jll v1.2.1+0
[e7412a2a] Ogg_jll v1.3.5+1
⌅ [656ef2d0] OpenBLAS32_jll v0.3.17+0
[458c3c95] OpenSSL_jll v1.1.20+0
[efe28fd5] OpenSpecFun_jll v0.5.5+0
[91d4177d] Opus_jll v1.3.2+0
[30392449] Pixman_jll v0.40.1+0
[ea2cea3b] Qt5Base_jll v5.15.3+2
[f50d1b31] Rmath_jll v0.4.0+0
⌅ [af6e375f] SCS_GPU_jll v3.2.1+0
⌅ [3f2553a9] SCS_MKL_jll v3.2.2+0
⌅ [f4f2fc5b] SCS_jll v3.2.1+0
[fb77eaff] Sundials_jll v5.2.1+0
[3428059b] SymEngine_jll v0.9.0+1
[a2964d1f] Wayland_jll v1.21.0+0
[2381bf8a] Wayland_protocols_jll v1.25.0+0
[02c8fc9c] XML2_jll v2.10.3+0
[aed1982a] XSLT_jll v1.1.34+0
[4f6342f7] Xorg_libX11_jll v1.6.9+4
[0c0b7dd1] Xorg_libXau_jll v1.0.9+4
[935fb764] Xorg_libXcursor_jll v1.2.0+4
[a3789734] Xorg_libXdmcp_jll v1.1.3+4
[1082639a] Xorg_libXext_jll v1.3.4+4
[d091e8ba] Xorg_libXfixes_jll v5.0.3+4
[a51aa0fd] Xorg_libXi_jll v1.7.10+4
[d1454406] Xorg_libXinerama_jll v1.1.4+4
[ec84b674] Xorg_libXrandr_jll v1.5.2+4
[ea2f1a96] Xorg_libXrender_jll v0.9.10+4
[14d82f49] Xorg_libpthread_stubs_jll v0.1.0+3
[c7cfdc94] Xorg_libxcb_jll v1.13.0+3
[cc61e674] Xorg_libxkbfile_jll v1.1.0+4
[12413925] Xorg_xcb_util_image_jll v0.4.0+1
[2def613f] Xorg_xcb_util_jll v0.4.0+1
[975044d2] Xorg_xcb_util_keysyms_jll v0.4.0+1
[0d47668e] Xorg_xcb_util_renderutil_jll v0.3.9+1
[c22f9ab0] Xorg_xcb_util_wm_jll v0.4.1+1
[35661453] Xorg_xkbcomp_jll v1.4.2+4
[33bec58e] Xorg_xkeyboard_config_jll v2.27.0+4
[c5fb5394] Xorg_xtrans_jll v1.4.0+3
[8f1865be] ZeroMQ_jll v4.3.4+0
⌃ [3161d3a3] Zstd_jll v1.5.4+0
⌅ [214eeab7] fzf_jll v0.29.0+0
[a4ae2306] libaom_jll v3.4.0+0
[0ac62f75] libass_jll v0.15.1+0
[f638f0a6] libfdk_aac_jll v2.0.2+0
[b53b4c65] libpng_jll v1.6.38+0
[a9144af2] libsodium_jll v1.0.20+0
[f27f6e37] libvorbis_jll v1.3.7+1
[1270edf5] x264_jll v2021.5.5+0
[dfaa095f] x265_jll v3.5.0+0
[d8fb68d0] xkbcommon_jll v1.4.1+0
[0dad84c5] ArgTools v1.1.1
[56f22d72] Artifacts
[2a0f44e3] Base64
[ade2ca70] Dates
[8bb1440f] DelimitedFiles
[8ba89e20] Distributed
[f43a241f] Downloads v1.6.0
[7b1f6079] FileWatching
[9fa8497b] Future
[b77e0a4c] InteractiveUtils
[4af54fe1] LazyArtifacts
[b27032c2] LibCURL v0.6.3
[76f85450] LibGit2
[8f399da3] Libdl
[37e2e46d] LinearAlgebra
[56ddb016] Logging
[d6f4376e] Markdown
[a63ad114] Mmap
[ca575930] NetworkOptions v1.2.0
[44cfe95a] Pkg v1.8.0
[de0858da] Printf
[9abbd945] Profile
[3fa0cd96] REPL
[9a3f8284] Random
[ea8e919c] SHA v0.7.0
[9e88b42a] Serialization
[1a1011a3] SharedArrays
[6462fe0b] Sockets
[2f01184e] SparseArrays
[10745b16] Statistics
[4607b0f0] SuiteSparse
[fa267f1f] TOML v1.0.0
[a4e569a6] Tar v1.10.1
[8dfed614] Test
[cf7118a7] UUIDs
[4ec0a83e] Unicode
[e66e0078] CompilerSupportLibraries_jll v1.0.1+0
[781609d7] GMP_jll v6.2.1+2
[deac9b47] LibCURL_jll v7.84.0+0
[29816b5a] LibSSH2_jll v1.10.2+0
[3a97d323] MPFR_jll v4.1.1+1
[c8ffd9c3] MbedTLS_jll v2.28.0+0
[14a3606d] MozillaCACerts_jll v2022.2.1
[4536629a] OpenBLAS_jll v0.3.20+0
[05823500] OpenLibm_jll v0.8.1+0
[efcefdf7] PCRE2_jll v10.40.0+0
[bea87d4a] SuiteSparse_jll v5.10.1+0
[83775a58] Zlib_jll v1.2.12+3
[8e850b90] libblastrampoline_jll v5.1.1+0
[8e850ede] nghttp2_jll v1.48.0+0
[3f19e933] p7zip_jll v17.4.0+0
Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

(@v1.8) pkg>

[maleadt]

So the Nu66l version slug used above is from CUDA_Runtime_jll 0.5. Even though that package depends on CUDA_Driver_jll 0.5, apparently it gets loaded during upgrades at the time only CUDA_Driver_jll 0.4 is available. That's very annoying, and essentially means we can't rely on compatibility bounds for package augmentation hooks...

[maleadt]

JuliaRegistries/General#81742 should fix this, hopefully