compat entry for HDF5_jll

[cgarling]

On Julia 1.8.3 executing ] add HDF5 installed HDF5 v0.16.14 with HDF5_jll v1.10.5+7. However, this combination failed to precompile due to a libhdf5 not defined error.
julia> import HDF5 [ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] ERROR: LoadError: UndefVarError: libhdf5 not defined...
I was able to get things working by explicitly running ] add HDF5_jll, which installed HDF5_jll v1.10.5+7 again, but then running ] update updated it to HDF5_jll v1.12.2+2. Updating HDF5_jll apparently required a downgrade of OpenSSL_jll from v3.0.8+0 ⇒ v1.1.20+0. I have not explicitly added OpenSSL_jll, but it is required by some other package.

My takeaway is that HDF5 v0.16.14 may not actually be compatible with HDF5_jll v1.10.5 and the compat entry may need to be updated. Additionally it seems that the newest versions of HDF5_jll do not support the newest versions of OpenSSL_jll, which can create conflicts in some environments.

[mkitti]

OpenSSL_jll v3.0.8 was recently introduced into the ecosystem by me:
JuliaRegistries/General#78542

For the most part, I would most packages to be using OpenSSL_jll v1.1.20+0.

There is only one package that is actually compatible with it at the moment, OpenSSL.jl.

I created a temporary environment with just HDF5 v0.16.14 and HDF5_jll v1.10.5+7, and did not find any issues. The environment seemed to compile fine on x86_64 glibc Linux.

(jl_7GUvPL) pkg> st
Status `/tmp/jl_7GUvPL/Project.toml`
  [f67ccb44] HDF5 v0.16.14
⌃ [0234f1f7] HDF5_jll v1.10.5+7
Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> using Pkg

julia> pkg"status"
Status `/tmp/jl_7GUvPL/Project.toml`
  [f67ccb44] HDF5 v0.16.14
⌃ [0234f1f7] HDF5_jll v1.10.5+7
Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> using HDF5

julia> HDF5.API.h5_get_libversion()
v"1.10.4"

Could you provide some additional details about your platform and your environment?

[cgarling]

Thanks for looking into this.
I just tried this combination on Ubuntu 22.04 with Julia v1.7.3 and it worked without error.

I originally encountered the above error on MacOS, and I can replicate the error on MacOS 13.3.1(a) with Julia 1.8.3 in an environment with only HDF5_jll and HDF5.jl:

Status ~/Downloads/tmp/Project.toml
  [f67ccb44] HDF5 v0.16.14
⌃ [0234f1f7] HDF5_jll v1.10.5+7

The full stacktrace is

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: UndefVarError: libhdf5 not defined
Stacktrace:
  [1] top-level scope
    @ ~/.julia/packages/HDF5/HtnQZ/src/api/types.jl:246
  [2] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
  [3] include(x::String)
    @ HDF5.API ~/.julia/packages/HDF5/HtnQZ/src/api/api.jl:1
  [4] top-level scope
    @ ~/.julia/packages/HDF5/HtnQZ/src/api/api.jl:17
  [5] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
  [6] include(x::String)
    @ HDF5 ~/.julia/packages/HDF5/HtnQZ/src/HDF5.jl:1
  [7] top-level scope
    @ ~/.julia/packages/HDF5/HtnQZ/src/HDF5.jl:67
  [8] include
    @ ./Base.jl:419 [inlined]
  [9] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1554
 [10] top-level scope
    @ stdin:1
in expression starting at /Users/txa5ge/.julia/packages/HDF5/HtnQZ/src/api/types.jl:246
in expression starting at /Users/txa5ge/.julia/packages/HDF5/HtnQZ/src/api/api.jl:1
in expression starting at /Users/txa5ge/.julia/packages/HDF5/HtnQZ/src/HDF5.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /Users/txa5ge/.julia/compiled/v1.8/HDF5/jl_NfGS3M.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144

[mkitti]

The issue is that we will have to retroactively modify the compat entries in the General registry to fully resolve this. However, I suspect the situation may be platform specific.

[tamasgal]

I just hit this. Currently cannot install e.g. HTTP.jl together with HDF5.jl on an M1 Mac since it picks HDF5_jll v1.10.5+7 which is not working on M1 ;(

edit: a workaround is installing HDF5lib with e.g. homebrew and then set

export JULIA_HDF5_PATH=/opt/homebrew/Cellar/hdf5/1.12.2_2

make sure to wipe the precompilations when you change the HDF5 path, those seem to remember quite a few things 😆

[mkitti]

Just specify [email protected].

(@v1.9) pkg> activate --temp
  Activating new project at `/tmp/jl_i1qFEP`

(jl_i1qFEP) pkg> add HTTP HDF5 [email protected]

(jl_i1qFEP) pkg> st
Status `/tmp/jl_i1qFEP/Project.toml`
  [f67ccb44] HDF5 v0.16.14
  [cd3eb016] HTTP v1.9.5
  [0234f1f7] HDF5_jll v1.12.2+2

[tamasgal]

The problem is that HDF5 is in quite a few of our packages and the automatic dependency solver falls back to v1.10. Pinning [email protected] would require to create a couple of new releases ;( but thanks

[mkitti]

Just to be clear, if I create a release without a 1.10 compat, that will not fix the problem because it probably will still solve for an old version of HDF5. To prevent this solution, one would have to edit the following file in the general registry:
https://github.com/JuliaRegistries/General/blob/master/H/HDF5/Compat.toml

Another solution is that we finish merging to release a HDF5_jll 1.14 which is compatible with OpenSSL v3.
JuliaPackaging/Yggdrasil#6551

[mkitti]

We just released HDF5_jll 1.14.

(jl_ILOPEd) pkg> st
Status `/redacted/Project.toml`
  [f67ccb44] HDF5 v0.16.15
  [cd3eb016] HTTP v1.9.5
  [4d8831e6] OpenSSL v1.4.1
  [0234f1f7] HDF5_jll v1.14.0+0
  [458c3c95] OpenSSL_jll v3.0.8+0

Will this solve the issue?

[tamasgal]

Perfect! Works fine :)

[mkitti]

HDF5 v0.16.15 supports HDF5_jll v1.14. HDF5_jll v1.14 is compatible with OpenSSL_jll v3.0.8. This should solve many of the solutions which try to use OpenSSL_jll v3.

A retroactive compat entry would cut off users from HDF5_jll v1.10 which works on on some platforms.