[deps] enable zstd support

[vtjnash]

Waiting on JuliaPackaging/Yggdrasil#11183 to update the LLVM shards also

[vtjnash]

Arg, all of that build effort and the llvm Findzstd script is buggy, producing incorrect results if used with (!MSVC && WIN32)

llvm/llvm-project#139945

[vtjnash]

i686-w64-mingw32 is running into the same clang ABI miscompiles as we've encountered before. I guess we got lucky before that zlib didn't smash anything too valuable, while zstd does

The simplest thing to do may be vendor our own copy of the bad code, until the clang compiler stops miscompiling sret stuff https://llvm.org/doxygen/Compression_8cpp_source.html#l00151

[gbaraldi]

Can we disable this on 32 bit windows instead?

[rfourquet]

Unless I messed up with git-bisect, since this commit, I can't compile (or run, even with juliaup) julia without updating LD_LIBRARY_PATH to point at some libstdc++.so. Maybe it's expected, but I thought julia was supposed to use its own version of this lib. This is the error when compiling:

When properly compiled, LD_LIBRARY_PATH is again needed at runtime to avoid having 7z fail; e.g. in Pkg mode, when typing add B:

(@v1.13) pkg> add B/data/cache/dot_julia/juliaup/julia-nightly/libexec/julia/7z: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

[KristofferC]

This should maybe not have been squashed. It does a lot of different things.

@rfourquet did you understand what happened? Does it still occur on master?

[rfourquet]

No I haven't tried to understand what happened, but the error still occurs on master.

[Zentrik]

It looks like the 7z binary in libexec doesn't point to the libstdc++ Julia ships with,

rag@pop-os:~$ ldd /home/rag/.julia/juliaup/julia-1.11.5+0.x64.linux.gnu/libexec/julia/7z
	linux-vdso.so.1 (0x000077d7eb465000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000077d7eb43c000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000077d7eb000000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000077d7eb355000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000077d7eb331000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000077d7eac00000)
	/lib64/ld-linux-x86-64.so.2 (0x000077d7eb467000)

rag@pop-os:~$ ldd /home/rag/.julia/juliaup/julia-nightly/libexec/julia/7z
	linux-vdso.so.1 (0x000071b72c455000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000071b72c42c000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000071b72c000000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000071b72c345000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000071b72c321000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000071b72bc00000)
	/lib64/ld-linux-x86-64.so.2 (0x000071b72c457000)

Indeed it doesn't have an rpath set unlike lld

rag@pop-os:~$ readelf -d /home/rag/.julia/juliaup/julia-nightly/libexec/julia/lld

Dynamic section at offset 0x5f4b70 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libzstd.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libLLVM.so.20.1jl]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../lib/julia]
 0x000000000000000c (INIT)               0x473000
 0x000000000000000d (FINI)               0x9209f4
 0x0000000000000019 (INIT_ARRAY)         0x9cf488
 0x000000000000001b (INIT_ARRAYSZ)       120 (bytes)
 0x0000000000000004 (HASH)               0x3ff358
 0x000000006ffffef5 (GNU_HASH)           0x405238
 0x0000000000000005 (STRTAB)             0x422da8
 0x0000000000000006 (SYMTAB)             0x40b510
 0x000000000000000a (STRSZ)              290403 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x9f5000
 0x0000000000000002 (PLTRELSZ)           17064 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x46e968
 0x0000000000000007 (RELA)               0x46ccd0
 0x0000000000000008 (RELASZ)             7320 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x46cb20
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x46abbc
 0x0000000000000000 (NULL)               0x0
rag@pop-os:~$ readelf -d /home/rag/.julia/juliaup/julia-nightly/libexec/julia/7z

Dynamic section at offset 0x2ef3a0 contains 24 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x402a30
 0x000000000000000d (FINI)               0x60d938
 0x0000000000000004 (HASH)               0x400260
 0x0000000000000005 (STRTAB)             0x401470
 0x0000000000000006 (SYMTAB)             0x4006c0
 0x000000000000000a (STRSZ)              1673 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x8ef578
 0x0000000000000002 (PLTRELSZ)           3144 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x401de8
 0x0000000000000007 (RELA)               0x401ce0
 0x0000000000000008 (RELASZ)             264 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x401c20
 0x000000006fffffff (VERNEEDNUM)         5
 0x000000006ffffff0 (VERSYM)             0x401afa
 0x0000000000000000 (NULL)               0x0

I'm confused why this errors for you though, do you not have libstdc++ on your ld search path?

[Zentrik]

I guess this PR broke your case because it removed adding the lib dir to the LD_LIBRARY_PATH in https://github.com/JuliaLang/julia/pull/58344/files#diff-c659988c47814d9b26b098385da4b721a02de0551d54b38e80d8bc5902b2b320L39-L45

[rfourquet]

I'm confused why this errors for you though, do you not have libstdc++ on your ld search path?

No, I'm on NixOS, in which LD_LIBRARY_PATH is essentially empty. Of course there are workarounds (e.g. using nix-ld, but there can be problems there too: for example on v1.11 with nix-ld enabled, I get an error with julia using the system's libcurl instead of its own version: undefined symbol: libssh2_session_callback_set2).

[Zentrik]

No, I'm on NixOS, in which LD_LIBRARY_PATH is essentially empty.

Makes sense, I thought I had built after this commit using nix but I guess not.

I get an error with julia using the system's libcurl instead of its own version: undefined symbol: libssh2_session_callback_set2

This is during build or just when using it?

[rfourquet]

This is during build or just when using it?

When using, but it might also be when building, I don't remember.

This looks like: