Avoid using fork() when probing system libstdc++
#60254
Conversation
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
3 times, most recently
from
79f02b0 to
b668d14
Compare
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
2 times, most recently
from
535e217 to
6916b81
Compare
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
2 times, most recently
from
50164ec to
39f68a7
Compare
|
@gbaraldi pointed out that Nonetheless our $ nix-shell -p julia_111-bin
$ strace julia -e '' 2>&1 | grep libstdc++
openat(AT_FDCWD, "/nix/store/i3ibgfskl99qd8rslafbpaa1dmxdzh1z-glibc-2.40-66/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/16hvpw4b3r05girazh4rnwbw0jgjkb4l-xgcc-14.3.0-libgcc/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/gs785h7im3vchbdxalrrjnmcan4dsign-julia-bin-1.11.6/lib/julia/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3so I think we can likely merge this as-is.
|
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
4 times, most recently
from
400adca to
47f1667
Compare
| **/ | ||
| const char *jl_loader_probe_system_library(const char *libname, const char *symbol) | ||
| { | ||
| char buf[PATH_MAX]; |
There was a problem hiding this comment.
PATH_MAX is much smaller than the maximum path length
There was a problem hiding this comment.
That's true, but exceeding PATH_MAX usually requires iterating path parts through openat etc. doesn't it?
I didn't think you could avoid ENAMETOOLONG if you try to open() a too-long string
There was a problem hiding this comment.
On some OS, it may be the variable {PATH_MAX} on some systems not the constant PATH_MAX (though on linux I suppose it is constant, and long enough): https://pubs.opengroup.org/onlinepubs/009696599/basedefs/limits.h.html, pathconf
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
3 times, most recently
from
f8ce0da to
c23c90d
Compare
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
2 times, most recently
from
b8e88e7 to
0d4bbc5
Compare
Rather than relying on the linker to do all of the heavy lifting, this changes our probing technique to directly parse the `ld.so.cache` and `libstdc++.so.6` files. As long as we can expect `ld.so.cache` to be available in `/etc` on all systems, this seems to be a reliable way to locate system libraries on Linux systems. That allows us to avoid `fork()`. For a small application that has just started up `fork()` is no big deal, but it's quite heavy-handed for Julia- as-a-library scenarios where resident memory may already be large. Many soft-embedded targets also do not support fork() well, so it is good for our compatibility to adjust this.
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
from
0d4bbc5 to
6d1477f
Compare
topolarity
force-pushed
the
ct/forkless-libstdcxx-probe
branch
from
6d1477f to
ea16b85
Compare
|
Works for me on Debian 8 now. 👍 Here's the test rig I have been using when loading JuliaC libraries into old libstdc++ programs: https://gist.github.com/xal-0/12a702f0c0e48848a33561b757107ec3 ( I have yet to try compiling a newer version of libstdc++ for Debian 8 and having Julia probe that, but will do that soon. |
Also some minor review feedback
Somewhat of a companion to #60248. For a small application that has just started up `fork()` is not a huge concern, but it's quite heavy-handed for Julia- as-a-library scenarios where resident memory may already be large. Many soft-embedded targets also do not support fork() well, so it is good for our compatibility to adjust this. Rather than relying on the linker to do all of the heavy lifting, this changes our `libstdcxx` probe sequence to directly parse the `ld.so.cache` and `libstdc++.so.6` files. As long as we can expect `/etc/ld.so.cache` to be the same path on all Linux systems, this seems to be a reliable way to locate system libraries. (cherry picked from commit ac4ee59)
Somewhat of a companion to #60248. For a small application that has just started up `fork()` is not a huge concern, but it's quite heavy-handed for Julia- as-a-library scenarios where resident memory may already be large. Many soft-embedded targets also do not support fork() well, so it is good for our compatibility to adjust this. Rather than relying on the linker to do all of the heavy lifting, this changes our `libstdcxx` probe sequence to directly parse the `ld.so.cache` and `libstdc++.so.6` files. As long as we can expect `/etc/ld.so.cache` to be the same path on all Linux systems, this seems to be a reliable way to locate system libraries. (cherry picked from commit ac4ee59)
3 participants
Somewhat of a companion to #60248.
For a small application that has just started up
fork()is not a huge concern, but it's quite heavy-handed for Julia- as-a-library scenarios where resident memory may already be large. Many soft-embedded targets also do not support fork() well, so it is good for our compatibility to adjust this.Rather than relying on the linker to do all of the heavy lifting, this changes our
libstdcxxprobe sequence to directly parse theld.so.cacheandlibstdc++.so.6files. As long as we can expect/etc/ld.so.cacheto be the same path on all Linux systems, this seems to be a reliable way to locate system libraries.