sysimg: Allow loading a system image that is already present in memory #51121
Conversation
|
I guess weak externals are supposed to work on windows, but are broken in the compiler https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90826 (issue by our very own @yuyichao). I think I'm gonna disable the weak symbol fast path on windows for the time being. |
|
Leaving this here for my own future reference to take another look: https://maskray.me/blog/2021-04-25-weak-symbol |
|
By already in memory you mean linking sys.so statically into the |
|
Essentially yes. There's various other configurations, but that's the prototypical one |
src/init.c
Outdated
| // loads sysimg if available, and conditionally sets jl_options.cpu_target | ||
| if (jl_options.image_file) | ||
| if (rel == JL_IMAGE_IN_MEMORY) | ||
| jl_set_sysimg_so(jl_RTLD_DEFAULT_handle); |
There was a problem hiding this comment.
This is normally an alias for jl_libjulia_internal_handle, but I think you wanted to assume jl_exe_handle here?
There was a problem hiding this comment.
Well, I wanted RTLD_DEFAULT, but I'm only testing Linux for now, so it doesn't make a difference. I'll switch this to jl_exe_handle (assuming nothing breaks), but of course we may need extra options in the future potentially.
There was a problem hiding this comment.
Seems like a reasonable approach. I have done this a few times too. I think usually I had altered the jl_options struct to take the handle(s) as an argument (since I think jl_preload_sysimg_so would early-exit if it was already populated), so that relying on the behaviors of notoriously different semantics for weak wasn't necessary. But it has been a while since I experimented with it, and that particular trick is probably broken now.
|
There is a comment in the code still that says you were supposed to do this by calling |
src/init.c
Outdated
| free(free_path); | ||
| free_path = NULL; | ||
| if (jl_options.image_file) { | ||
| if (jl_options.image_file && rel != JL_IMAGE_IN_MEMORY) { |
There was a problem hiding this comment.
I think you still need to process this field, as julia expects it to hold a valid value in various places (for Distributed, precompilation, and other similar optimizations).
I think it'd probably still work, but it still assumes that you have a dynamic linking environment, which you may not (hence the weak symbols here). I think this API is cleaner, but the other API is useful for the use case where you're embedding into a big host C/C++ environment and you already loaded the sysimage as a shared library. |
I've written this code probably three times at this point, but for some reason it never made it into a PR. This allows loading a system image that has already been loaded into memory. This happen when wanting to distribute a static or mostly-static binary of julia code where the system image (and optionally other libraries like libjulia, etc.) are linked directly into the main executable. It is also useful for deployment to environments that do not have (or have incomplete) support for dynamic linking (e.g. wasm or embedded).
Aliases are not supported on mac it turns out.
Keno
force-pushed
the
kf/sysimgselfload
branch
from
bf93cb8 to
c83fa3c
Compare
I've written this code probably three times at this point, but for some reason it never made it into a PR. This allows loading a system image that has already been loaded into memory. This happen when wanting to distribute a static or mostly-static binary of julia code where the system image (and optionally other libraries like libjulia, etc.) are linked directly into the main executable. It is also useful for deployment to environments that do not have (or have incomplete) support for dynamic linking (e.g. wasm or embedded).