-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Add sysimage autoload mechanism #40472
Conversation
This is the second part of the plan described in #40414 (though complimentary to the PR itself). In particular, this PR makes it possible to quickly replace a system image during initial startup. This is done by adding a hook early in the startup sequence (after the system image, but before any dependent libraries are initialized) for Julia to look at the specified project file and decide to load a different sysimage instead. In the current version of the PR, this works as following: - If the `--autoload` argument is specified, julia will hash the contents of the currently active project's manifest path. - If a corresponding .so is found in `~/.julia/sysimages`, it will load that sysimage instead. - If not, loading will proceed as usual, a warning is generated but before any user code is run, Julia will `require` any dependencies specified in the Project.toml. The third point is there such that independent of whether or not the system image is found, the environment upon transfer of control to the user is always the same (e.g. a package may have type-pirated a method, which is available independent of whether the user ever explicitly did `using`). This is highly incomplete. In particular, these scheme to find the system image needs to take account of preferences and should probably exlcude any packages that are `dev`'ed (or their dependents). I'm not sure I'll have the time to get around to finishing this, but I'm hoping somebody else would be willing to jump in for that part. The underlying mechanism seems to work fine at this point, so this work should be mostly confined to loading.jl.
Note: I think it would be better to make this a flag inside the Project.toml rather than a command line argument, but at the moment we don't parse the Project.toml during startup, so I wasn't sure whether adding that was a good idea. I think we can make that decision once we've experimented with |
I think it's worth considering what should happen with stacked environments. It may be required to parse the TOMLs, merge them, and then hash it. Also, since multiple versions and builds of Julia can exist, and also the file system can be shared, it may be better to include the triplet here.
Since the "effective manifest" is the result of stacked environment, I think it can be confusing as a UI to associate a single Project.toml to the entire stack of environments, especially if the sysimage path depends on the content of the manifest. FYI, I discussed an alternative design in my PR #35794. The idea is that, since the stacked environment mechanism is too dynamic, make sysimage location very flexible, too. Instead, the default sysimage can be set at the time the sysimage is built. |
In the world of pkgimages is this still beneficial? |
Ref the discussion in JuliaLang/PrecompileTools.jl#14 (comment) |
This is the second part of the plan described in #40414
(though complimentary to the PR itself). In particular, this
PR makes it possible to quickly replace a system image during
initial startup. This is done by adding a hook early in the
startup sequence (after the system image, but before any
dependent libraries are initialized) for Julia to look at
the specified project file and decide to load a different
sysimage instead.
In the current version of the PR, this works as following:
--autoload
argument is specified, julia will hashthe contents of the currently active project's manifest path.
~/.julia/sysimages
, it willload that sysimage instead.
but before any user code is run, Julia will
require
anydependencies specified in the Project.toml.
The third point is there such that independent of whether or not
the system image is found, the environment upon transfer of
control to the user is always the same (e.g. a package may
have type-pirated a method, which is available independent
of whether the user ever explicitly did
using
).This is highly incomplete. In particular, these scheme to find
the system image needs to take account of preferences and should
probably exlcude any packages that are
dev
'ed (or theirdependents). I'm not sure I'll have the time to get around to
finishing this, but I'm hoping somebody else would be willing
to jump in for that part. The underlying mechanism seems to work
fine at this point, so this work should be mostly confined to
loading.jl.