Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add sysimage autoload mechanism #40472

Closed
wants to merge 1 commit into from
Closed

WIP: Add sysimage autoload mechanism #40472

wants to merge 1 commit into from

Conversation

Keno
Copy link
Member

@Keno Keno commented Apr 13, 2021

This is the second part of the plan described in #40414
(though complimentary to the PR itself). In particular, this
PR makes it possible to quickly replace a system image during
initial startup. This is done by adding a hook early in the
startup sequence (after the system image, but before any
dependent libraries are initialized) for Julia to look at
the specified project file and decide to load a different
sysimage instead.

In the current version of the PR, this works as following:

  • If the --autoload argument is specified, julia will hash
    the contents of the currently active project's manifest path.
  • If a corresponding .so is found in ~/.julia/sysimages, it will
    load that sysimage instead.
  • If not, loading will proceed as usual, a warning is generated
    but before any user code is run, Julia will require any
    dependencies specified in the Project.toml.

The third point is there such that independent of whether or not
the system image is found, the environment upon transfer of
control to the user is always the same (e.g. a package may
have type-pirated a method, which is available independent
of whether the user ever explicitly did using).

This is highly incomplete. In particular, these scheme to find
the system image needs to take account of preferences and should
probably exlcude any packages that are dev'ed (or their
dependents). I'm not sure I'll have the time to get around to
finishing this, but I'm hoping somebody else would be willing
to jump in for that part. The underlying mechanism seems to work
fine at this point, so this work should be mostly confined to
loading.jl.

This is the second part of the plan described in #40414
(though complimentary to the PR itself). In particular, this
PR makes it possible to quickly replace a system image during
initial startup. This is done by adding a hook early in the
startup sequence (after the system image, but before any
dependent libraries are initialized) for Julia to look at
the specified project file and decide to load a different
sysimage instead.

In the current version of the PR, this works as following:
 - If the `--autoload` argument is specified, julia will hash
   the contents of the currently active project's manifest path.
 - If a corresponding .so is found in `~/.julia/sysimages`, it will
   load that sysimage instead.
 - If not, loading will proceed as usual, a warning is generated
   but before any user code is run, Julia will `require` any
   dependencies specified in the Project.toml.

The third point is there such that independent of whether or not
the system image is found, the environment upon transfer of
control to the user is always the same (e.g. a package may
have type-pirated a method, which is available independent
of whether the user ever explicitly did `using`).

This is highly incomplete. In particular, these scheme to find
the system image needs to take account of preferences and should
probably exlcude any packages that are `dev`'ed (or their
dependents). I'm not sure I'll have the time to get around to
finishing this, but I'm hoping somebody else would be willing
to jump in for that part. The underlying mechanism seems to work
fine at this point, so this work should be mostly confined to
loading.jl.
@Keno Keno added help wanted Indicates that a maintainer wants help on an issue or pull request compiler:latency Compiler latency compiler:precompilation Precompilation of modules labels Apr 13, 2021
@Keno
Copy link
Member Author

Keno commented Apr 13, 2021

  • If the --autoload argument is specified, julia will hash
    the contents of the currently active project's manifest path.

Note: I think it would be better to make this a flag inside the Project.toml rather than a command line argument, but at the moment we don't parse the Project.toml during startup, so I wasn't sure whether adding that was a good idea. I think we can make that decision once we've experimented with --autoload to decide whether it's useful enough that it's worth spending the time parsing the Project.toml.

@vtjnash
Copy link
Member

vtjnash commented Apr 14, 2021

Oh, maybe I should now rebase #38119 on top of this. It was previously a bit slow to build, lacking #40414, and hard to configure at launch, lacking this. But if this has good performance, does that sound like a good path to you?

@tkf
Copy link
Member

tkf commented Apr 15, 2021

  • If the --autoload argument is specified, julia will hash
    the contents of the currently active project's manifest path.

I think it's worth considering what should happen with stacked environments. It may be required to parse the TOMLs, merge them, and then hash it.

Also, since multiple versions and builds of Julia can exist, and also the file system can be shared, it may be better to include the triplet here.

make this a flag inside the Project.toml rather than a command line argument

Since the "effective manifest" is the result of stacked environment, I think it can be confusing as a UI to associate a single Project.toml to the entire stack of environments, especially if the sysimage path depends on the content of the manifest.

FYI, I discussed an alternative design in my PR #35794. The idea is that, since the stacked environment mechanism is too dynamic, make sysimage location very flexible, too. Instead, the default sysimage can be set at the time the sysimage is built.

@IanButterworth
Copy link
Member

In the world of pkgimages is this still beneficial?

@KristofferC
Copy link
Member

Ref the discussion in JuliaLang/PrecompileTools.jl#14 (comment)

@vtjnash vtjnash closed this May 4, 2023
@vtjnash vtjnash deleted the kf/sysimgautoload branch May 4, 2023 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:latency Compiler latency compiler:precompilation Precompilation of modules help wanted Indicates that a maintainer wants help on an issue or pull request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants