Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Precompile pidlocks: add to NEWS and docs #50385

Merged
merged 3 commits into from
Sep 17, 2023
Merged

Precompile pidlocks: add to NEWS and docs #50385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0ad687c
update hpc docs given cache pidlocks
IanButterworth Jul 2, 2023
ef0f0d9
add precompile pidlocking to NEWS
IanButterworth Jul 2, 2023
d7a7768
tryfix
IanButterworth Sep 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ Compiler/Runtime improvements
* The mark phase of the garbage collector is now multi-threaded ([#48600]).
* [JITLink](https://llvm.org/docs/JITLink.html) is enabled by default on Linux aarch64 when Julia is linked to LLVM 15 or later versions ([#49745]).
This should resolve many segmentation faults previously observed on this platform.
* The precompilation process now uses pidfile locks and orchestrates multiple julia processes to only have one process
spend effort precompiling while the others wait. Previously all would do the work and race to overwrite the cache files.
([#49052])

Command-line option changes
---------------------------
Expand Down
20 changes: 9 additions & 11 deletions doc/src/manual/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1052,17 +1052,15 @@ Modifying OpenBLAS settings or compiling Julia with a different BLAS library, eg

### How do I manage precompilation caches in distributed file systems?

When using `julia` in high-performance computing (HPC) facilities, invoking
_n_ `julia` processes simultaneously creates at most _n_ temporary copies of
precompilation cache files. If this is an issue (slow and/or small distributed
file system), you may:

1. Use `julia` with `--compiled-modules=no` flag to turn off precompilation.
2. Configure a private writable depot using `pushfirst!(DEPOT_PATH, private_path)`
where `private_path` is a path unique to this `julia` process. This
can also be done by setting environment variable `JULIA_DEPOT_PATH` to
`$private_path:$HOME/.julia`.
3. Create a symlink from `~/.julia/compiled` to a directory in a scratch space.
When using Julia in high-performance computing (HPC) facilities with shared filesystems, it is recommended to use a shared
depot (via the `JULIA_DEPOT_PATH` environment variable). Since Julia v1.10, multiple Julia processes on functionally similar
workers and using the same depot will coordinate via pidfile locks to only spend effort precompiling on one process while the
others wait. The precompilation process will indicate when the process is precompiling or waiting for another that is
precompiling. If non-interactive the messages are via `@debug`.

However, due to caching of binary code, the cache rejection since v1.9 is more strict and users may need to set the
`[JULIA_CPU_TARGET](@ref)` environment variable appropriately to get a single cache that is usable throughout the HPC
environment.

## Julia Releases

Expand Down