Skip to content

precompilepkgs: optionally run precompilation in background task with keyboard controls#60943

Merged
IanButterworth merged 2 commits intoJuliaLang:masterfrom
IanButterworth:ib/background_precompile
Apr 1, 2026
Merged

precompilepkgs: optionally run precompilation in background task with keyboard controls#60943
IanButterworth merged 2 commits intoJuliaLang:masterfrom
IanButterworth:ib/background_precompile

Conversation

@IanButterworth
Copy link
Copy Markdown
Member

@IanButterworth IanButterworth commented Feb 6, 2026 •
edited
Loading

Run package precompilation in a background task, with a monitor task that streams output and handles keyboard input. This enables users to detach from parallel precompilation (letting it continue silently) and reattach later.

Also any precompile calls (directly or indirectly via auto-precompilation or package loading) will merge into the existing running job. In the case of package loading, it will add the work, monitor it, then detach early once the package is done, so it can be loaded.

Keyboard controls during precompilation:

  • 'c' cancels all subprocesses (SIGKILL) with a quick and tidy stop.
  • 'd' detaches from monitoring (when detachable=true i.e. Pkg.precompile() uses this)
  • 'i' sends a profiling signal (SIGINFO/SIGUSR1) for profile peek. Output from workers prints at end, or immediately if the main package.
  • 'v' enables verbose mode with stats on the workers
  • Ctrl-C interrupts subprocesses (SIGINT) and shows output (the idea is it does this more gracefully than it used to)

Handling ctrl-c manually means a lot of ugly try-catches can be removed.

The last two operations here are a quick cancel via c and then a ctrl-c showing the interrupt trace.

Screen.Recording.2026-02-14.at.7.32.59.PM.mov

The Pkg.jl side of this change is in JuliaLang/Pkg.jl#4602.
Closes JuliaLang/Pkg.jl#4359

Developed using Claude Opus 4.6

Note, this temporarily includes the Pkg fix for testing purposes.

@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 2 times, most recently from 54f7094 to 76ad115 Compare February 6, 2026 02:18
@IanButterworth IanButterworth mentioned this pull request Feb 6, 2026
Merged
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 13 times, most recently from aea1c3e to 4370805 Compare February 8, 2026 15:31
@IanButterworth IanButterworth marked this pull request as ready for review February 8, 2026 15:48
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch from 4370805 to 0e122b0 Compare February 8, 2026 15:48
@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Feb 8, 2026

@vtjnash given you filed JuliaLang/Pkg.jl#4359 I recommend trying this out to see if the behavior makes sense before the implementation review

@IanButterworth IanButterworth mentioned this pull request Feb 8, 2026
Closed
@vtjnash
Copy link
Copy Markdown
Member

vtjnash commented Feb 8, 2026

This sounds great, even before trying it. I'd suggest adding '?' and 'h' to print help. Possibly also 'q' and ] as an alias for detach

@vtjnash
Copy link
Copy Markdown
Member

vtjnash commented Feb 8, 2026 •
edited
Loading

Also, depending on how you're handling interruptions and IO, you might need to make the background loop have the ability to print all queued messages when using loads a related package (and to start loading the new package without oversubscription–right now the precompilepkg code fails to sync the work limit globally and implicitly thus just assumes that it is only safe to load code from a single Task).

@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Feb 8, 2026 •
edited
Loading

The pidfile lock currently prints a message about another task in this process precompiling, but yeah it could be clearer. Fixed. It now dips into the precompile job and monitors until that specific package finishes, then proceeds to load it.

For ctrl-c it uses raw mode to intercept it then sets an early exit flag, then interrupts existing processes deep in compilecache, then lets the mechanics tidy up and show output. So that the mechanics aren't interrupted like they use to be, which was tricky and a mess.

@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Feb 8, 2026

Updates:

  • Loading a package that's being precompiled in the background now makes it dip into monitor the precompile work, and exit when that package finishes, to continue to load it
  • Added key aliases
  • Added a help message

@IanButterworth IanButterworth force-pushed the ib/background_precompile branch from fa33310 to ee030d5 Compare February 9, 2026 15:43
IanButterworth
IanButterworth commented Feb 9, 2026
View reviewed changes
base/stream.jl
sendbuf::Union{IOBuffer, Nothing}
lock::ReentrantLock # advisory lock
throttle::Int
raw_lock::ReentrantLock # exclusive access to raw mode
Copy link
Copy Markdown
Member Author

@IanButterworth IanButterworth Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vtjnash Base.runtests has its own ctrl-c key listener that sets raw mode on stdin, and that was crashing into the precompilepkgs key listener during CI, so I added this lock. Maybe there should be a raw!(f, io) method that takes the lock

@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Feb 14, 2026

This now merges new precompile requests into existing precompile runs, rather than chaining the jobs together serially.

Given it was quite a big change I took the opportunity to refactor things including breaking large functions up by using a new PrecompileSession state.

@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 2 times, most recently from e802f06 to d8f1e1f Compare February 14, 2026 05:52
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch from 42351ce to cef1e6d Compare February 14, 2026 14:07
@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Feb 15, 2026

Now with a verbose mode (via v) with stats

Screen.Recording.2026-02-14.at.10.20.45.PM.mov

@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 2 times, most recently from 34da580 to 695cebd Compare February 15, 2026 20:51
@vtjnash vtjnash mentioned this pull request Feb 18, 2026
Open
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 2 times, most recently from 72106bf to 96a9e31 Compare February 27, 2026 00:26
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch 2 times, most recently from 547dd41 to ec9df6d Compare March 15, 2026 23:47
@IanButterworth IanButterworth mentioned this pull request Mar 16, 2026
Merged
@IanButterworth @claude
precompilation: run precompilation in background task with keyboard c…
92c7602 
…ontrols

Refactor the precompilation system to support running package precompilation
as a detachable background task, with interactive keyboard controls for
managing the process.

The core precompilation logic in base/precompilation.jl is restructured around
a PrecompileSession struct that encapsulates all state for a precompilation run,
including the dependency graph, job tracking, I/O, progress display, and
configuration. The previously monolithic _precompilepkgs() function is broken
into focused functions: build_dep_graph(), filter_dep_graph!(),
detect_circular_deps!(), spawn_precompile_tasks!(), report_precompile_results!(),
and others.

Background precompilation support:
- Precompilation can be detached to run in the background, returning control
  to the REPL while compilation continues.
- A new background precompile monitor displays a spinner and status in the
  REPL prompt area, with the same keyboard controls available.
- Background tasks can be re-attached or monitored, and new package load
  events can inject additional precompilation jobs into a running session.
- Loading (require) waits for any in-progress background precompilation of
  the requested package before attempting to load from cache.

Interactive keyboard controls (when running interactively):
- c: Cancel precompilation by killing subprocesses (with confirmation)
- d: Detach to background, returning to the REPL
- i: Send a profiling signal (SIGINFO/SIGUSR1) for a profile peek
- v: Toggle verbose mode showing elapsed time, CPU%, and memory per worker
- Ctrl-C: Send SIGINT to subprocesses and display their output

Supporting changes:
- base/loading.jl: Add signal_channel and pid_channel parameters to
  compilecache() for subprocess signal forwarding and PID reporting.
  Add wait_for_pending_package() integration point in require.
- base/stream.jl: Add raw_lock to TTY for exclusive raw mode access,
  preventing conflicts between concurrent raw mode users.
- base/process.jl: Add SIGUSR1 and SIGINFO signal constants.
- test/runtests.jl: Update stdin keyboard monitor to use raw_lock and
  add errormonitor for robustness.
- stdlib/Pkg.version: Point to Pkg branch with background precompile support.
- doc/src/manual/performance-tips.md: Document keyboard controls.

Co-authored-by: Claude <noreply@anthropic.com>
@IanButterworth IanButterworth force-pushed the ib/background_precompile branch from ec9df6d to 92c7602 Compare March 16, 2026 23:57
@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Mar 19, 2026

@vtjnash friendly bump for review

@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Apr 1, 2026

I'm going to go ahead with this. The new features I'm sure will benefit from some real world testing on master.

I'll merge with Pkg on this temporary branch, then get JuliaLang/Pkg.jl#4602 in and bumped shortly after.

@IanButterworth IanButterworth merged commit cb2e1ec into JuliaLang:master Apr 1, 2026
8 checks passed
@IanButterworth IanButterworth deleted the ib/background_precompile branch April 1, 2026 13:23
@nsajko
Copy link
Copy Markdown
Member

nsajko commented Apr 1, 2026

This change causes precompilation to make the cursor disappear in my XTerm. Furthermore, the cursor is still gone after exiting julia. A workaround is to exit both Julia and XTerm after each precompilation.

@IanButterworth IanButterworth mentioned this pull request Apr 1, 2026
Open
@IanButterworth
Copy link
Copy Markdown
Member Author

IanButterworth commented Apr 1, 2026

Not the fix, but a general mitigation for this kind of bug #61469

@IanButterworth IanButterworth mentioned this pull request Apr 1, 2026
Merged
@topolarity topolarity mentioned this pull request Apr 7, 2026
Closed
@IanButterworth IanButterworth mentioned this pull request Apr 8, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vtjnash vtjnash Awaiting requested review from vtjnash

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Pkg.precompile() needs to be moved to background task

3 participants

@IanButterworth @vtjnash @nsajko