Per-user compiled artifact cache

[djc]

MVP design: cache intermediate artifacts across projects with minimal cache poisoning

• Only immutable items (registry, git, not path)
• Idempotent builds (no build script executions + dependents, no proc macro dependents)

Roadmap:

• Split out a build-dir to communicate out early what will be broken (Redefine CARGO_TARGET_DIR to be only an artifacts directory #14125)
• Re-organize build-dir into cacheable units (Re-organize build-dir by package + hash, rather than artifact type #15010)
• Experiments
  • More granular locking in build-dir to see what could apply to a shared cache (More granular locking in cargo_rustc #4282)
  • GC for build-dir in prep for GC for shared cache (cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026)
  • Implement shared cache without gc and with coarse locking
• Polish shared cache
  • GC
  • granular locking

Parallel efforts to improve cache efficiency

• Reduce the need for users to write build scripts #14948
• Tracking issue for RFC 2196, "metabuild: semantic build scripts for Cargo" #14903
• Tracking Issue for RFC 3698 "Declarative macro_rules! derive macros" (macro_derive) rust#143549
• Tracking Issue for RFC 3697 "Declarative macro_rules! attribute macros" (macro_attr) rust#143547

Future possibilities

• Remote cache plugin system
• Means for proc-macros and build scripts to be cached
  • Pinky promise flag to be cacheable
  • Means for guarenteeing they are only accessing declared resources

Open questions

• How does this interact with cargo clean?
• How does this interact with pipelined builds?

---

I was wondering if anyone has contemplated somehow sharing compiled crates. If I have a number of projects on disk that often have similar dependencies, I'm spending a lot of time recompiling the same packages. (Even correcting for features, compiler flags and compilation profiles.) Would it make sense to store symlinks in ~/.cargo or equivalent pointing to compiled artefacts?

[alexcrichton]

There's been musings about this historically but never any degree of serious consideration. I've always wanted to explore it though! (I think it's definitely plausible)

[aidanhs]

sccache is one option here - it has a local disk cache in addition to the more exotic options to store compiled artifacts in the cloud.

[djc]

sccache would be good for the compilation time part, but it'd be nice to also get a handle on the disk size part of it.

[Eh2406]

cc #6229

[Vlad-Shcherbina]

I think you can put

[build]
target-dir = "/my/shared/target/dir"

in ~/.cargo/config.

But I have no idea if this mode is officially supported. Is it?

[Eh2406]

Yes it is, as is setting it with the corresponding environment variable. However the problems with cargo never deleting the unused artifacts gets to be dramatic quickly. Hence the connection to #6229

[epage]

@joshtriplett and I had a brainstorming session on this at RustNL last week.

It'd be great if cargo could have a very small subset of sccache's logic: per-user caching of intermediate build artifacts. By building this into cargo, we can tie it into all that cargo knows and cane make extensions to better support it.

Risks

• Poisoning the cache from
  • Broken builds (e.g. from incremental compilation bugs)
  • Non-deterministic builds
  • mtime bugs
• Races with parallel builds
  • Performance hits from locking
• Running out of disk space

To mitigate problems with cache poisoning

• Packages must advertise that they are deterministic
  • Long term: Wasm build scripts and proc-macros to ensure determinism
• Initially limit the caching to packages from immutable sources
  • No mtime bugs
  • No need for incremental compilation

As a contingency for if the cache is poisoned, we need a way to clear the cache (see also #3289)

To mitigate running out of disk space, we need a GC / prune (see also #6509)

• One strategy is to clear caches for a rust version that is no longer installed
• We could manually track atime

Locking strategy to mitigate race conditions / locking performance

• Assumptions:
  • Parallel builds are likely
  • Parallel builds of the same package fingerprint are unlikely
• Design:
  1. Does it exist (without lock)
  2. Build outside the cache
  3. If can't do atomic rename, move into a temp dir, then do an atomic rename. If it now exists, just delete the tmpdir
• Read/prune lock
  • Multi-Reader/single-writer lock
  • What about reading it?
    • Hold lock wile building, blocking prunes
    • Could copy out while holding lock to minimize lock time
    • Hold a lock for copy it
    • Or don't bother copying, just block the prune until lock is available
  • Prune by renaming and then deleting

Transition plan (modeled off of sparse registry)

• Steps:
  1. Unstable
  2. Hacky env variable to opt-in (unstable)
  • On stable, warn if env variable is set so people can set it globally and use versions of Rust from Step 2 and Step 3
  4. Hacky env variable to opt-in (stable)
  5. Stable
• Don't need full cleanup/pruning strategy until it is stable

[epage]

Wonder if something like reflink would be useful