Do not lock the artifact-dir for check builds + fix uplifting #16307
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
A-layout
epage
approved these changes
Dec 1, 2025
rustbot
removed
the
S-waiting-on-review
This was referenced Dec 2, 2025
bors
added a commit
to rust-lang/rust
that referenced
this pull request
Dec 3, 2025
Update cargo submodule 14 commits in 2a7c4960677971f88458b0f8b461a866836dff59..bd979347d814dfe03bba124165dbce9554d0b4d8 2025-11-25 19:58:07 +0000 to 2025-12-02 16:03:50 +0000 - fix(completion): Put host-tuple before actual tuples (rust-lang/cargo#16327) - fix(lints): use plural form correctly (rust-lang/cargo#16324) - fix(completions): include `all` in `cargo tree --target` candidates (rust-lang/cargo#16322) - fix(lints): show lint error number (rust-lang/cargo#16320) - chore(deps): update compatible (rust-lang/cargo#16318) - chore(deps): update crate-ci/typos action to v1.40.0 (rust-lang/cargo#16316) - Do not lock the artifact-dir for check builds + fix uplifting (rust-lang/cargo#16307) - Properly validate crate names in `cargo install` (rust-lang/cargo#16314) - Support --filter-platform=host for cargo metadata rust-lang/cargo#9423 (rust-lang/cargo#16312) - Update to mdbook 0.5 (rust-lang/cargo#16292) - refactor(clean): Better divide old / new layout (rust-lang/cargo#16304) - update: silent failure on non-matching package specs with --breaking (rust-lang/cargo#16276) - fix(log): break timing-info message to multiple (rust-lang/cargo#16303) - fix(clean): Clean hosts builds with new layout (rust-lang/cargo#16300)
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 3, 2025
### What does this PR try to resolve? Previously if rustc version mismatches, Cargo removes all `doc` directories including target platforms that are not part of the build. This PR makes it `--target` aware and stops excessive cleanup, by putting `.rustdoc_fingerprint.json` in each target platform directory. ### How to test and review this PR? I'd like to reuse this file to track doc parts directories in <#16309>, and noticed that this file is for the entire workspace rather than per-target, hich is not compatible with mergeable cross-crate info JSONs. For concurrent safety, `build-dir` should be locked already since Cargo locks every intent except check (see <#16307>). This file is touched only under `UserIntent::Doc`.
This was referenced Dec 29, 2025
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 30, 2025
This PR adds fine grain locking for the build cache using build unit level locking. I'd recommend reading the design details in this description and then reviewing commit by commit. Part of #4282 Previous attempt: #16089 ## Design decisions / rational - Still hold `build-dir/<profile>/.cargo-lock` - to protect against `cargo clean` (exclusive) - changed from exclusive to shared for builds - Using build unit level locking with a single lock per build unit. - Before checking fingerprint freshness we take a shared lock. This prevents reading a fingerprint while another build is active. - For units that are dirty, when the job server queues the job we take an exclusive lock to prevent others from reading while we compile. - This is done by dropping the shared lock and then acquiring an exclusive lock, rather than downgrading the lock, to protect against deadlock, see #16155 (comment) - After the unit's compilation is complete, we downgrade back to a shared lock allowing other readers. - All locks are released at the end of the entire build process - artifact-dir was handled in #16307. For the rational for this design see the discussion [#t-cargo > Build cache and locking design @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Build.20cache.20and.20locking.20design/near/561677181) ## Open Questions - [ ] Do we need rlimit checks and dynamic rlimits? #16155 (comment) - [ ] Proper handling of blocking message (#16155 (comment)) - Update Dec 18 2025: With updated impl, we now get the blocking message when taking the initial shared lock, but we get no message when taking the exclusive lock right before compiling. - [ ] Reduce parallelism when blocking - [x] How do we want to handle locking on the artifact directory? - We could simply continue using coarse grain locking, locking and unlocking when files are uplifted. - One downside of locking/unlocking multiple times per invocation is that artifact-dir is touch many times across the compilation process (for example, there is a pre-rustc [clean up step](https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/mod.rs#L402) Also we need to take into account other commands like `cargo doc` - Another option would to only take a lock on the artifact-dir for commands that we know will uplift files. (e.g. `cargo check` would not take a lock artifact-dir but `cargo build` would). This would mean that 2 `cargo build` invocations would not run in parallel because one of them would hold the lock artifact-dir (blocking the other). This might actually be ideal to avoid 2 instances fighting over the CPU while recompiling the same crates. - Solved by #16307 - [ ] What should our testing strategy for locking be? - My testing strategy thus far has been to run cargo on dummy projects to verify the locking. - For the max file descriptor testing, I have been using the Zed codebase as a testbed as it has over 1,500 build units which is more than the default ulimit on my linux system. (I am happy to test this on other large codebase that we think would be good to verify against) - It’s not immediately obvious to me as to how to create repeatable unit tests for this or what those tests should be testing for. - For performance testing, I have been using hyperfine to benchmark builds with and without `-Zbuild-dir-new-layout`. With the current implementation I am not seeing any perf regression on linux but I have yet to test on windows/macos. --- <details><summary>Original Design</summary> - Using build unit level locking instead of a temporary working directory. - After experimenting with multiple approaches, I am currently leaning to towards build unit level locking. - The working directory approach introduces a fair bit of uplifting complexity and I further along I pushed my prototype the more I ran into unexpected issues. - mtime changes in fingerprints due to uplifting/downlifting order - tests/benches need to be ran before being uplifted OR uplifted and locked during execution which leads to more locking design needed. (also running pre-uplift introduces other potential side effects like the path displayed to the user being deleted as its temporary) - The trade off here is that with build unit level locks, we need a more advanced locking mechanism and we will have more open locks at once. - The reason I think this is a worth while trade of is that the locking complexity can largely be contained to single module where the uplifting complexity would be spread through out the cargo codebase anywhere we do uplifting. The increased locks count while unavoidable can be mitigated (see below for more details) - Risk of too many locks (file descriptors) - On Linux 1024 is a fairly common default soft limit. Windows is even lower at 256. - Having 2 locks per build unit makes is possible to hit with a moderate amount of dependencies - There are a few mitigations I could think of for this problem (that are included in this PR) - Increasing the file descriptor limits of based on the number of build units (if hard limit is high enough) - Share file descriptors for shared locks across jobs (within a single process) using a virtual lock - This could be implemented using reference counting. - Falling back to coarse grain locking if some heuristic is not met ### Implementation details - We have a stateful lock per build unit made up of multiple file locks `primary.lock` and `secondary.lock` (see [`locking.rs`](http://locking.rs) module docs for more details on the states) - This is needed to enable pipelined builds - We fall back to coarse grain locking if fine grain locking is determined to be unsafe (see `determine_locking_mode()`) - Fine grain locking continues to take the existing `.cargo-lock` lock as RO shared to continue working with older cargo versions while allowing multiple newer cargo instances to run in parallel. - Locking is disabled on network filesystems. (keeping existing behavior from #2623) - `cargo clean` continues to use coarse grain locking for simplicity. - File descriptors - I added functionality to increase the file descriptors if cargo detects that there will not be enough based on the number of build units in the `UnitGraph`. - If we aren’t able to increase a threshold (currently `number of build units * 10`) we automatically fallback to coarse grain locking and display a warning to the user. - I picked 10 times the number of build units a conservative estimate for now. I think lowering this number may be reasonable. - While testing, I was seeing a peak of ~3,200 open file descriptors while compiling Zed. This is approximately x2 the number of build units. - Without the `RcFileLock` I was seeing peaks of ~12,000 open fds which I felt was quiet high even for a large project like Zed. - We use a global `FileLockInterner` that holds on to the file descriptors (`RcFileLock`) until the end of the process. (We could potentially add it to `JobState` if preferred, it would just be a bit more plumbing) See #16155 (comment) for proposal to transition away from this to the current scheme </details>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR try to resolve?
This PR un-reverts the changes from #16230 as well as adds a bug fix for a crash reported in #16297
Note: The bug fix relies on behavior in the reverted code, so I added that back before the bug fix
How to test and review this PR?
I added a new check test that verifies the existing behavior (crash) and is updated to verify no files are added to the target dir with no crash.
r? @epage
cc: @stewartadam (thanks, again for the bug report)
Fixes #16305