fix: Never let compile cache storage failures fail the build - #13299
Merged
Merged
Conversation
…jection The executor suppressed the entire compile-cache injection when any injected key was already present in the task environment. That guard was too blunt: this repository's own setup-environment action exports CARGO_INCREMENTAL=0, which globalPassThroughEnv carries into every task env, so the injection engaged nowhere in CI — the exact environment the CI-only compile cache exists for. Move the composition decision into the toolchain, which is the only layer that knows which pre-existing variables signal a competing configuration versus ambient hygiene: a user-supplied RUSTC_WRAPPER or any SCCACHE_* variable stands the whole set down (injecting on top could hijack that setup's backend), while a pre-set CARGO_INCREMENTAL is tolerated — injection proceeds without overriding the explicit value. The executor now injects whatever the toolchain returns, keeping generic code free of toolchain knowledge.
Two layers, addressing the fatal path found in bench run 4 (storage self-check failure at sccache server startup exited every wrapper invocation with code 2, failing the whole cargo build): - Gate the proxy on the resolved runtime remote-cache status instead of configuration alone. A run that already learned from the preflight that its credentials are bad (or the API is unreachable, or the team is over limit) no longer hands sccache a doomed backend. - Inject SCCACHE_IGNORE_SERVER_IO_ERROR=1 and pick up the fork port of mozilla/sccache#2756: when the server cannot be connected to or started for any reason mid-run, the wrapper warns and runs the compiler directly instead of failing the compilation. Also tightens the CARGO_INCREMENTAL tolerance from the conflict-guard fix: only an ambient value of 0 composes; any other explicit value now stands the injection down, because sccache's wrapper hard-exits when it sees CARGO_INCREMENTAL=1 — which would fail the build.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
# Conflicts: # crates/turborepo-repository/src/cargo.rs
anthonyshew
pushed a commit
that referenced
this pull request
Jul 7, 2026
## Release v2.10.5-canary.3 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/28874392650) ### Changes - release(turborepo): 2.10.5-canary.2 (#13297) (`e6e6fb6`) - fix: Stop ambient CARGO_INCREMENTAL from suppressing compile cache injection (#13298) (`784af75`) - fix: Never let compile cache storage failures fail the build (#13299) (`a6fc6c5`) - test: Add outputs path-traversal negative-case regression tests (#13300) (`2ff6df9`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew
pushed a commit
that referenced
this pull request
Jul 13, 2026
## Release v2.10.5 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/29267191097) ### Changes - perf: Evaluate simple include globs without wax compilation (#13285) (`189897a`) - chore: Dogfood native Cargo support in this repository (#13283) (`42f067b`) - release(turborepo): 2.10.4 (#13286) (`f2fce38`) - ci: Remove dead sccache configuration (#13289) (`558df3f`) - feat: Serve the Remote Cache as an sccache backend for Cargo tasks (#13288) (`0d9803f`) - fix: Reject output path traversal (#13290) (`733ccca`) - fix: Disable the sccache proxy when remote cache use is off (#13291) (`3249e22`) - feat: Embed sccache so the Cargo compile cache needs no installation (#13293) (`b6d0035`) - release(turborepo): 2.10.5-canary.1 (#13294) (`e6cd498`) - fix: Pin a flag-aware turbo canary for the eve-agent deployment (#13295) (`e60a4bd`) - fix: Make the sccache compile cache actually cache (#13296) (`2ac099c`) - release(turborepo): 2.10.5-canary.2 (#13297) (`e6e6fb6`) - fix: Stop ambient CARGO_INCREMENTAL from suppressing compile cache injection (#13298) (`784af75`) - fix: Never let compile cache storage failures fail the build (#13299) (`a6fc6c5`) - test: Add outputs path-traversal negative-case regression tests (#13300) (`2ff6df9`) - release(turborepo): 2.10.5-canary.3 (#13302) (`7e44a29`) - ci: Authenticate to Remote Cache via OIDC token exchange (#13303) (`a86839c`) - fix: Keep pnpm patches with version range keys during prune (#13307) (`3c27a89`) - test: Scrub ambient turbo configuration from integration test children (#13306) (`b03d0d6`) - fix: Show toolchain tasks in the TUI and never run in silence (#13308) (`d7622b6`) - ci: Fix change detection on push events (#13304) (`eec3d61`) - refactor: Rename the Cargo toolchain id to rust (#13311) (`a548b02`) - fix: Remove extraneous bun.lock entries during prune (#13317) (`382e9f5`) - feat: Require a user-declared name for the Cargo workspace package (#13312) (`5f01746`) - feat: Parse and validate the task command field (#13313) (`389ea49`) - chore: Harden turbo-vsc to invoke turbo without a shell (#13319) (`4a19b6e`) - feat: Resolve and execute task command overrides (#13315) (`a549baa`) - release(turborepo): 2.10.5-canary.4 (#13325) (`6626261`) - feat: Run the Rust workspace tests through nextest via command override (#13316) (`6711bbc`) - feat: Engage the Remote Cache and sccache compile cache for Rust CI (#13292) (`1ae2065`) - ci: Pin GitHub Actions to full commit SHAs (#13143) (`ad614d6`) - fix: Make npm prune rehoisting deterministic and complete (#13323) (`2dac737`) - feat: Report incremental cache reuse in the run summary (#13327) (`9f3d24a`) - chore: Remove planning docs (#13329) (`5cd1d02`) - docs: Migrate docs site to @vercel/geistdocs package (#13320) (`5517bb2`) - ci: Update Remote Cache action (#13330) (`4a39887`) - release(turborepo): 2.10.5-canary.5 (#13331) (`f699719`) - ci: Remove path-based workflow scheduling (#13332) (`84f2b2c`) - fix: Resolve EADDRINUSE in kitchen-sink api dev script (#13328) (`5886c71`) - ci: Disable telemetry messages in workflows (#13334) (`2218dea`) - fix: Prevent Cargo run tasks from being cached (#13335) (`68f449b`) - fix: Isolate Cargo cache by host platform (#13337) (`c71f5cd`) - fix: Continue TUI text selection beyond viewport (#13338) (`9cbfd90`) - ci: Dogfood Cargo target restoration (#13336) (`8ad90a7`) - fix: Require current Cargo lockfiles for caching (#13339) (`3a3d381`) - fix: Reject unsupported Cargo local packages (#13340) (`708d656`) - fix: Allow outputs outside package roots (#13342) (`de6a0d3`) - fix: Resolve Cargo lock dependencies by source (#13343) (`2223a33`) - docs: Update Cargo workspace support (#13349) (`241ada8`) - fix: Build Ghostty for baseline CPUs (#13352) (`a2a04cc`) - fix: Preserve watch rerun semantics for task inputs (#13351) (`6ed5eab`) - fix: Isolate Command Overrides From Toolchain Cache I/O (#13354) (`e76f0b4`) - release(turborepo): 2.10.5-canary.6 (#13355) (`f82e2c7`) - fix: Include patched Ghostty crate in Cargo workspace (#13357) (`b3cd7a1`) - ci: Shard Rust tests across runners (#13356) (`ef122d9`) - fix: Hash Cargo build environment inputs (#13348) (`d533266`) - fix: Synchronize Cargo prune Docker lockfile (#13350) (`ed17249`) - chore: Update agents app Eve dependency (#13364) (`38aa7d2`) - fix: Preserve Yarn package extension ranges when pruning (#13363) (`6ed2fb4`) - refactor: Add `Toolchain` output availability (#13360) (`82bcfb6`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bench run 28839285743 exposed the compile cache's worst failure mode: with unauthorized credentials, sccache's background server failed its storage self-check at startup, refused to start, and every wrapper invocation exited 2 — cargo saw failed rustc calls and the build failed. An optimization must degrade to "slower", never to "broken".
What
Two independent layers:
RemoteCacheStatus(which folds in the preflight: auth failure, unreachable API, team over limit/paused) instead of configuration alone. The bench run had literally printedRemote caching unavailable (Authentication failed)before starting the proxy anyway.SCCACHE_IGNORE_SERVER_IO_ERROR=1— now injected — a client that cannot connect to or start the server warns and runs the compiler directly. Covered by two integration tests on the fork (forced startup failure → fallback with the env var, hard error without).Also tightens #13298's
CARGO_INCREMENTALtolerance: reading sccache's wrapper source revealed it hard-exits onCARGO_INCREMENTAL=1, so tolerating arbitrary pre-set values could itself fail builds. Only an ambient0composes; any other explicit value stands the injection down (incremental compilation was deliberately requested and is incompatible with sccache).How
Layer 1 kills the predictable class before any wrapper runs; layer 2 is the guarantee for everything unpredictable. Fork rev bumped to
52bfcd0(vercel/main-from-args). Verified: fork integration tests for the fallback both ways;compile_cache_envmatrix updated (=0tolerated,=1stands down); workspace check/lint/fmt green.