Skip to content

fix: Make the sccache compile cache actually cache - #13296

Merged
anthonyshew merged 1 commit into
mainfrom
shew/sccache-proxy-webdav-writes
Jul 7, 2026
Merged

anthonyshew merged 1 commit into
mainfrom
shew/sccache-proxy-webdav-writes

Conversation

@anthonyshew

@anthonyshew anthonyshew commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Why

End-to-end dogfooding against the real Remote Cache (which CI cannot do on PRs — placeholder credentials) surfaced two bugs that made the compile cache pure overhead: a cold 430-crate build produced 812 cache write errors and a 0% hit rate, so the follow-up build paid per-object network misses on top of full compilation.

  1. The proxy didn't speak opendal's write dialect. sccache's storage client stats ancestor "directories" with PROPFIND and creates them with MKCOL before any PUT. The proxy only routed GET/HEAD/PUT, so storage initialization failed and every write died without reaching a handler — silently, since unrouted methods produce no proxy logs. The existing tests missed it because they drove the proxy with hand-rolled HTTP instead of the real client.
  2. sccache's background server used the global default port (4226). Any user- or CI-image-managed sccache server already on that port captures turbo's wrapper traffic with whatever storage it was started with — silently bypassing the Remote Cache (this exact hijack corrupted the first measurement attempt).

What

  • The proxy serves the webdav write surface: MKCOL always succeeds (collections are virtual — the Remote Cache is a flat object store), PROPFIND answers directory-shaped paths as existing collections and object paths from artifact_exists, with the minimal multistatus body opendal's parser requires. Routing moved to a single method dispatcher (also fixes the unmatched root path /).
  • SCCACHE_SERVER_PORT is injected alongside the other wrapper env, derived per repository in a range disjoint from the proxy's.
  • Root turbo.json joins the native-lib CI path filter, so future-flag changes run js_native_packages — the job that exercises Cargo tasks (the dogfood PR's job was silently skipped without this).

How

Regression coverage now goes through the genuine article: a dev-dep on opendal 0.55 (sccache's exact webdav client version) doing the probe-write/fan-out-key write/read cycle against the proxy + mock API. That test fails on the old router with the exact production error (missing field getlastmodified → storage init failure).

Measured locally (M-series macOS, debug-build turbo as the wrapper, production Remote Cache) on a 430-crate cold build, cargo clean between runs, --force — CI cannot produce these numbers until this ships in a canary, since PR runs use placeholder credentials by design:

wall compile cache
populate 107s 406 misses stored, 0 write errors
rebuild 42s 406/406 hits (100%)

~2.5x — with an unoptimized debug turbo as the wrapper; release binaries in CI should do better. The rebuild also validated endpoint stability: its wrapper invocations were served by the persistent server spawned in the first run.

First GitHub-Actions-visible verification will be consecutive push-to-main runs of js_native_packages once this is in the canary the dogfood PR (#13292) runs on: look for sccache compile cache proxy listening in the logs and compare turborepo-napi#build wall time across runs.

Dogfooding against the real Remote Cache surfaced two bugs that turned
the compile cache into pure overhead (100% write failures, 0% hits):

- The proxy only routed GET/HEAD/PUT, but opendal (sccache's storage
  client) stats ancestors with PROPFIND and creates them with MKCOL
  before every PUT. Storage init failed and all 812 writes in a cold
  build errored without reaching a handler. The proxy now serves the
  webdav write surface (virtual collections, minimal multistatus), and
  a regression test drives it through the real opendal client.
- sccache's background server defaulted to the global port 4226, where
  any user- or image-managed sccache server captures turbo's wrapper
  traffic with whatever storage it was started with. Inject a
  repo-derived SCCACHE_SERVER_PORT alongside the other wrapper env.

Also includes root turbo.json in the native-lib CI path filter so flag
changes run the job that exercises Cargo tasks.

Measured on a 430-crate cold build (cargo clean between runs, --force,
real Remote Cache): 107s populating, 42s with 406/406 compile cache
hits — ~2.5x, with a debug-build wrapper.
@anthonyshew
anthonyshew requested a review from a team as a code owner July 7, 2026 01:40
@anthonyshew
anthonyshew requested review from tknickman and removed request for a team July 7, 2026 01:40
@vercel

vercel Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-designsystem-docs Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-gatsby-web Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-kitchensink-blog Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-nonmonorepo Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-svelte-web Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-tailwind-web Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
examples-vite-web Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am
turbo-site Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 1:41am

@anthonyshew
anthonyshew merged commit 2ac099c into main Jul 7, 2026
42 of 43 checks passed
@anthonyshew
anthonyshew deleted the shew/sccache-proxy-webdav-writes branch July 7, 2026 02:00
anthonyshew pushed a commit that referenced this pull request Jul 7, 2026
## Release v2.10.5-canary.2

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/28836211177)

### Changes

- release(turborepo): 2.10.5-canary.1 (#13294) (`e6cd498`)
- fix: Pin a flag-aware turbo canary for the eve-agent deployment
(#13295) (`e60a4bd`)
- fix: Make the sccache compile cache actually cache (#13296)
(`2ac099c`)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew pushed a commit that referenced this pull request Jul 13, 2026
## Release v2.10.5

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/29267191097)

### Changes

- perf: Evaluate simple include globs without wax compilation (#13285)
(`189897a`)
- chore: Dogfood native Cargo support in this repository (#13283)
(`42f067b`)
- release(turborepo): 2.10.4 (#13286) (`f2fce38`)
- ci: Remove dead sccache configuration (#13289) (`558df3f`)
- feat: Serve the Remote Cache as an sccache backend for Cargo tasks
(#13288) (`0d9803f`)
- fix: Reject output path traversal (#13290) (`733ccca`)
- fix: Disable the sccache proxy when remote cache use is off (#13291)
(`3249e22`)
- feat: Embed sccache so the Cargo compile cache needs no installation
(#13293) (`b6d0035`)
- release(turborepo): 2.10.5-canary.1 (#13294) (`e6cd498`)
- fix: Pin a flag-aware turbo canary for the eve-agent deployment
(#13295) (`e60a4bd`)
- fix: Make the sccache compile cache actually cache (#13296)
(`2ac099c`)
- release(turborepo): 2.10.5-canary.2 (#13297) (`e6e6fb6`)
- fix: Stop ambient CARGO_INCREMENTAL from suppressing compile cache
injection (#13298) (`784af75`)
- fix: Never let compile cache storage failures fail the build (#13299)
(`a6fc6c5`)
- test: Add outputs path-traversal negative-case regression tests
(#13300) (`2ff6df9`)
- release(turborepo): 2.10.5-canary.3 (#13302) (`7e44a29`)
- ci: Authenticate to Remote Cache via OIDC token exchange (#13303)
(`a86839c`)
- fix: Keep pnpm patches with version range keys during prune (#13307)
(`3c27a89`)
- test: Scrub ambient turbo configuration from integration test children
(#13306) (`b03d0d6`)
- fix: Show toolchain tasks in the TUI and never run in silence (#13308)
(`d7622b6`)
- ci: Fix change detection on push events (#13304) (`eec3d61`)
- refactor: Rename the Cargo toolchain id to rust (#13311) (`a548b02`)
- fix: Remove extraneous bun.lock entries during prune (#13317)
(`382e9f5`)
- feat: Require a user-declared name for the Cargo workspace package
(#13312) (`5f01746`)
- feat: Parse and validate the task command field (#13313) (`389ea49`)
- chore: Harden turbo-vsc to invoke turbo without a shell (#13319)
(`4a19b6e`)
- feat: Resolve and execute task command overrides (#13315) (`a549baa`)
- release(turborepo): 2.10.5-canary.4 (#13325) (`6626261`)
- feat: Run the Rust workspace tests through nextest via command
override (#13316) (`6711bbc`)
- feat: Engage the Remote Cache and sccache compile cache for Rust CI
(#13292) (`1ae2065`)
- ci: Pin GitHub Actions to full commit SHAs (#13143) (`ad614d6`)
- fix: Make npm prune rehoisting deterministic and complete (#13323)
(`2dac737`)
- feat: Report incremental cache reuse in the run summary (#13327)
(`9f3d24a`)
- chore: Remove planning docs (#13329) (`5cd1d02`)
- docs: Migrate docs site to @vercel/geistdocs package (#13320)
(`5517bb2`)
- ci: Update Remote Cache action (#13330) (`4a39887`)
- release(turborepo): 2.10.5-canary.5 (#13331) (`f699719`)
- ci: Remove path-based workflow scheduling (#13332) (`84f2b2c`)
- fix: Resolve EADDRINUSE in kitchen-sink api dev script (#13328)
(`5886c71`)
- ci: Disable telemetry messages in workflows (#13334) (`2218dea`)
- fix: Prevent Cargo run tasks from being cached (#13335) (`68f449b`)
- fix: Isolate Cargo cache by host platform (#13337) (`c71f5cd`)
- fix: Continue TUI text selection beyond viewport (#13338) (`9cbfd90`)
- ci: Dogfood Cargo target restoration (#13336) (`8ad90a7`)
- fix: Require current Cargo lockfiles for caching (#13339) (`3a3d381`)
- fix: Reject unsupported Cargo local packages (#13340) (`708d656`)
- fix: Allow outputs outside package roots (#13342) (`de6a0d3`)
- fix: Resolve Cargo lock dependencies by source (#13343) (`2223a33`)
- docs: Update Cargo workspace support (#13349) (`241ada8`)
- fix: Build Ghostty for baseline CPUs (#13352) (`a2a04cc`)
- fix: Preserve watch rerun semantics for task inputs (#13351)
(`6ed5eab`)
- fix: Isolate Command Overrides From Toolchain Cache I/O (#13354)
(`e76f0b4`)
- release(turborepo): 2.10.5-canary.6 (#13355) (`f82e2c7`)
- fix: Include patched Ghostty crate in Cargo workspace (#13357)
(`b3cd7a1`)
- ci: Shard Rust tests across runners (#13356) (`ef122d9`)
- fix: Hash Cargo build environment inputs (#13348) (`d533266`)
- fix: Synchronize Cargo prune Docker lockfile (#13350) (`ed17249`)
- chore: Update agents app Eve dependency (#13364) (`38aa7d2`)
- fix: Preserve Yarn package extension ranges when pruning (#13363)
(`6ed2fb4`)
- refactor: Add `Toolchain` output availability (#13360) (`82bcfb6`)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch had an error being deployed

1 failed deployment
Preview – turborepo-eve-agent 9fd48a78 Deployed Jul 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant