Skip to content

fix: Back off remote artifact requests during outages - #14222

Merged
anthonyshew merged 3 commits into
vercel:mainfrom
anthonyshew:shew/backoff-remote-cache-outages
Sep 25, 2026
Merged

anthonyshew merged 3 commits into
vercel:mainfrom
anthonyshew:shew/backoff-remote-cache-outages

Conversation

@anthonyshew

Copy link
Copy Markdown
Contributor

Summary

  • Open a short-lived, per-run remote-cache circuit after repeated artifact server or connectivity failures instead of repeating failing requests for every task.
  • Permit one probe after cooldown and restore remote caching on recovery; keep the permanent forbidden-response disable policy separate.
  • Keep local reads and writes available during an outage. A remote-only write still reports an error rather than claiming to have cached the artifact.
remote failures → cooldown (use local cache) → one probe
                                       ├→ success: resume remote cache
                                       └→ failure: another cooldown

A recovered service may not be used until the cooldown expires.

@anthonyshew
anthonyshew requested review from a team and tknickman September 24, 2026 19:57
@vercel

vercel Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

@anthonyshew is attempting to deploy a commit to the Internal Apps Team on Vercel.

A member of the Team first needs to authorize it.

@anthonyshew
anthonyshew merged commit 103d81c into vercel:main Sep 25, 2026
37 of 47 checks passed
github-actions Bot added a commit that referenced this pull request Sep 25, 2026
## Release v2.11.5-canary.1

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/36081266558)

### Changes

- chore: Release Turborepo 2.11.3-canary.4 (#14162) (`fb2fb0c`)
- fix: Stop with tasks when parent exits (#14163) (`fde6637`)
- fix: Respect negated global dependencies in affected detection
(#14164) (`83f6d32`)
- chore: Release Turborepo 2.11.3 (#14165) (`53629a0`)
- test: Share Go compilation caches across integration tests (#14166)
(`684edb6`)
- refactor: Add injectable manifest loading to repository graphs
(#14167) (`bfa7ee1`)
- test: Move Go and Cargo scope planning into crate tests (#14168)
(`17bed92`)
- refactor: Extract subprocess-free Go observation assembly (#14169)
(`6aa7584`)
- test: Move Cargo environment hash contracts into crates (#14170)
(`417e7b0`)
- test: Move Cargo output layout matrices to repository contracts
(#14171) (`2f5f2ad`)
- test: Cover Go discovery identities in memory (#14172) (`133dfd1`)
- test: Cover Go hash inputs in repository contracts (#14174)
(`bf40164`)
- test: Move uv hash input contracts into crates (#14175) (`4681950`)
- test: Cover Go task selection in the engine (#14173) (`63960c5`)
- test: Inject uv discovery for native task contracts (#14176)
(`22d279f`)
- test: Inject query contracts over repository graphs (#14177)
(`5cef46a`)
- test: Inject dry-run task summary contracts (#14178) (`526b026`)
- test: Expose prune workspace selection plan (#14179) (`a05ba5d`)
- test: Inject change detection at scope entry point (#14180)
(`91bb563`)
- test: Move uv lock affectedness planning into repository contracts
(#14184) (`8976bb8`)
- test: Use virtual clock for package watcher events (#14181)
(`efd2a5b`)
- test: Move Cargo task query and dry-run planning into contracts
(#14185) (`34cad2f`)
- test: Move uv quality query and dry-run projections into contracts
(#14186) (`a895310`)
- test: Move JavaScript hash and framework projections into contracts
(#14188) (`b07c3b7`)
- test: Cover prune closure and native layout plans in process (#14189)
(`1f20198`)
- test: Cover mixed-toolchain scope filters through injection (#14190)
(`a9345fb`)
- test: Make package watcher suppression and rediscovery deterministic
(#14191) (`3c533c3`)
- test: Inject watch run lifecycle and event collection (#14192)
(`cf435db`)
- test: Move recursive root-script guard into process-free contract
(#14193) (`86fc080`)
- test: Move Go query and summary parity into crate contracts (#14187)
(`d5602e6`)
- test: Remove duplicate CLI filter matrix cases (#14194) (`9679df3`)
- test: Trim duplicate prune materialization cases (#14195) (`4f88f0f`)
- test: Match Go library summary path on Windows (#14197) (`03a2700`)
- test: Cover mixed-toolchain watch task selection in process (#14196)
(`6b6a46e`)
- test: Preserve Windows SystemRoot in strict-env fixture (#14198)
(`8d74590`)
- test: Retain real watch smokes after selection migration (#14199)
(`e27a4e7`)
- fix: Only reject secondary `go.work` files Go reads for workspace
members (#14183) (`69cc2c4`)
- test: Split Go north-star cache invalidation into hash contracts
(#14201) (`2441e33`)
- test: Avoid deleting built prune output in cross-toolchain prune test
(#14202) (`a4caf0b`)
- test: Move turbo.json validation cases into crate contracts (#14203)
(`59f0fc3`)
- chore: Update with-solid example (#14200) (`aef1e05`)
- refactor: Split RunBuilder execution context construction into phases
(#14204) (`fb37817`)
- refactor: Inject remote cache status probe (#14205) (`b58b280`)
- refactor: Inject inputs into config resolution (#14206) (`01f1a11`)
- feat: Add affected query APIs to turborepo-query (#14207) (`6b7e675`)
- refactor: Wire CLI through affected query APIs (#14208) (`742fe18`)
- feat: Add pure microfrontend port resolver (#14209) (`7d70cab`)
- refactor: Inject change detection into affected run paths (#14210)
(`6d0afae`)
- test: Move affectedTasks planning cases into query contracts (#14214)
(`7871007`)
- refactor: Route get-mfe-port through the pure resolver (#14215)
(`3826a04`)
- test: Count native discovery for JavaScript-only selections (#14216)
(`4fa1096`)
- fix: Disable remote cache after unrecoverable forbidden responses
(#14218) (`a0a5dff`)
- perf: Send cache analytics in larger batches (#14219) (`96e20b6`)
- perf: Batch dry-run remote cache existence checks (#14220) (`e1ab2d6`)
- test: Move config precedence cases into funnel contracts (#14217)
(`935c9ea`)
- test: Move affected run planning into RunBuilder contracts (#14212)
(`ec329a0`)
- fix: Coordinate artifact requests after rate limiting (#14221)
(`7353b16`)
- fix: Share concurrent remote-cache token recovery (#14223) (`813bed0`)
- refactor: Make repository context inputs injectable (#14224)
(`58a4889`)
- refactor: Group Run state into execution and services contexts
(#14225) (`4b72ac9`)
- test: Move Cargo prune layout checks into in-process plan contracts
(#14226) (`8a9a10a`)
- test: Move Cargo and Go discovery cases into contracts (#14227)
(`4d72915`)
- fix: Put version in `turbo query` JSON output instead of a separate
banner (#14229) (`f74650c`)
- chore: Release Turborepo 2.11.4 (#14231) (`5d2e55f`)
- test: Move Python opt-in hint into filter contracts (#14228)
(`f5b6fea`)
- test: Move dependency-output summary assertion into contract (#14230)
(`3ee0227`)
- test: Move dependency-output selection and validation cases to engine
contracts (#14232) (`9bd011f`)
- test: Move Cargo exclude-only scope coverage (#14238) (`c99ad87`)
- test: Move Cargo cache-authority cases to contracts (#14240)
(`1ad8190`)
- test: Move JIT dependency graph scenario into engine contract (#14235)
(`e427a23`)
- test: Move dependency-output hash cases into task-hash contracts
(#14233) (`c17b62c`)
- test: Move Cargo task-filter cases into crate contracts (#14239)
(`fa6b8db`)
- test: Move JIT input hash timing cases into task-hash contracts
(#14234) (`c928e42`)
- test: Move package input normalization cases into engine contracts
(#14236) (`e6c6d77`)
- test: Move structured startup and JIT input cases into contracts
(#14237) (`a29c843`)
- fix: Back off remote artifact requests during outages (#14222)
(`103d81c`)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot added a commit that referenced this pull request Sep 28, 2026
## Release v2.11.5

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/36361927571)

### Changes

- chore: Release Turborepo 2.11.4 (#14231) (`5d2e55f`)
- test: Move Python opt-in hint into filter contracts (#14228)
(`f5b6fea`)
- test: Move dependency-output summary assertion into contract (#14230)
(`3ee0227`)
- test: Move dependency-output selection and validation cases to engine
contracts (#14232) (`9bd011f`)
- test: Move Cargo exclude-only scope coverage (#14238) (`c99ad87`)
- test: Move Cargo cache-authority cases to contracts (#14240)
(`1ad8190`)
- test: Move JIT dependency graph scenario into engine contract (#14235)
(`e427a23`)
- test: Move dependency-output hash cases into task-hash contracts
(#14233) (`c17b62c`)
- test: Move Cargo task-filter cases into crate contracts (#14239)
(`fa6b8db`)
- test: Move JIT input hash timing cases into task-hash contracts
(#14234) (`c928e42`)
- test: Move package input normalization cases into engine contracts
(#14236) (`e6c6d77`)
- test: Move structured startup and JIT input cases into contracts
(#14237) (`a29c843`)
- fix: Back off remote artifact requests during outages (#14222)
(`103d81c`)
- chore: Release Turborepo 2.11.5-canary.1 (#14245) (`d5e648b`)
- test: Make new-package lockfile filter deterministic (#14241)
(`9eb77e6`)
- test: Trim redundant Go cache builds (#14242) (`02e8359`)
- test: Reduce Go versioned-name test matrix (#14243) (`98282b7`)
- test: Move Go binary checks into contracts (#14246) (`16a3157`)
- test: Consolidate Go format smokes (#14244) (`b92aa7e`)
- test: Reuse Cargo build artifacts safely (#14247) (`b46920c`)
- test: Separate prune planning matrix from buildability smoke (#14248)
(`5911e14`)
- feat: Bundle docs in `turbo` npm package (#14108) (`b19f6a5`)
- chore: Release Turborepo 2.11.5-canary.2 (#14249) (`ce88467`)
- fix: Stabilize pnpm per-workspace lockfile hashes (#14253) (`c42dc53`)
- chore: Upgrade factory Next.js to 16.3.3 (#14254) (`1015b29`)
- fix: Update js-yaml to address CVE-2026-84375 (#14255) (`2d4ce79`)
- chore: Upgrade Next.js in docs and factory for CVE-2026-94545 (#14256)
(`47eacd0`)
- fix: Remove unmaintained proc-macro-error2 (#14257) (`8881df1`)
- refactor: Replace clap in production CLIs with usage-rs (#14259)
(`7c5e794`)
- chore: Remove tiny-gradient dependency (#14260) (`79d6521`)
- refactor: Replace human-panic with local crash reporting (#14261)
(`629fb03`)
- chore: Remove unused struct_iterable dependency (#14258) (`26080f2`)
- fix: Sign complete on-disk artifacts after local cache writes (#14265)
(`f21a507`)
- chore: Remove `derive_setters` dependency (#14266) (`0dd88c1`)
- chore: Remove port_scanner dependency (#14267) (`275ac3e`)
- chore: Remove unused Rust dependency declarations (#14268) (`40beaa7`)
- refactor: Remove unused public Rust APIs (#14269) (`4228cf1`)
- refactor: Use workspace Rust edition across crates (#14270)
(`ba06e40`)
- chore: Clean up Rust Clippy exceptions (#14271) (`d839f58`)
- refactor: Group task-hash and uv generation context (#14279)
(`937abd5`)
- refactor: Remove unused global hash wrapper (#14281) (`dc04225`)
- refactor: Remove unused file-hash telemetry argument (#14282)
(`14ef7dd`)
- refactor: Remove dead CLI code and crate-wide allowance (#14283)
(`71f5689`)
- refactor: Remove dead Microfrontends config code (#14284) (`f175cd9`)
- refactor: Remove unused UI task types and dead-code allowance (#14286)
(`0c74400`)
- refactor: Remove unused Cache archive test helper and dead code
allowance (#14287) (`0ec6610`)
- chore: Narrow Run dead-code allowances (#14285) (`91d2df2`)
- chore: Remove resolved TODO markers (#14288) (`ed9a3e9`)
- fix: Use map_while for invariant text prefixes (#14289) (`7c37556`)
- test: Cover globwalk parent traversal (#14290) (`81d92e0`)
- fix: Wait for watcher setup with Tokio wait_for (#14292) (`3e7b084`)
- test: Exercise SCM package hashing for symlinks (#14291) (`355fd86`)
- fix: Reject empty tokens when reading token files (#14293) (`9b91670`)
- test: Verify Yarn transparent workspace handling (#14294) (`4bbb6a9`)
- fix: Format pnpm v5 package names (#14295) (`0cf7ad3`)
- fix: Report package manager error source paths (#14296) (`6f7a147`)
- fix: Resolve LSP task references with JSONC AST (#14298) (`06ffea4`)
- chore: Remove stale package watcher TODO (#14299) (`007bd7e`)
- fix: Validate scope selector glob bases (#14300) (`cc7adf0`)
- fix: Add path context to globwatch errors (#14301) (`e4cfcaa`)
- docs: Document wax pattern and walk semantics (#14302) (`9b16351`)
- chore: Remove resolved filewatch input-set TODO (#14303) (`dbf46c4`)
- docs: Document FSEvents event translation (#14305) (`504c6d3`)
- refactor: Group run-cache task context (#14304) (`e0d8d5d`)
- chore: Document package watcher task-input filtering (#14306)
(`4df5a4f`)
- chore: Remove stale const_format lint workaround (#14307) (`6d4cda1`)
- refactor: Route regex groups through Grouping (#14308) (`6af1efe`)
- perf: Parse character classes without compiling regex (#14310)
(`6ee40f9`)
- chore: Drop unused Biome dependencies from turbopath (#14311)
(`2950a60`)
- chore: Unify jsonc-parser versions (#14312) (`9e48922`)
- chore: Remove error-only dependency edges (#14313) (`5d22e3e`)
- perf: Emit casing flags only when they change (#14309) (`b571578`)
- perf: Avoid formatting Bun lockfiles during parsing (#14314)
(`9277ef3`)
- test: Assert process manager stop postconditions (#14297) (`b06334b`)
- perf: Serve a static GraphiQL page (#14315) (`14ddc92`)
- perf: Upgrade serde_json to reduce CLI size (#14316) (`dc15bd0`)
- perf: Upgrade Ratatui to reduce TUI allocations (#14317) (`1f0308d`)
- fix: Restore TUI log streaming after Ratatui upgrade (#14318)
(`9c53011`)
- feat: Add version-matched agent guidance (#14250) (`1405272`)
- fix: Restore Rust builds after agent guidance changes (#14319)
(`956a35b`)
- chore: Release Turborepo 2.11.5-canary.3 (#14320) (`c76ac66`)
- docs: Update Turborepo schema URL example (#14321) (`d6b852c`)
- feat: Publish versioned standalone turbo archives (#14322) (`7f57aa3`)
- fix: Run standalone archive job on manual releases (#14323)
(`8427e6e`)
- chore: Release Turborepo 2.11.5-canary.4 (#14324) (`a46f82a`)
- feat: Add standalone installers URLs for `curl | bash` (#14325)
(`5108f6c`)
- fix: Isolate installer test environment (#14326) (`8cb6766`)
- fix: Pass repository to release asset commands (#14327) (`c9602f3`)
- chore: Release Turborepo 2.11.5-canary.5 (#14328) (`a50ee65`)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant