Skip to content

fix: Stale cache hits when --only drops dependency edges - #4

Open
solsson wants to merge 2 commits into
mainfrom
fix-dashdashonly-dependency-hashing
Open

solsson wants to merge 2 commits into
mainfrom
fix-dashdashonly-dependency-hashing

Conversation

@solsson

@solsson solsson commented Jul 27, 2026 •

Copy link
Copy Markdown
Owner

Replaces #3 (now that #1 is no longer needed.

The bug (see #2)

--only discards dependency edges that point outside the filter set. Those tasks never execute, so they never produce a task hash, so nothing about them reaches the downstream task's cache key.

The consequence is not a missing edge in a graph diagram. It is that turbo reports a cache hit and restores a stale artifact after a real source change, with no warning.

packages/types      (source of truth for shared types)
packages/runtime    depends on types; "bundle": { "dependsOn": ["^bundle"] }
$ turbo bundle --only --filter=runtime
runtime#bundle: cache miss, executing 8f3a...

# edit packages/types/src/index.ts -- a real, semantic change

$ turbo bundle --only --filter=runtime
runtime#bundle: cache hit, replaying logs 8f3a...   # <-- same hash. stale bundle.

The second run ships an artifact built against the old types. Nothing in the output suggests anything was skipped. This is the failure mode that motivated the fork: it reached production.

What this does not change

It does not touch the task graph. --only still drops exactly the edges it dropped before, and the set of tasks that execute is unchanged.

That is deliberate. Changing which nodes survive --only would alter documented flag semantics, and it is the part of the codebase most likely to conflict on every upstream rebase. The graph is upstream's; only the cache key is ours.

What it does

For each dropped edge, the engine records it. At hash time the excluded package's source file hashes are folded into the downstream task's dependency hash list, standing in for the task hash that was never produced.

The material is mixed in before the task hash is computed rather than by re-hashing a finished hash. Two consequences worth reviewing:

  • The tracker's recorded hash is compensated too, so dependents of an affected task also invalidate. Compensating after the fact would have fixed the directly-affected task while leaving everything downstream of it stale.
  • It covers every hashing path in one place, including calculate_task_hash_with_deferred_inputs. A task that uses jit/dependencyOutputs and has an edge dropped by --only would otherwise silently skip compensation — which is exactly our bundle task, so this is not hypothetical.

We hash the excluded package's entire tracked file set, not the excluded task's declared inputs. Since that task never runs we cannot know which of its inputs mattered, and over-invalidating is the correct direction for a correctness fix. Worth knowing when reviewing: this is intentionally coarser than a normal task hash.

A warning now names the dropped tasks and suggests the --filter flags that would run them.

Known limitation

A package that does not define the depended-on task is not covered. No ^task edge is ever created for it, so there is no dropped edge to record and nothing to hash.

A types-only package with no bundle script is precisely the case people assume is handled. It is not. Use $TURBO_ROOT$ inputs for those. Fixing this properly means hashing package dependencies that were never task-graph nodes, which is a larger change and a separate PR.

Upstream status

Unaddressed. I diffed crates/turborepo-engine/src/builder.rs between our previous fork base (4b3e7cc96d, release 2.9.10) and v2.10.6: the --only edge-dropping logic is byte-for-byte identical at both sites, still a silent return / continue with no warning and no cache-key contribution.

The v2.10.x deferred-hashing work does not help here. jit and dependencyOutputs change when inputs are hashed, not which edges survive --only; a dropped edge leaves nothing to defer on.

This looks like something upstream simply does not know about. Given they took #8051 seriously, it is worth reporting rather than carrying indefinitely. The tripwire test below is a ready-made repro.

Relationship to the existing PRs

This supersedes all three and is a fresh start from v2.10.6 rather than a rebase.

Deliberately not carried over from v2.9.10-hashdepends.2:

  • The RwLock<HashMap> in turborepo-task-hash. Upstream #13273 removed locking from that path for performance; re-adding it would reintroduce the contention they just eliminated.
  • dep_output_overlap.rs (569 lines), the dispatch-time rehashing and cascade, TaskHashTracker::set_hash, and the <DEFERRED> / <DEPENDS_ON_OUTPUT> dry-run display — all superseded by upstream's PrecomputedTask::Deferred and hash: null + hashReason.

Net effect: the fork's diff against upstream drops from 1,405 lines to 565, and turborepo-hash is no longer modified at all.

Improvements over the previous implementation

  • Fold into the dependency hash list rather than re-hashing the final hash (propagates to dependents; covers all three hash paths).
  • Retain dropped_dependencies in prune_to_reachable, so compensation survives watch and subgraph runs. Previously lost.
  • Memoize per-package hashing — several tasks commonly drop the same package, and hashing walks the whole package tree.
  • Deduplicate dropped edges; a package can be both a topological and a direct dependency.
  • Pass repo_index through so hashing uses the fast path from #13213.
  • Drop the now-unnecessary hash_string helper.

Tests

Three new positive tests, none of which existed before — the previous branch tested none of this behavior:

  • test_engine_tasks_only_records_dropped_topological_dependency — the ^build path
  • test_engine_tasks_only_records_dropped_task_dependency — the explicit a#build path (separate code path in the builder)
  • test_engine_records_no_dropped_dependencies_without_tasks_only — nothing recorded on the common path

Plus test_tasks_only_drops_package_dependency_edges, reworded from a bug report into a deliberate tripwire. It asserts the edge is dropped, phrased as what we would want if edges were preserved, with a pinned panic message. If upstream ever changes its mind, it starts failing and we revisit whether the compensation is still needed.

Verified locally: cargo check clean, cargo clippy --all-targets exit 0 with no new warnings, cargo fmt --check clean, and 129 engine + 39 task-hash + 441 turborepo-lib tests passing.

Not verified

  • No end-to-end run. libghostty-vt-sys (new in 2.10, builds ghostty from source via zig) will not link on macOS 26 — zig 0.15.2 fails to resolve the SDK. Type-checking and unit tests were run with DOCS_RS=1, which skips the native build, so no integration test or real turbo binary was exercised. Smoke-test against a real repo before tagging.
  • The release workflow's zig step is untested. Pinned to 0.15.2: ghostty requires >= 0.15.2 and 0.16 breaks its std.Io.Dir usage, so the window is narrow. First tag push is the real test. fail-fast: false is set so one flaky cross-compile does not sink the whole release.

Merging

Base this on v2.10.6, not the current main. Fork main is still at 2.9.10 and fast-forwards cleanly to v2.10.6 (511 upstream commits, no conflicts) — do that first, otherwise the PR diff shows all 511 of them instead of the single commit.

Release workflow also needs the toolchain pin moved to nightly-2026-05-22 to match v2.10.6; that is included here.

Tag as v2.10.6-onlyuncache.1.

`--only` removes dependency edges pointing outside the filter set. Those
tasks never run, never produce a task hash, and so contribute nothing to
the downstream task's cache key. A change in an excluded package then
produces a cache *hit* and turbo restores a stale artifact.

Upstream treats the edge dropping as intended behavior for `--only`, and
this change does not argue otherwise: the task graph is left exactly as
upstream builds it. Instead the engine records each dropped edge, and the
excluded package's source hashes are folded in where the missing task
hash would have gone.

Fresh start from upstream v2.10.6, squashing what remains relevant from
#3 (previously released as v2.9.10-hashdepends.2).

The dependsOn-output half of that PR is deliberately dropped: upstream
solved it in v2.9.17-v2.10.6 via structured `inputs` modes (vercel#13043,
vercel#13045, vercel#13125, vercel#13127, vercel#13129), closing vercel#8051. Express
those cases with `mode: "jit"` or `mode: "dependencyOutputs"` instead.
Notably vercel#13273 removed locking from task hash precomputation, which is
the opposite of the `RwLock<HashMap>` the old branch carried.

Changes from the previous implementation:

- Fold the stand-in hashes into the task's dependency hash list rather
  than re-hashing the finished task hash. The tracker's copy is then
  compensated too, so dependents of an affected task also invalidate.
  This also covers all three hashing paths at once, including
  `calculate_task_hash_with_deferred_inputs`, which the old post-hoc
  approach would have missed for tasks that use `jit`/`dependencyOutputs`
  and have an edge dropped -- the exact combination in our bundle task.
- Retain `dropped_dependencies` in `prune_to_reachable` so the
  compensation survives watch and subgraph runs.
- Memoize per-package hashing; several tasks commonly drop the same
  package, and hashing walks the whole package tree.
- Deduplicate dropped edges; a package can be both a topological and a
  direct dependency.
- Pass `repo_index` so hashing uses the repo index fast path (vercel#13213).
- Drop the `hash_string` helper, unnecessary under the new approach;
  turborepo-hash is now untouched by the fork.
- Add positive tests that dropped edges are recorded on both the
  topological and direct-dependency paths, plus one asserting nothing is
  recorded without `--only`. The old branch tested none of this.
- Reword the `#[should_panic]` test as a deliberate tripwire rather than
  a bug report, and pin its expected panic message.

Refs: #1, #2, #3

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@solsson
solsson force-pushed the fix-dashdashonly-dependency-hashing branch from 9cf213f to af8b48a Compare July 27, 2026 17:47
Builds `turbo` for our four deploy targets on tag push and attaches them to
the GitHub release.

The setup steps deliberately mirror the `build-rust` job in
turborepo-release.yml and reuse upstream's composite actions
(setup-protoc, setup-capnproto, setup-zig) rather than hand-rolling
toolchain installation.

That is a deliberate correction of how this workflow was maintained on the
previous fork branch, where both CI commits (6c38fe0, df287ef) were
hand-rolled, and df287ef existed only to work around
dtolnay/rust-toolchain fighting rust-toolchain.toml. The first tag built
from a hand-rolled version of this file, v2.10.6-onlyuncache.1, failed on
all four targets: mlugg/setup-zig@v1 returned HTTP 404 for zig 0.15.2.
Upstream's setup-zig already pins that exact version, downloads it from
ziglang.org, and exports ZIG_EXECUTABLE/ZIG_GLOBAL_CACHE_DIR.

Consequences of following upstream here:

- aarch64-unknown-linux-musl gets the linker defsyms upstream added for
  aws-lc-sys (`--defsym=__isoc23_sscanf=sscanf`,
  `--defsym=__isoc23_strtol=strtol`). Without them that link fails, which
  would have been the next failure after zig.
- No toolchain version is pinned in this file; actions-rust-lang respects
  rust-toolchain.toml, so it cannot drift out of sync on rebase and
  df287ef's `rustup target add` workaround is unnecessary.
- protoc and capnproto come from pinned actions, not apt/brew resolution.
- Runners match upstream: macos-15-xlarge and ubuntu-24.04.

`fail-fast: false` plus `always()` on the release job so one failing target
does not suppress publishing the others.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@solsson
solsson force-pushed the fix-dashdashonly-dependency-hashing branch from 232f33c to 190dbb0 Compare July 29, 2026 03:29
@solsson

solsson commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

Regarding failed checks there's vercel-specific GHA stuff. We only did 190dbb0 to get binaries for https://github.com/solsson/turbo/releases/tag/v2.10.6-onlyuncache.1

solsson added a commit to Yolean/ystack that referenced this pull request Jul 29, 2026
@solsson

solsson commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Review from cicd-v1 maintainer, for follow-up work

turbo-fork-pr4-review-from-cicd.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant