Conversation
solsson
commented
May 8, 2026
- Includes all of Include outputs from dependsOn tasks in same turbo run input hashes #1
- Includes the failure expectation from Document cache behavior with --only #2 and a proposed fix
…h-with-dependson-outputs
When a task's inputs match outputs of its dependsOn tasks, defer file hashing to dispatch time -- after dependencies have executed and their output files exist on disk. This replaces the upfront hash computation that saw stale or missing files. Correctly handles: - Deterministic deps: second run is FULL TURBO - Non-deterministic deps (cache:false): dependent misses when output changes - Transitive chains: prepare -> transform -> build Implementation: - Tasks with dep-output overlaps are skipped in precompute_task_hashes() - At dispatch time (after deps executed), compute_deferred_hash() re-hashes the task's input files and computes the full task hash - TaskHasher.hashes uses RwLock to allow updating through &self 4 integration tests: simple caching, transitive caching, config info messages, and non-deterministic output detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Builds turbo binaries for 4 architectures (linux-amd64, linux-arm64, darwin-amd64, darwin-arm64) and publishes them as GitHub release assets with a sha256 checksums file. Triggered by tag push. No npm publish. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a non-deferred task depends on a deferred task, its precomputed hash includes the deferred task's stale dependency hash. At dispatch time, after the deferred task is re-hashed, propagate the rehash to all downstream tasks by tracking which tasks changed. Previously, deferred tasks were skipped entirely in precompute, causing "Missing hash for dependent task" errors when non-deferred downstream tasks tried to compute their dependency hashes. Fix: all tasks precompute normally (with potentially stale file state). At dispatch time, deferred tasks re-hash. Any task with a re-hashed dependency also re-hashes. This cascade is safe because the engine dispatches in dependency order. Adds integration test for the pattern: prepare -> schemas (deferred) -> checks (not deferred). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add config-only detection that identifies tasks whose inputs include files produced by their dependencies' outputs. This pattern causes unstable cache hashes because turbo computes file hashes before dependencies execute. Detection uses exact string matching, glob matching (via wax), and directory containment (bare "dir" input matches "dir/**" output), including through transitive dependency chains. Only same-package dependencies are checked since cross-package tasks write to different directories. Results are deduplicated by task name and shown as info messages in the run prelude, informing the config author regardless of --filter. 15 unit tests cover pattern matching (7), engine-level graph traversal (5), and config-level deduplication (3). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config-level flagging identifies turbo.json patterns where a task's inputs match a dependency's outputs. At runtime, only defer hashing for tasks where the dependency package actually has the script in its package.json. If the dep has no script, the task is a no-op and won't produce output files -- upstream behavior is preserved. This eliminates unnecessary deferred rehashing for packages that inherit root turbo.json task definitions but don't implement the dependency task (e.g. eslint-config-y inheriting schemas/fetch config but having no fetch script). Also renames "detection" to "flagging" in comments and docs to distinguish config-level analysis from runtime behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tasks whose inputs match a dependency's outputs have unreliable precomputed hashes -- the real hash is only known after dependencies execute. Mark these tasks with <DEFERRED> in the TaskHashTracker so dry run output and summaries don't imply a stable cache prediction. The real hash is still used internally for cache operations via task_cache (correct because at dispatch time deps have executed). Uses the same deferred_hash_tasks set for both dry and non-dry modes -- no separate code paths. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The tracker override was applied unconditionally, causing build run summaries to show <DEFERRED> instead of the real hash. Now only applied in dry mode where deps haven't executed and the hash is based on stale file state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace generic <DEFERRED> with <DEPENDS_ON_OUTPUT: dep#task,...> listing which dependency tasks produce the outputs that this task's inputs match. This makes the dry run output actionable -- the user can see exactly which dependency relationship causes deferred hashing. Changes deferred_hash_tasks from HashSet to HashMap<TaskId, Vec<TaskId>> to carry the triggering dependency information through to display. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace all "deferred"/"defer" naming introduced by this branch with "depends-on-output" to clearly indicate the feature's purpose. The term "deferred" was already used in the repo for telemetry, SCM, and logging with generic "do later" semantics. Renames: deferred_hash_tasks -> depends_on_output_tasks, compute_deferred_hash -> compute_depends_on_output_hash, rehashed_tasks -> output_changed_tasks. Debug log entries now use "depends-on-output" prefix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents a cache correctness issue where --only drops ^dependsOn edges even for direct package.json dependencies. This causes stale cache hits when upstream packages change. The test asserts that `turbo bundle --only --filter=runtime` should keep types#bundle in the graph (since runtime depends on types and bundle declares ^bundle). Currently it doesn't — the edge is dropped and changes in types are invisible to the cache key. Marked #[should_panic] because the assertion currently fails. When the engine is fixed to preserve these edges, remove should_panic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When --only filters out a task from the dependency graph, emit a tracing::warn with the dropped edge, the affected task, and a suggestion to add --filter for the missing package. The warnings fire at the exact point where edges are dropped in the engine builder, so they cannot drift from actual behavior. Example output: --only: dropping topological dependency types#bundle from runtime#bundle. Changes in types won't affect the cache key for runtime#bundle. To include it, add --filter=types Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ctness When --only excludes a dependency task from execution, the downstream task's cache key now includes file hashes from the excluded dependency's package directory. This means source changes in dependency packages correctly invalidate the cache even though those packages' tasks don't run. The implementation: - Engine records dropped dependencies (from the warning commit) - Visitor computes SCM file hashes for each dropped dep's package - Hashes are combined with the original task hash via xxh64 This fixes cache staleness for patterns like: turbo bundle --only --filter=checkit-runtime where ^bundle dependencies (e.g. yolean-op) are excluded from execution but their source changes must still invalidate the cache. Limitation: only works for packages that define the depended-on task (e.g. yolean-op has a bundle script). Packages without the task (e.g. yolean-types with no bundle script) have no ^bundle edge to drop, so their files aren't tracked. Use $TURBO_ROOT$ inputs for those cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
48cec6d to
d57d360
Compare
dtolnay/rust-toolchain installs targets for its configured toolchain, but rust-toolchain.toml overrides to a different nightly. Add explicit rustup target add to ensure the cross target is available for the active toolchain. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Research by Opus 5: PR #3 — Fewer false cache positives: hash
|
| Half | Upstream status | Action |
|---|---|---|
| DependsOn output/input overlaps | Solved in v2.9.17–v2.10.6, opt-in | Drop from this PR — see #1 |
--only drops dependency edges |
Untouched | Keep; this becomes the whole PR |
Half 1: dependsOn overlaps — drop it
Upstream shipped structured inputs with startup / jit / dependencyOutputs modes (#13125), dependency-output hashing (#13129), and deferral cascade to dependents (#13045). Issue #8051 is closed.
Our bundle/fetch case — the motivating one, with a cache: false dependency — expresses directly. I verified against validate_dependency_outputs_inputs in v2.10.6 that the only constraint on a selected dependency is that it declares outputs; cache: false is not rejected:
{
"tasks": {
"fetch": { "cache": false, "outputs": ["target-fetched/**"] },
"bundle": {
"dependsOn": ["fetch", "^bundle"],
"outputs": ["target/**"],
"inputs": [
"$TURBO_DEFAULT$",
"!target-fetched/**",
{ "mode": "dependencyOutputs", "globs": ["target-fetched/**"], "from": ["fetch"] }
]
}
}
}Concretely, these commits should come out of this PR:
8612eb1ff5defer file hashing for tasks with dependency output inputs8d09f6af36rehash tasks whose dependencies were deferred9f343fbbaaonly defer when dependency task has a scriptcf9d0e2e08/86fcb58bd6/59d9f818f0<DEFERRED>and<DEPENDS_ON_OUTPUT>dry-run display64274254aadeferred → depends-on-output rename- the
turborepo-task-hashRwLock<HashMap>+update_file_hashchange TaskHashTracker::set_hash
The RwLock one matters beyond redundancy: upstream #13273 explicitly removed locking from the hash precompute path for performance. Keeping ours would reintroduce the contention they just eliminated, and it will conflict on every rebase.
dep_output_overlap.rs (569 lines, 15 unit tests) is the exception — it has no upstream counterpart and should survive, but re-targeted. Instead of implementing deferral itself, it can emit a synthesized dependencyOutputs input at engine-build time and let upstream's machinery take over. The glob intersection it already computes is precisely what vercel#13129's coverage validation demands. That turns ~800 lines of runtime machinery into a detection pass plus an injection point.
Half 2: --only — this is the real remaining content
Upstream v2.10.6's builder.rs still silently drops these edges, identical to 2.9.10 (details in the #2 comment). Keep:
Engine::dropped_dependencies/add_dropped_dependencyand both recording sites inbuilder.rsturborepo_hash::hash_string- the visitor fold-in of dropped-dependency SCM package hashes
c07e27839ethe warning, and2dbb4c5987the#[should_panic]tripwire test
Rebase note: the fold-in currently patches one call site, calculate_task_hash. Upstream now has two — calculate_task_hash and calculate_task_hash_with_deferred_inputs (visitor lines ~763–800 at v2.10.6). Both need the dropped-dependency hash folded in, or a task that uses jit/dependencyOutputs and has an edge dropped by --only will silently skip the compensation. That combination is exactly our bundle task, so it is not hypothetical. Worth a test.
The documented limitation is still there
From d57d36034e:
only works for packages that define the depended-on task. Packages without the task have no
^bundleedge to drop, so their files aren't tracked.
Upstream has not changed anything that helps here — a package with no bundle script never produces a ^bundle edge, so there is nothing to record as dropped. $TURBO_ROOT$ inputs remain the workaround. This should stay in the PR description and ideally in the warning text, since it is the case most likely to bite: a types-only package with no build step is precisely the kind of dependency people expect to be tracked.
Suggested action
- Retitle to scope it to
--onlyalone, e.g. "--onlymust not drop dependency packages from the cache key". - Rebase onto v2.10.6 with only the Half-2 commits, plus
dep_output_overlap.rsif we pursue auto-injection — otherwise defer that to its own PR. - Add coverage for the deferred-inputs +
--onlycombination noted above. - Fold Document cache behavior with --only #2 in here.
- Tag the result
v2.10.6-hashdepends.1. The release workflow needs its toolchain pin moved fromnightly-2026-01-16tonightly-2026-05-22to match v2.10.6'srust-toolchain.toml; therelease-turborepoprofile and the cross-compilation-target step are unchanged.
`--only` removes dependency edges pointing outside the filter set. Those tasks never run, never produce a task hash, and so contribute nothing to the downstream task's cache key. A change in an excluded package then produces a cache *hit* and turbo restores a stale artifact. Upstream treats the edge dropping as intended behavior for `--only`, and this change does not argue otherwise: the task graph is left exactly as upstream builds it. Instead the engine records each dropped edge, and the excluded package's source hashes are folded in where the missing task hash would have gone. Fresh start from upstream v2.10.6, squashing what remains relevant from #3 (previously released as v2.9.10-hashdepends.2). The dependsOn-output half of that PR is deliberately dropped: upstream solved it in v2.9.17-v2.10.6 via structured `inputs` modes (vercel#13043, vercel#13045, vercel#13125, vercel#13127, vercel#13129), closing vercel#8051. Express those cases with `mode: "jit"` or `mode: "dependencyOutputs"` instead. Notably vercel#13273 removed locking from task hash precomputation, which is the opposite of the `RwLock<HashMap>` the old branch carried. Changes from the previous implementation: - Fold the stand-in hashes into the task's dependency hash list rather than re-hashing the finished task hash. The tracker's copy is then compensated too, so dependents of an affected task also invalidate. This also covers all three hashing paths at once, including `calculate_task_hash_with_deferred_inputs`, which the old post-hoc approach would have missed for tasks that use `jit`/`dependencyOutputs` and have an edge dropped -- the exact combination in our bundle task. - Retain `dropped_dependencies` in `prune_to_reachable` so the compensation survives watch and subgraph runs. - Memoize per-package hashing; several tasks commonly drop the same package, and hashing walks the whole package tree. - Deduplicate dropped edges; a package can be both a topological and a direct dependency. - Pass `repo_index` so hashing uses the repo index fast path (vercel#13213). - Drop the `hash_string` helper, unnecessary under the new approach; turborepo-hash is now untouched by the fork. - Add positive tests that dropped edges are recorded on both the topological and direct-dependency paths, plus one asserting nothing is recorded without `--only`. The old branch tested none of this. - Reword the `#[should_panic]` test as a deliberate tripwire rather than a bug report, and pin its expected panic message. Refs: #1, #2, #3 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>