perf: Overlap external dependency hashing with package file hashing - #13234
Merged
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
tknickman
approved these changes
Jul 4, 2026
anthonyshew
pushed a commit
that referenced
this pull request
Jul 4, 2026
## Release v2.10.4-canary.1 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/28719728846) ### Changes - release(turborepo): 2.10.3-canary.8 (#13200) (`5987ed1`) - fix(watch): recover from slow initial hash instead of timing out (#13159) (`05e26cb`) - chore: Update vitest example to vitest 4 and use native test merging (#13202) (`e2b61dc`) - feat: Toggle between the TUI and streamed logs (#13203) (`1325455`) - fix: Resolve Bun scoped package deps that share a name with a dependency (#13207) (`e4773ab`) - feat: Select tasks by clicking rows in the TUI task list (#13206) (`c568f67`) - perf: Avoid deep-cloning dependency maps in lockfile closure cache (#13209) (`abd84cf`) - perf: Cache root internal dependencies in package graph (#13211) (`7ec8c6d`) - feat: Automatically copy TUI selection to clipboard on mouse release (#13208) (`61d6013`) - perf: Cache env wildcard matches across tasks during hashing (#13210) (`148b1dd`) - perf: Replace per-package graph traversals in scope filtering (#13212) (`28d1871`) - perf: Derive dirty hash from repo index (#13213) (`b521d32`) - fix: Include untracked symlinks in repo-index dirty hash (#13218) (`c3c91ab`) - refactor: Use upstream libghostty-vt crates instead of vendored bindings (#13205) (`c2115dc`) - fix: Remove devtools feature flag (#13219) (`65efe27`) - fix: Harden TUI terminal restore during shutdown (#13220) (`dccab93`) - fix: Correct gitignore precedence in untracked walk and memoize matcher chains (#13221) (`bc32fcc`) - release(turborepo): 2.10.3 (#13222) (`02f4ed0`) - ci: Retry Windows nextest aborts from transient 0xc0000142 spawn failures (#13240) (`222ffff`) - perf: Use mimalloc as the global allocator (#13237) (`c42a8a9`) - perf: Parse yarn v1 lockfiles in a single pass (#13241) (`6207099`) - ci: Fix cache outputs for Eve app (#13243) (`d8b556d`) - perf: Stop materializing spans for the disabled daemon log layer (#13244) (`623aadc`) - fix: Keep ancestor-scoped bun dependencies resolvable after prune renames (#13236) (`8658b66`) - perf: Overlap external dependency hashing with package file hashing (#13234) (`445b0c7`) - perf: Build tracked repo index concurrently with package graph (#13232) (`f4699c6`) - perf: Release tokio runtime at exit so background DNS lookups never stall the user (#13231) (`ae635bd`) - perf: Overhaul pnpm lockfile parsing and dependency closure computation (#13228) (`669676d`) - feat: Add futureFlags.experimentalCargoWorkspaces flag (no-op) (#13227) (`f546930`) - feat: Introduce toolchain provider abstraction (#13235) (`f03addb`) - perf: Parse Berry lockfiles in a single pass (#13242) (`6d36125`) - fix: Apply input exclusion globs to the filesystem walk (#13224) (`d3f9dd3`) - fix: Stop root-directory packages from claiming every file in change mapping (#13225) (`f80203a`) - chore: Rename turborepo-repository napi package to @turbo/repository (#13226) (`c1972b3`) - perf: Hash git blobs with hardware-accelerated SHA-1 (#13245) (`17ffa0c`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew
added a commit
that referenced
this pull request
Jul 6, 2026
…13249) ## Why Run-summary construction (`--dry`, `--summarize`, observability) calls `get_external_deps_hash` per *task*: it re-sorts the package's transitive dependency closure and rebuilds a capnp message every time, even though task hashing already computed exactly this value once per package. On a 1191-package monorepo dry run this pipeline (capnp wire helpers + closure sorts) was 15.9% of total CPU — the largest single subsystem left after the lockfile and hashing work merged. ## What Expose the per-package cache that task hashing already maintains (`TaskHasher::external_deps_hash_cache`, populated since #13234) and thread it through `RunTracker::finish` into `TaskSummaryFactory`, which now looks up instead of recomputing. The fallback path (cache absent or package missing) calls the same function as before and produces byte-identical output — both implementations sort by `(key, version)` and hash the same `LockFilePackagesRef` message. Normal runs (no dry/summarize/observability) are unaffected: they already skip summary construction entirely. ## Benchmarks Interleaved A/B, 12 pairs, `turbo run build --dry=json` on a 1191-package/43k-file pnpm monorepo (4-core Linux), fresh git index: | binary | wall (mean) | wall (stdev) | CPU (mean) | |---|---|---|---| | main | 440ms | 21ms | 1091ms | | this PR | **401ms (−8.9%)** | 16ms | **941ms (−13.7%)** | capnp+sort symbols drop from 15.9% to 6.6% of profile samples (the remainder is the legitimate once-per-package/per-task hashing). Full dry-run JSON is byte-identical (only the random run `.id` differs).
anthonyshew
pushed a commit
that referenced
this pull request
Jul 6, 2026
## Release v2.10.4 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/28819266360) ### Changes - release(turborepo): 2.10.3 (#13222) (`02f4ed0`) - ci: Retry Windows nextest aborts from transient 0xc0000142 spawn failures (#13240) (`222ffff`) - perf: Use mimalloc as the global allocator (#13237) (`c42a8a9`) - perf: Parse yarn v1 lockfiles in a single pass (#13241) (`6207099`) - ci: Fix cache outputs for Eve app (#13243) (`d8b556d`) - perf: Stop materializing spans for the disabled daemon log layer (#13244) (`623aadc`) - fix: Keep ancestor-scoped bun dependencies resolvable after prune renames (#13236) (`8658b66`) - perf: Overlap external dependency hashing with package file hashing (#13234) (`445b0c7`) - perf: Build tracked repo index concurrently with package graph (#13232) (`f4699c6`) - perf: Release tokio runtime at exit so background DNS lookups never stall the user (#13231) (`ae635bd`) - perf: Overhaul pnpm lockfile parsing and dependency closure computation (#13228) (`669676d`) - feat: Add futureFlags.experimentalCargoWorkspaces flag (no-op) (#13227) (`f546930`) - feat: Introduce toolchain provider abstraction (#13235) (`f03addb`) - perf: Parse Berry lockfiles in a single pass (#13242) (`6d36125`) - fix: Apply input exclusion globs to the filesystem walk (#13224) (`d3f9dd3`) - fix: Stop root-directory packages from claiming every file in change mapping (#13225) (`f80203a`) - chore: Rename turborepo-repository napi package to @turbo/repository (#13226) (`c1972b3`) - perf: Hash git blobs with hardware-accelerated SHA-1 (#13245) (`17ffa0c`) - release(turborepo): 2.10.4-canary.1 (#13247) (`a31389e`) - feat: Discover Cargo crates as packages (#13248) (`d813d12`) - perf: Reuse per-package external dependency hashes in run summaries (#13249) (`ef7ef06`) - perf: Stat workspace turbo.json files concurrently during discovery (#13251) (`57be8e5`) - chore: Add tracing spans to workspace discovery and globwalk phases (#13252) (`bd76946`) - fix: Avoid non-reentrant libc calls in concurrent shutdown process scans (#13256) (`bf2d827`) - perf: Memoize resolved task definitions during engine construction (#13257) (`9085289`) - perf: Compute transitive closures and external dependency hashes concurrently with run setup (#13250) (`b305fe2`) - perf: Probe microfrontends configs in parallel (#13262) (`ea7c1f8`) - feat: Execute Cargo crate tasks via cargo (#13261) (`ee81349`) - perf: Pre-size engine task collections (#13265) (`17bc22e`) - feat: Expose resolved experimentalCI task configuration in `turbo query` (#13264) (`0f564e5`) - feat: Derive input and output globs for Cargo tasks (#13263) (`05b0e74`) - perf: Parse large pnpm lockfile sections in parallel (#13266) (`ac4155e`) - feat: Hash Cargo external dependencies per-crate (#13267) (`71300aa`) - ci: Install pnpm 10 in musl containers for Library Release (#13269) (`5ba9f66`) - ci: Force pnpm overwrite in Library Release musl containers (#13270) (`560ee2f`) - perf: Defer the untracked-scan barrier to first file-hash use (#13268) (`96b0f5a`) - ci: Fetch API-created commit before updating local ref (#13271) (`4cbaf48`) - release(library): 0.0.1-canary.22 (#13272) (`d2e0691`) - perf: Remove lock and dispatch overhead from task hash precomputation (#13273) (`dd6de07`) - fix: Fall back to lockfile detection in @turbo/repository when package manager is undeclared (#13275) (`8cf0e75`) - test: Add end-to-end coverage for Cargo workspaces (#13274) (`ce18f0a`) - release(library): 0.0.1-canary.23 (#13276) (`8e3a59f`) - release(turborepo): 2.10.4-canary.2 (#13278) (`7e02f94`) - fix: Collapse nested package-manager fallback conditional (#13279) (`af01fdf`) - feat: Make turbo watch Cargo-aware (#13280) (`39d623e`) - perf: Skip dependency-closure assembly for toolchains that derive nothing (#13277) (`ff0d508`) - feat: Make turbo prune Cargo-aware (#13281) (`ddc584d`) - fix: Raise the open-file soft limit at startup (#13282) (`947b478`) - fix: Stop flagging relative imports that resolve into node_modules in boundaries (#13284) (`11a68c7`) - perf: Evaluate simple include globs without wax compilation (#13285) (`189897a`) - chore: Dogfood native Cargo support in this repository (#13283) (`42f067b`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Between the package graph finishing and the first task dispatching, two chunks of hashing work run serially that don't need to:
TaskHasherconstruction computes external dependency hashes for every package (~47ms on a 1191-package monorepo) after the parallelhash_scopeblock has already finished, even though it depends only on the package graph.calculate_file_hashesdeduplicates file hashing by unique(package, inputs)key, but then recomputes the summary hash per task when distributing results — 1866 hash computations where ~1200 suffice (~20ms of the previously uninstrumentedhash_scopetail).Both sit directly on the time-to-first-task critical path.
What
Three commits, separable if a reviewer wants:
chore:phase spans (collect_task_hash_keys,hash_unique_inputs,distribute_task_file_hashes) insidecalculate_file_hashes— this instrumentation is how the per-task re-hashing was found, and it keeps the phase visible in profiles.perf:compute each unique key's summary hash once, inside the existing parallel phase; distribution becomes pure map inserts.perf:move external dependency hashing into the existing 4-wayrayon::scopeinRun::run_with_opts, passing the precomputed cache intoVisitor::new(which falls back to the old serial path if not provided). The per-task fallback for cache misses is unchanged.How to verify
On a 1191-package internal-scale monorepo (4-core machine, interleaved pairs vs main, telemetry disabled):
visitor_newblock disappears;hash_scopeabsorbs the work (78ms → 97.5ms wall on 4 cores, so net ~−27ms there — the win grows with core count since the added work is fully parallel).distribute_task_file_hashesdrops 20.1ms → ~2ms.--dry=jsonoutput byte-identical to main (modulo run id) on two large monorepos — task hashes are unchanged by construction.