perf: Build resolution identity lists in parallel - #13643
Merged
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
tknickman
approved these changes
Aug 3, 2026
anthonyshew
added a commit
that referenced
this pull request
Aug 3, 2026
…ures (#13641) ### Description > **Base of a 3-PR stack**: #13641 (this) → #13642 (parallel fingerprint) → #13643 (parallel identity build). This PR targets `main`; review in order. Together they cut package-graph construction ~264 ms → ~178 ms (~33%) on the profiled repo. Profiling package-graph construction on a large real-world pnpm monorepo (~1,200 workspaces) showed `resolve_dependencies` materializing every workspace's external closure independently: one `ExternalPackageIdentity` per closure member per workspace (two string clones plus a `human_name` lockfile lookup each — ~390k constructions total), then one fingerprint hash per workspace over the full `(key, version)` sequence. Only ~two-thirds of those closures are actually distinct — the rest is duplicated work. Changes: - `PackageResolution` now stores its identity list as `Arc<[ExternalPackageIdentity]>`. The list is sorted/deduped at construction (`new`, the only constructor) and immutable afterward. - `resolve_dependencies` shares one materialized list across workspaces whose closures are identical, detected by the closure's `Arc` pointer sequence. Closure members are interned `Arc`s whenever the shared closure DP produced them (pnpm today; npm/yarn1 with #13635), so identical pointer sequences ⇔ identical closures. Legacy-walk formats produce distinct pointers per workspace and simply keep building independently — behavior unchanged, no penalty. - `ExternalResolutionGeneration::build` fingerprints each distinct list once, memoized by slice address — at most one map probe per *package* (not per member), so unshared lists pay nothing measurable beyond the hash they already required. Its per-package re-sort is dropped since `new` already guarantees the invariant. Public surface is unchanged: `identities()` still returns `&[ExternalPackageIdentity]`, `new()` has the same signature, and cargo/uv resolution producers are untouched. Cloning a resolution/generation is now a refcount bump instead of a deep identity-list copy. **Measured** on the profiled repo (release, 8 warm samples per side): package graph construction ~264 ms → ~249 ms median (**~6%**) from this PR alone, with a ~35% cut in identity-list allocations. The two PRs stacked on top take it to ~178 ms. For transparency: two cheaper alternatives were tried first — per-member identity memoization and content-keyed fingerprint dedup — and both measured neutral-to-worse (per-member map probes cost what they saved), which is why this lands as the structural pointer-keyed sharing instead. ### Testing Instructions - `cargo test -p turborepo-repository` — all 419 tests pass (identity ordering/dedup through `new`, generation building, cross-domain validation). - `cargo check -p turborepo-lib` compiles unchanged against the new field representation. - `cargo clippy -p turborepo-repository --all-targets` is clean. - Fingerprint semantics are unchanged: the hash input is the same sorted, deduped `(key, version)` sequence as before; sharing only avoids recomputing it for byte-identical sequences. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU Co-authored-by: Claude <noreply@anthropic.com>
Stacked on the shared-identity-list change: packages that share an identity list already share its Arc, so fingerprint each *distinct* list exactly once (keyed by slice address) and hash those distinct lists in parallel with rayon. Each is an independent Cap'n Proto canonicalization plus xxHash64, and on a monorepo with ~1,200 workspaces the fingerprint loop is the dominant cost of building a generation. Only the borrowed identity slices cross into the parallel iterator; the slice-pointer keys stay on the sequential side. Output is unchanged: the canonical bytes are deterministic per input and packages are re-sorted deterministically afterward, so fingerprints and ordering are byte identical to the sequential version — the cache-compatibility-critical hash algorithm and its byte layout are untouched, only the iteration changes. Measured on a 4-core machine over a ~1,200-workspace repo: package graph construction drops from ~249ms (shared lists alone) to ~213ms median, and from ~264ms on the base branch — the fingerprint phase falls from ~41ms to single-digit ms. The speedup scales with core count; small repos are unaffected (rayon runs tiny inputs effectively serially). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU
Stacked on the shared-identity-list and parallel-fingerprint changes. With identical closures already sharing one Arc-backed identity list, build the lists for the *distinct* closures in parallel with rayon. This is the bulk of resolution assembly on large monorepos: a string clone for each closure member's key and version plus a lockfile human_name lookup, repeated across every distinct closure (~800 of them, ~300 members each, on the profiled repo). Distinct closures are bucketed sequentially by their Arc pointer sequence (the raw pointers never leave that side); only the borrowed closure slices cross into the parallel build. The sort/dedup invariant now lives in one place, PackageResolution::shared_identity_list, used by both new() and the parallel path so from_shared stays consistent. Output is unchanged — each list is sorted/deduped identically and attributed to the same packages; only the wall-clock changes. Measured on a 4-core machine over a ~1,200-workspace repo: identity construction drops from ~45ms to ~12ms, taking package graph build from ~213ms (rest of the stack) to ~178ms median — ~33% below the ~264ms pre-stack base. Scales with core count; small repos are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU
anthonyshew
force-pushed
the
claude/perf-parallel-identity-build
branch
from
August 3, 2026 21:35
38bb05e to
8ad29d5
Compare
anthonyshew
added a commit
that referenced
this pull request
Aug 3, 2026
…13645) ### Description > **4th entry in the resolution stack** → #13641 → #13642 → #13643 → this. Base is `claude/perf-parallel-identity-build`; review the three below first. The diff shown here is only this PR's commit. A shared dependency appears in the transitive closure of many workspaces. Even after #13641/#13643 share identity *lists* across identical closures, building each distinct closure's list still cloned every member's `key` and `version` strings and re-ran the lockfile `human_name` lookup — so a package appearing in N distinct closures allocated its identity N times. **dhat pinpointed this as the single largest allocation source** in package-graph construction: three sites at ~386,800 blocks each (`key.clone()`, `version.clone()`, `human_name()`) — ~1.16M allocations, over half the total. This stores `ExternalPackageIdentity`'s `key`, `version`, and `human_name` as `Arc<str>` and interns one identity per distinct package. The shared closure DP already hands us the same `Arc<Package>` everywhere a package occurs, so identity is bucketed by `Arc` pointer: each distinct package's strings are allocated once (in parallel), and every closure that contains it clones a refcount-bumped identity instead of re-cloning strings. Raw pointers stay on the sequential bucketing side; the parallel identity build and list assembly only touch `Sync` data. Public surface unchanged: `new()`/`with_human_name()` still accept anything `Into<Arc<str>>` (both `String` and `&str` qualify, so existing callers — cargo, uv, tests — compile untouched), and the accessors still return `&str`. **Verified with dhat** on a ~1,200-workspace pnpm monorepo (`dhat::HeapStats` around `PackageGraphBuilder::build`): | | heap allocations | bytes | |---|---|---| | `main` | ~2,128,000 | ~300 MB | | stack (#13641→#13643) | ~1,876,000 | ~268 MB | | **+ this PR** | **~989,000** | **~238 MB** | This PR alone removes **~887,000 allocations** (47% of what remained) and ~30 MB. Wall-time is within noise on a warm 4-core box, but fewer allocations is a real reduction in allocator work and memory pressure that compounds on constrained CI runners, slower allocators, and larger repos. ### Testing Instructions - `cargo test -p turborepo-repository` — all 419 tests pass, including resolution/fingerprint tests. Identities are *shared*, not changed: same `(key, version)`, same ordering, same fingerprints. - `cargo clippy -p turborepo-repository --all-targets` is clean. - Allocation reduction reproduced with a dhat-instrumented harness around `PackageGraphBuilder::build`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU --- _Generated by [Claude Code](https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
anthonyshew
pushed a commit
that referenced
this pull request
Aug 5, 2026
## Release v2.10.9-canary.1 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31044275876) ### Changes - refactor: Delete legacy external-resolution PackageInfo state (#13526) (`fbe88d3`) - chore: Release Turborepo 2.10.8-canary.4 (#13557) (`887d9b1`) - refactor: Remove external declaration compatibility paths (#13527) (`3c8448d`) - refactor: Produce immutable native task and command knowledge (#13528) (`18d0bc3`) - refactor: Migrate native task registration and suggestions (#13529) (`7b5ee63`) - refactor: Migrate turbo-json native task synthesis (#13530) (`2d004f0`) - refactor: Migrate persistent and recursive task validation (#13531) (`dd7519b`) - refactor: Migrate native task definition precedence (#13532) (`8792ce9`) - refactor: Migrate engine native command planning (#13533) (`67e4e9e`) - refactor: Migrate executor native command resolution (#13534) (`1b8accc`) - refactor: Migrate native task query, devtools, and LSP views (#13535) (`2753b76`) - refactor: Migrate command summaries and delete legacy task paths (#13536) (`adbad50`) - refactor: Produce immutable task-contract knowledge (#13537) (`97bff31`) - refactor: Migrate engine task-contract composition (#13538) (`fab50c5`) - refactor: Migrate hashing engines to task contracts (#13539) (`76206f8`) - refactor: Exclude JavaScript from toolchain task-I/O dispatch (#13540) (`115e850`) - refactor: Migrate change classification to immutable knowledge (#13543) (`5a55e8d`) - refactor: Separate prune rendering with golden coverage (#13546) (`62a047a`) - refactor: Delete JS format interpretation from prune orchestration (#13554) (`f1b37f6`) - refactor: Audit remaining JS knowledge consumer reads (#13556) (`7f12c3f`) - refactor: Migrate MFE dependency detection off PackageInfo (#13558) (`4cb9d94`) - refactor: Remove prune PackageInfo dependencies (#13562) (`1bb9233`) - refactor: Remove residual runtime PackageInfo gates (#13564) (`c8eaedc`) - refactor: Migrate boundary diagnostics off PackageInfo (#13571) (`f977db3`) - docs: Audit Cargo package knowledge (#13572) (`53dd13e`) - refactor: Complete Cargo relationship and resolution knowledge (#13576) (`b29e30c`) - refactor: Port Cargo task and contract knowledge (#13581) (`74446c5`) - refactor: Port Cargo watch and prune knowledge (#13582) (`acd3be1`) - refactor: Remove runtime toolchain dispatch (#13584) (`2c10201`) - ci: Restore Cargo target for lockfile tests (#13588) (`e848ad1`) - refactor: Replace toolchains with repository contributors (#13585) (`5d06e95`) - refactor: Remove Cargo contributor plumbing (#13586) (`6ea1daf`) - refactor: Remove ToolchainId runtime dispatch (#13587) (`24fbd3e`) - refactor: Route task behavior through contract domains (#13589) (`da88240`) - refactor: Route package consumers by manifest (#13591) (`bb12c72`) - refactor: Route MFE eligibility by manifest (#13592) (`08c7a74`) - refactor: Project manifest-derived repository facts (#13593) (`64e2a7e`) - refactor: Route residual task behavior by capability (#13595) (`fe9b72d`) - refactor: Route resolution through explicit domains (#13596) (`be21801`) - refactor: Route N-API package listing by manifest (#13597) (`8f44636`) - docs: Vercel Remote Cache authentication with OIDC policies (#13140) (`c84ed36`) - refactor: Own resolution fingerprints in repository (#13598) (`16708a9`) - refactor: Fail closed on invalid relationships (#13599) (`1a237de`) - perf: Reuse Cargo metadata discovery snapshot (#13600) (`2fa79c5`) - refactor: Resolve MFE package ownership from graph (#13601) (`0253836`) - refactor: Remove retained package payloads (#13603) (`58d9660`) - feat: Add native Cargo format task (#13606) (`ab15587`) - refactor: Compose repository graphs for optional toolchains (#13608) (`915e82b`) - ci: Invalidate Cap'n Proto caches (#13616) (`5cf35ad`) - feat: Discover uv workspaces (#13609) (`8715646`) - feat: Run native uv tasks (#13610) (`4195e41`) - feat: Hash uv lockfile closures (#13611) (`e14de24`) - fix: Make Windows Cap'n Proto cache relocatable (#13621) (`00538d0`) - feat: Watch uv workspace changes (#13612) (`b2e25d4`) - feat: Prune uv workspaces (#13613) (`f8f288e`) - fix: Fall back to polling on macOS (#13622) (`b3dc99b`) - test: Add uv workspace integration coverage (#13602) (`dd87718`) - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew
pushed a commit
that referenced
this pull request
Aug 7, 2026
## Release v2.10.9 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31216733075) ### Changes - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) - chore: Release Turborepo 2.10.9-canary.1 (#13687) (`c09a92f`) - docs: Document dependency-driven Python tasks (#13676) (`a98e5cd`) - fix: Prune Bun wildcard workspace dev dependencies (#13694) (`efe4e1b`) - fix: Prevent Windows process cleanup PID reuse (#13695) (`3b0e57f`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
After #13641 made identical closures share one
Arc-backed identity list and #13642 parallelized fingerprinting, the largest remaining cost inresolve_dependenciesis building the identity lists themselves: aStringclone for each closure member's key and version, plus a lockfilehuman_namelookup — repeated across every distinct closure (~800 of them, ~300 members each, on the profiled repo). Measured at ~45 ms, it was the single biggest remaining lever in package-graph construction.This builds the lists for the distinct closures in parallel with rayon. Distinct closures are bucketed sequentially by their
Arcpointer sequence (the raw pointers never leave that side); only the borrowed closure slices cross into the parallel build. The sort/dedup invariant now lives in one place —PackageResolution::shared_identity_list, used by bothnew()and the parallel path — sofrom_sharedstays consistent.Output is unchanged: each list is sorted/deduped identically and attributed to the same packages; only wall-clock changes.
Measured (4-core machine, ~1,200-workspace repo, release, warm samples): identity construction ~45 ms → ~12 ms, taking package graph build from ~213 ms (rest of the stack) to ~178 ms median — ~33% below the ~264 ms pre-stack base. Scales with core count; small repos are unaffected. This runs inside the discovery
block_in_placeregion, consistent with the already-parallel manifest parsing and the #13642 fingerprint loop.Testing Instructions
cargo test -p turborepo-repository— all 419 tests pass; package counts and resolutions are identical (verified against two real monorepos: 1,226 and 143 workspaces, package counts unchanged).cargo clippy -p turborepo-repository --all-targetsis clean.Full stack summary
On the ~1,200-workspace repo, the three PRs together cut package-graph construction from ~264 ms to ~178 ms (~33%): shared lists (~264→249), parallel fingerprint (~249→213), parallel identity build (~213→178).
🤖 Generated with Claude Code
https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU
Generated by Claude Code