perf: Stat workspace turbo.json files concurrently during discovery - #13251
Merged
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 5, 2026
anthonyshew
added a commit
that referenced
this pull request
Jul 6, 2026
…3255) ## Why Manifest parsing sits on the critical path of every run: one parse per workspace package. The general-purpose biome CST parser earns its ~27µs/manifest through error recovery machinery that does nothing useful on valid manifests — which is nearly every manifest in every repo. A purpose-built scanner produces the same values, the same spans, and equivalent errors in ~1-2µs. ## What `crate::manifest_parser`: a single-pass, fully validating JSON scanner that builds `PackageJson` directly. It is the only parse path — no tiering, no fallback — and the biome deserializer plus its manifest types are deleted outright. biome and turborepo-unescape drop out of the crate's dependency tree entirely. - **Spans preserved**: `Spanned` fields carry the value token's full source range (quotes/braces included), so miette snippets in downstream diagnostics — packageManager mismatch, devEngines validation (including key-search sub-spans), recursive-turbo — keep their highlighted snippets. Parse errors point at the offending token. The preexisting span-assertion tests pass unchanged. - **Acceptance contract**: strict JSON, tolerated leading BOM, duplicate keys last-win, explicit `null` is a type error in typed fields but a valid `devEngines` declaration. Known change: serde_json's 128-level nesting limit applies when materializing unstructured field values. - **Self-contained tests**: the behavior contract is pinned by dependency-agnostic tests asserting parsed values, exact byte ranges (including BOM-offset alignment), attached text/path metadata, and accept/reject decisions across ~45 acceptance and malformed-input cases — escapes, duplicates, bad numbers, bad escapes, mismatched delimiters, control characters, type errors. - Adds `workspace_discovery`/`manifest_parse`/`add_packages` spans to break down `parse_package_jsons` in `--profile` output (this breakdown located the cost). ## How to verify - `cargo test -p turborepo-repository` — 289 tests, including all preexisting package-manager span assertions, unchanged. - Measured on a 1191-package monorepo (interleaved A/B, 8 runs, medians): manifest parse wall 29ms → 8.8ms, parser CPU 46-60ms → 3ms (residue is turbo.json parsing, a separate code path), TTFT 258.7ms → 250.4ms. `--dry=json` byte-identical. Stacked on #13250. Compounds with #13251, which removes the sequential turbo.json stats dominating the same discovery phase.
anthonyshew
added a commit
that referenced
this pull request
Jul 6, 2026
#13258) ## Why Untracked-file discovery is the largest single block on the time-to-first-task critical path (~98ms on a 1191-package/43k-file monorepo), and it waited for the package graph solely to compute per-package scan prefixes. That dependency turns out to be vestigial: the scan scope was always exactly the repo-root subtree — every package lives under the repo root (enforced at discovery), the root package's prefix is always in the prefix set, and `UntrackedScope` deduplicates nested prefixes. The per-package prefixes never restricted anything beyond the repo root's own prefix. ## What - The untracked scan runs on the same background task that performs SCM detection and builds the tracked index, using the repo-root prefix directly (`repo_prefix_for_repo_index`: repo root anchored to the git root — empty for the common case, the subtree prefix when the repo is nested in a larger git repository). It starts as soon as the tracked index exists (~32ms) instead of after graph construction (~87ms). - The SCM handle is sent back over a oneshot channel the moment detection finishes, so the main flow (graph build, turbo.json loading, engine construction) never waits behind the scan. The index is joined where it was before, at the first consumer. - `all_package_prefixes` is deleted; the prefix-anchoring tests are repurposed to cover the repo-root prefix in both the nested-git-root and matching-root layouts. ## How to verify - Scan-scope equivalence argument above; `turborepo-scm`'s untracked/regression suites (191 tests) and `turborepo-lib` (432) pass unchanged; `--dry=json` byte-identical on the monorepo above. - Timeline on that monorepo: scan moves from ~87→185ms to ~32→125ms; with #13257's engine memo the scan+engine phase of the critical path shrinks ~188ms → ~165ms, and engine construction itself drops ~73ms → ~40ms from reduced core contention. TTFT medians move less than the phase numbers on a saturated 4-core machine (~246 → ~238ms) because a previously-hidden serial segment between graph completion and engine start now gates; that segment is the next target. Stacked on #13250. Compounds with #13251 and #13257.
anthonyshew
pushed a commit
that referenced
this pull request
Jul 6, 2026
## Release v2.10.4-canary.2 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/28806467849) ### Changes - release(turborepo): 2.10.4-canary.1 (#13247) (`a31389e`) - feat: Discover Cargo crates as packages (#13248) (`d813d12`) - perf: Reuse per-package external dependency hashes in run summaries (#13249) (`ef7ef06`) - perf: Stat workspace turbo.json files concurrently during discovery (#13251) (`57be8e5`) - chore: Add tracing spans to workspace discovery and globwalk phases (#13252) (`bd76946`) - fix: Avoid non-reentrant libc calls in concurrent shutdown process scans (#13256) (`bf2d827`) - perf: Memoize resolved task definitions during engine construction (#13257) (`9085289`) - perf: Compute transitive closures and external dependency hashes concurrently with run setup (#13250) (`b305fe2`) - perf: Probe microfrontends configs in parallel (#13262) (`ea7c1f8`) - feat: Execute Cargo crate tasks via cargo (#13261) (`ee81349`) - perf: Pre-size engine task collections (#13265) (`17bc22e`) - feat: Expose resolved experimentalCI task configuration in `turbo query` (#13264) (`0f564e5`) - feat: Derive input and output globs for Cargo tasks (#13263) (`05b0e74`) - perf: Parse large pnpm lockfile sections in parallel (#13266) (`ac4155e`) - feat: Hash Cargo external dependencies per-crate (#13267) (`71300aa`) - ci: Install pnpm 10 in musl containers for Library Release (#13269) (`5ba9f66`) - ci: Force pnpm overwrite in Library Release musl containers (#13270) (`560ee2f`) - perf: Defer the untracked-scan barrier to first file-hash use (#13268) (`96b0f5a`) - ci: Fetch API-created commit before updating local ref (#13271) (`4cbaf48`) - release(library): 0.0.1-canary.22 (#13272) (`d2e0691`) - perf: Remove lock and dispatch overhead from task hash precomputation (#13273) (`dd6de07`) - fix: Fall back to lockfile detection in @turbo/repository when package manager is undeclared (#13275) (`8cf0e75`) - test: Add end-to-end coverage for Cargo workspaces (#13274) (`ce18f0a`) - release(library): 0.0.1-canary.23 (#13276) (`8e3a59f`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew
pushed a commit
that referenced
this pull request
Jul 6, 2026
## Release v2.10.4 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/28819266360) ### Changes - release(turborepo): 2.10.3 (#13222) (`02f4ed0`) - ci: Retry Windows nextest aborts from transient 0xc0000142 spawn failures (#13240) (`222ffff`) - perf: Use mimalloc as the global allocator (#13237) (`c42a8a9`) - perf: Parse yarn v1 lockfiles in a single pass (#13241) (`6207099`) - ci: Fix cache outputs for Eve app (#13243) (`d8b556d`) - perf: Stop materializing spans for the disabled daemon log layer (#13244) (`623aadc`) - fix: Keep ancestor-scoped bun dependencies resolvable after prune renames (#13236) (`8658b66`) - perf: Overlap external dependency hashing with package file hashing (#13234) (`445b0c7`) - perf: Build tracked repo index concurrently with package graph (#13232) (`f4699c6`) - perf: Release tokio runtime at exit so background DNS lookups never stall the user (#13231) (`ae635bd`) - perf: Overhaul pnpm lockfile parsing and dependency closure computation (#13228) (`669676d`) - feat: Add futureFlags.experimentalCargoWorkspaces flag (no-op) (#13227) (`f546930`) - feat: Introduce toolchain provider abstraction (#13235) (`f03addb`) - perf: Parse Berry lockfiles in a single pass (#13242) (`6d36125`) - fix: Apply input exclusion globs to the filesystem walk (#13224) (`d3f9dd3`) - fix: Stop root-directory packages from claiming every file in change mapping (#13225) (`f80203a`) - chore: Rename turborepo-repository napi package to @turbo/repository (#13226) (`c1972b3`) - perf: Hash git blobs with hardware-accelerated SHA-1 (#13245) (`17ffa0c`) - release(turborepo): 2.10.4-canary.1 (#13247) (`a31389e`) - feat: Discover Cargo crates as packages (#13248) (`d813d12`) - perf: Reuse per-package external dependency hashes in run summaries (#13249) (`ef7ef06`) - perf: Stat workspace turbo.json files concurrently during discovery (#13251) (`57be8e5`) - chore: Add tracing spans to workspace discovery and globwalk phases (#13252) (`bd76946`) - fix: Avoid non-reentrant libc calls in concurrent shutdown process scans (#13256) (`bf2d827`) - perf: Memoize resolved task definitions during engine construction (#13257) (`9085289`) - perf: Compute transitive closures and external dependency hashes concurrently with run setup (#13250) (`b305fe2`) - perf: Probe microfrontends configs in parallel (#13262) (`ea7c1f8`) - feat: Execute Cargo crate tasks via cargo (#13261) (`ee81349`) - perf: Pre-size engine task collections (#13265) (`17bc22e`) - feat: Expose resolved experimentalCI task configuration in `turbo query` (#13264) (`0f564e5`) - feat: Derive input and output globs for Cargo tasks (#13263) (`05b0e74`) - perf: Parse large pnpm lockfile sections in parallel (#13266) (`ac4155e`) - feat: Hash Cargo external dependencies per-crate (#13267) (`71300aa`) - ci: Install pnpm 10 in musl containers for Library Release (#13269) (`5ba9f66`) - ci: Force pnpm overwrite in Library Release musl containers (#13270) (`560ee2f`) - perf: Defer the untracked-scan barrier to first file-hash use (#13268) (`96b0f5a`) - ci: Fetch API-created commit before updating local ref (#13271) (`4cbaf48`) - release(library): 0.0.1-canary.22 (#13272) (`d2e0691`) - perf: Remove lock and dispatch overhead from task hash precomputation (#13273) (`dd6de07`) - fix: Fall back to lockfile detection in @turbo/repository when package manager is undeclared (#13275) (`8cf0e75`) - test: Add end-to-end coverage for Cargo workspaces (#13274) (`ce18f0a`) - release(library): 0.0.1-canary.23 (#13276) (`8e3a59f`) - release(turborepo): 2.10.4-canary.2 (#13278) (`7e02f94`) - fix: Collapse nested package-manager fallback conditional (#13279) (`af01fdf`) - feat: Make turbo watch Cargo-aware (#13280) (`39d623e`) - perf: Skip dependency-closure assembly for toolchains that derive nothing (#13277) (`ff0d508`) - feat: Make turbo prune Cargo-aware (#13281) (`ddc584d`) - fix: Raise the open-file soft limit at startup (#13282) (`947b478`) - fix: Stop flagging relative imports that resolve into node_modules in boundaries (#13284) (`11a68c7`) - perf: Evaluate simple include globs without wax compilation (#13285) (`189897a`) - chore: Dogfood native Cargo support in this repository (#13283) (`42f067b`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Package discovery checks for a
turbo.jsonnext to every workspace'spackage.jsonwith a sequential async stream — 1191 one-at-a-timetry_existscalls on a 1191-package monorepo, ~20ms of the time-to-first-task critical path.What
Drive the stats through
futures'buffered(64)(order-preserving, so discovery output is unchanged) instead oftokio_stream's sequentialthen. Also adds two tracing spans (workspace_glob_walk,turbo_json_stat_stream) that split package discovery's cost in--profileoutput — finding this required exactly that split.tokio-streamwas only used here, so the dependency is dropped in favor of thefuturesworkspace dependency.Benchmarks
1191-package/43k-file pnpm monorepo (4-core Linux), measured on top of #13250:
turbo_json_stat_streamspan: 20.5ms → ~7ms typical (one cold-start outlier at 39ms in 8 runs, from tokio's blocking-pool spin-up; the serial version pays a similar cold-start tax spread across the stream)queue_task, 8 runs): median 339ms → 322ms