perf: Start untracked-file discovery before the package graph is built - #13258
Merged
anthonyshew merged 1 commit intoJul 6, 2026
Merged
Conversation
Untracked-file discovery waited for the package graph so it could compute per-package scan prefixes. But the scan scope was always exactly the repo-root subtree: every package lives under the repo root (enforced at discovery), the root package's prefix is always in the set, and UntrackedScope deduplicates nested prefixes — so the per-package prefixes never restricted anything beyond the repo root's own prefix. The scan now runs on the same background task that performs SCM detection and builds the tracked index, using the repo-root prefix directly, and starts as soon as the tracked index exists instead of after graph construction. The SCM handle is sent back over a oneshot channel so the main flow proceeds without waiting behind the scan; the index is joined where it was before, at the first consumer. On a 1191-package/43k-file monorepo the scan now runs at ~32-125ms instead of ~87-185ms, moving engine construction and task hashing off its tail: the scan+engine phase of the critical path shrinks from ~188ms to ~165ms, with reduced core contention also speeding the engine build itself. Dry-run JSON is byte-identical.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
anthonyshew
added a commit
that referenced
this pull request
Jul 6, 2026
## Why On a 1191-package monorepo with a 5.4MB pnpm-lock.yaml, the fast-parse tier spent ~57ms walking the document with one sequential scanner — almost all of it in three top-level blocks of comparable size (`importers` ~22ms, `packages` ~16ms, `snapshots` ~10ms). That parse sits at the front of every command that builds the package graph. ## What - Split the document positionally at column-zero key lines; each top-level block is an independent subtree for the line scanner, so blocks parse concurrently as standalone mini-documents. - Re-split the three big sections at their child keys into core-count chunks. - Anything the splitter can't prove safe — and any fragment the scanner rejects — falls back to the existing sequential tiers, so the parallel path is a pure optimization: same accepted language, same duplicate-key semantics (`set_once` across sections, insert-checks within them). Isolated parse: **57.2ms → 20.5ms** median; in-process `parse_lockfile` span **~81ms → ~29ms**. ## How to verify - Differential tests now run the parallel splitter on the whole corpus alongside the scanner/saphyr/serde tiers, plus new tests for split boundaries, duplicate keys across fragments and chunks, and a synthetic lockfile large enough to cross the production size gates. `cargo test -p turborepo-lockfiles`: 243 passing. - `--dry=json` output on the monorepo above is byte-identical. - Honest scope note: time-to-first-task on that particular repo is unchanged, because since #13250/#13258 the parse there already overlaps the untracked-file scan. The win is CPU time and the critical path for repos where lockfile parse is the gate (bigger lockfiles, smaller file counts, `turbo prune`, cold starts).
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Untracked-file discovery is the largest single block on the time-to-first-task critical path (~98ms on a 1191-package/43k-file monorepo), and it waited for the package graph solely to compute per-package scan prefixes. That dependency turns out to be vestigial: the scan scope was always exactly the repo-root subtree — every package lives under the repo root (enforced at discovery), the root package's prefix is always in the prefix set, and
UntrackedScopededuplicates nested prefixes. The per-package prefixes never restricted anything beyond the repo root's own prefix.What
repo_prefix_for_repo_index: repo root anchored to the git root — empty for the common case, the subtree prefix when the repo is nested in a larger git repository). It starts as soon as the tracked index exists (~32ms) instead of after graph construction (~87ms).all_package_prefixesis deleted; the prefix-anchoring tests are repurposed to cover the repo-root prefix in both the nested-git-root and matching-root layouts.How to verify
turborepo-scm's untracked/regression suites (191 tests) andturborepo-lib(432) pass unchanged;--dry=jsonbyte-identical on the monorepo above.Stacked on #13250. Compounds with #13251 and #13257.