Skip to content

refactor: Separate package graph node assembly - #13441

Merged
anthonyshew merged 3 commits into
mainfrom
shew/package-node-assembly
Jul 23, 2026
Merged

anthonyshew merged 3 commits into
mainfrom
shew/package-node-assembly

Conversation

@anthonyshew

@anthonyshew anthonyshew commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Why this PR exists

PackageGraphBuilder currently owns two different jobs at the same time:

  1. Observe native repository state: discover workspaces, parse manifests, resolve package-manager behavior, and prepare package metadata.
  2. Maintain graph structure: create the root/sentinel nodes, insert package nodes, reject duplicate names, and keep the package map, petgraph nodes, and lookup indexes synchronized.

That coupling is manageable while PackageJson is the source of package identity and paths. It becomes a blocker for the next PR, where parser-neutral repository knowledge must become authoritative. If we introduce that knowledge directly into the existing builder, the same change would simultaneously alter discovery, package identity, path ownership, duplicate handling, and graph mutation. A regression would be difficult to localize, and reviewers could not tell whether changed graph output came from the new observation model or from rewritten graph assembly.

We need a narrow boundary where already-observed package facts can be handed to graph-owned code:

Before

native discovery + manifest parsing + package facts + graph invariants
                         all inside BuildState

After this PR

native discovery + manifest parsing
                |
                v
     package facts / compatibility payload
                |
                v
      PackageGraphAssembler
      - root and sentinel setup
      - package node insertion
      - duplicate validation
      - lookup/index maintenance

The next PR can therefore replace the source of package facts without also redesigning how graph nodes are assembled. This is the strangler seam: new repository knowledge will feed the assembler, while graph topology remains core-owned and behaviorally unchanged.

Why these fields move together

The package map, petgraph graph, root indexes, and node lookup are one invariant set. Adding a package must update all of them together; otherwise graph traversal and package lookup can disagree. PackageGraphAssembler gives that invariant one owner instead of leaving the fields directly mutable throughout BuildState.

It also makes the distinction between these two existing concepts explicit in one place:

  • PackageNode::Root: the graph implementation sentinel.
  • PackageNode::Workspace(PackageName::Root): the root execution scope.

That distinction matters once repository roots, package roots, and aggregate scopes become separate knowledge concepts.

What changes

This PR extracts a private, graph-owned assembler responsible for:

  • Root and root-workspace node initialization.
  • Structural root edge creation.
  • Package node insertion.
  • Duplicate package-name diagnostics.
  • Capacity reservation and lookup-table maintenance.
  • Returning the assembled graph state when construction finishes.

BuildState remains the caller, so construction order and graph output remain unchanged.

What deliberately does not change

This is a mechanical prerequisite, not the package-knowledge cutover. It does not change:

  • Workspace discovery.
  • Manifest parsing or PackageJson compatibility data.
  • Package identity or path semantics.
  • Dependency splitting or relationship edges.
  • Package-manager or lockfile behavior.
  • External dependency closure calculation.
  • Filtering, task graph, hashing, caching, watch, or prune behavior.

The temporary PackageInfo payload is still present here. The following PR changes package identity/path authority to parser-neutral repository knowledge; doing that here would defeat the purpose of separating the two independently reviewable moves.

Review guidance

The expected result is structural equivalence. Useful review questions are:

  • Are root/sentinel nodes and their structural edge initialized exactly once?
  • Does every package insertion update the package map, graph, and node lookup together?
  • Are duplicate diagnostics preserved?
  • Did relationship and lockfile logic remain in its previous phase?
  • Does finish() transfer the same graph-owned state into PackageGraph?

Stack: #13440 -> #13441 -> #13442 -> #13447 -> #13448 -> #13443

Testing Instructions

  • cargo test -p turborepo-repository
  • cargo lint
  • pnpm run format

@anthonyshew
anthonyshew requested a review from a team as a code owner July 22, 2026 21:47
@anthonyshew
anthonyshew requested review from tknickman and removed request for a team July 22, 2026 21:47
@vercel

vercel Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-designsystem-docs Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-gatsby-web Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-kitchensink-blog Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-nonmonorepo Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-svelte-web Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-tailwind-web Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
examples-vite-web Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
turbo-site Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 4:56pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
turborepo-eve-agent Skipped Skipped Open in v0 Jul 23, 2026 4:56pm

Base automatically changed from shew/package-knowledge-characterization to main July 23, 2026 18:02
anthonyshew added a commit that referenced this pull request Jul 23, 2026
### Description

Part 1 of the package/scope knowledge migration stack.

- Characterize canonical package identities, directories, native
definition paths, root scope behavior, unnamed workspaces, duplicate
diagnostics, and single-package mode.
- Assert workspace discovery parity across npm, pnpm, Yarn, Berry, Bun,
Nub, and Aube.
- Preserve malformed-manifest source paths in observable diagnostics.
- Make no production behavior changes.

Current producers are JavaScript package discovery, `DiscoveryResponse`,
`WorkspaceData`, `JavaScriptToolchain::discover_packages`, and package
graph `BuildState`. Package/scope consumers include graph assembly,
filtering, configuration loading, boundaries, devtools, query, and
microfrontends.

Phase 1 deletion checkpoint: package identity/path consumers no longer
read `PackageJson`; native definition paths are not universally named
`package_json_path`; repository root, root execution scope, aggregate
scopes, and graph sentinel are distinct; the old package/scope producer
and temporary projections are deleted.

Stack: #13440 -> #13441 -> #13442 -> #13447 -> #13448 -> #13443

### Testing Instructions

- `cargo test -p turborepo-repository`
- `cargo test -p turbo --test invalid_package_json_test`
- `cargo test -p turbo --test unnamed_packages_test --test
single_package_graph_test`
- `pnpm run format`
@anthonyshew
anthonyshew merged commit e9b0a29 into main Jul 23, 2026
53 checks passed
@anthonyshew
anthonyshew deleted the shew/package-node-assembly branch July 23, 2026 19:06
anthonyshew added a commit that referenced this pull request Jul 23, 2026
### Why this PR exists

Package identity and path ownership currently remain entangled with
`PackageJson` compatibility data. That makes JavaScript manifest
structure an accidental core package model and prevents Cargo or future
ecosystems from contributing equivalent package facts without
synthesizing JavaScript descriptors.

This PR introduces one immutable, parser-neutral package/scope
generation and makes it authoritative for graph assembly. The package
graph retains the exact generation used to create its nodes, so
consumers cannot observe package facts from a different repository
snapshot.

### What changes

- Add crate-private `RepositoryKnowledge` for repository root, root
JavaScript execution scope, package and aggregate identities,
directories, native definition paths, and open `ToolchainId` provenance.
- Build package/scope knowledge once from existing JavaScript discovery
and the current Cargo compatibility output.
- Use that knowledge as the authority for package graph node assembly.
- Retain `PackageInfo` only as temporary relationship/task compatibility
data. Its package identity, path, scope kind, and provenance are
projected from knowledge.
- Keep one identity between discovery facts and temporary descriptors.
- Reject native definition paths outside the repository, including
symlink escapes.
- Expose narrow knowledge-backed `PackageGraph` queries for later
consumer migrations.
- Update architecture documentation for the package/scope cutover.

### What does not change

This PR does not introduce workspace-root uniqueness, nested workspace
detection, relationship projections, external resolution, task
knowledge, watch generations, or prune behavior. Those concerns move in
their own stack layers or later phases.

Stack: #13440 -> #13441 -> #13442 -> #13447 -> #13448 -> #13443

### Testing Instructions

- `cargo test -p turborepo-repository`
- `cargo test -p turborepo-engine`
- `cargo test -p turborepo-lib`
- `cargo test -p turbo --test cargo_workspace_test`
- `cargo test -p turbo --test final_hash_contract`
- `cargo lint`
- `pnpm run format`
anthonyshew added a commit that referenced this pull request Jul 23, 2026
### Why this PR exists

The repository model now has authoritative packages and scopes, but it
still has no universal way to say which native workspace authority
produced them. Without that fact, core cannot enforce the repository
constraint that one native kind contributes at most one physical
workspace root. Putting this behind another behavioral `Toolchain`
callback would retain the architecture we are replacing.

This PR extends the existing discovery result into a data envelope:
every toolchain contributes packages/scopes and parser-neutral
workspace-root kind/path observations together. Core binds provenance to
the registry entry that produced the envelope and owns validation.

### What changes

- Add open `WorkspaceRoot` kind/path observations and a
`DiscoveredPackages` envelope.
- Change the existing `Toolchain::discover_packages` result rather than
adding another trait method.
- Make JavaScript report its repository root using the authoritative
package-manager command family.
- Make Cargo report the workspace root used by its current metadata
invocation.
- Bind root provenance in core, preventing adapters from spoofing
another `ToolchainId`.
- Require every toolchain that contributes packages to own an accepted
root.
- Reject different physical roots with the same native kind.
- Deduplicate aliases of one physical root while retaining all producing
owners.
- Reject root paths outside the repository, including unresolved paths
beneath symlink escapes.
- Reject JavaScript discovery responses whose manager family disagrees
with the authoritative manager; pnpm versions and Yarn/Berry remain
canonical families.
- Replace release-only duplicate registry assertions with typed errors.

### What does not change

This layer reports only each producer’s base active root. It does not
inspect discovered JavaScript package manifests for nested workspace
authorities; #13448 adds that native behavior. It also does not validate
lockfile/resolution domains or change watch invalidation.

Stack: #13440 -> #13441 -> #13442 -> #13447 -> #13448 -> #13443

### Testing Instructions

- `cargo test -p turborepo-repository`
- `cargo test -p turborepo-engine -p turborepo-lib`
- `cargo test -p turbo --test cargo_workspace_test --test
single_package_graph_test`
- `cargo lint`
- `pnpm run format`
anthonyshew added a commit that referenced this pull request Jul 23, 2026
### Description

Part 4 of the package/scope knowledge migration stack.

Move the first consumer family, `turborepo-scope`, to knowledge-backed
package enumeration and directory ownership:

- Package inference uses real package names and directories, preserving
longest-prefix nested-package selection.
- Default, exclude-only, parent-directory, and changed-package candidate
sets use authoritative package/scope knowledge.
- Cargo aggregate scopes remain selectable by name while remaining
distinct from real packages and the root scope.
- Test aggregate toolchains report their native workspace root through
the shared discovery envelope.
- Reserved root identity is not reconstructed from user-facing package
names.

Dependency/dependent traversal and affectedness propagation remain on
the existing package graph relationships.

Stack: #13440 -> #13441 -> #13442 -> #13447 -> #13443

### Testing Instructions

- `cargo test -p turborepo-scope`
- `cargo test -p turbo --test filter_run_test`
- `cargo test -p turbo --test cargo_workspace_test`
- `cargo lint`
- `pnpm run format`
anthonyshew pushed a commit that referenced this pull request Jul 25, 2026
## Release v2.10.7-canary.1

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/30132657821)

### Changes

- release(turborepo): 2.10.6-canary.5 (#13424) (`ceee06b`)
- chore: Upgrade brace-expansion (#13429) (`d45bf2a`)
- chore: Update tar to 7.5.18 (#13428) (`d0dbd5a`)
- chore: Upgrade js-yaml (#13427) (`c8e36ad`)
- fix: Match JIT inputs for affected tasks (#13426) (`1b95fcb`)
- fix: Preserve project Yarn package extensions (#13425) (`f08f35c`)
- release(turborepo): 2.10.6 (#13431) (`56bf418`)
- chore: Refine bug report template help section (#13433) (`48d609c`)
- fix: Verify version after attempting codemod upgrade (#13446)
(`7ce7c1c`)
- test: Characterize JavaScript package discovery (#13440) (`51a8a03`)
- refactor: Separate package graph node assembly (#13441) (`e9b0a29`)
- fix: Upgrade @vercel/connect (#13450) (`92db24c`)
- refactor: Add authoritative package knowledge (#13442) (`0729f3b`)
- feat: Select runnable task entrypoints (#13452) (`0a9cb5b`)
- refactor: Add workspace root contribution contract (#13447)
(`a66bf45`)
- refactor: Use package knowledge for scope selection (#13443)
(`0e00d89`)
- fix: Prevent mobile homepage overflow (#13453) (`4778467`)
- docs: Document service readiness probes (#13472) (`f925b44`)
- fix: Preserve pnpm root dependencies during prune (#13476) (`0746adc`)
- fix: Preserve pnpm aliased dependencies during prune (#13477)
(`96dc00c`)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew added a commit that referenced this pull request Jul 26, 2026
## Release v2.10.7

> [!CAUTION]
> Versioned docs aliasing failed during the original release run. [View
logs](https://github.com/vercel/turborepo/actions/runs/30162780405).

The npm packages are already published. This PR restores the failed
release PR and advances `version.txt` to `2.10.8-canary.0`.

### Changes

- release(turborepo): 2.10.6 (#13431) (`56bf418`)
- chore: Refine bug report template help section (#13433) (`48d609c`)
- fix: Verify version after attempting codemod upgrade (#13446)
(`7ce7c1c`)
- test: Characterize JavaScript package discovery (#13440) (`51a8a03`)
- refactor: Separate package graph node assembly (#13441) (`e9b0a29`)
- fix: Upgrade @vercel/connect (#13450) (`92db24c`)
- refactor: Add authoritative package knowledge (#13442) (`0729f3b`)
- feat: Select runnable task entrypoints (#13452) (`0a9cb5b`)
- refactor: Add workspace root contribution contract (#13447)
(`a66bf45`)
- refactor: Use package knowledge for scope selection (#13443)
(`0e00d89`)
- fix: Prevent mobile homepage overflow (#13453) (`4778467`)
- docs: Document service readiness probes (#13472) (`f925b44`)
- fix: Preserve pnpm root dependencies during prune (#13476) (`0746adc`)
- fix: Preserve pnpm aliased dependencies during prune (#13477)
(`96dc00c`)
- release(turborepo): 2.10.7-canary.1 (#13478) (`2a02c34`)
- test: Add Terminal UI sanity tests (#13475) (`07ebd1d`)
- chore: Upgrade Next.js (#13482) (`0a329d6`)
- chore: Upgrade brace-expansion (#13483) (`549fc99`)
- fix: Support multiple macOS file watch roots (#13484) (`2f8a962`)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
Preview – turborepo-eve-agent — 6ef0d38c Deployed Jul 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants