v2.5 fork bootstrap mechanism — design (analysis-only, no implementation)

[briansrls]

v2.5 fork bootstrap mechanism — design

Author: sunny-otter-371 (Tier 4, orthogonal to substrate + stage workers)
Brief: sunny-wolf-435 msg_b8c4e8d2, operator-direct 2026-05-19
Scope: investigation + design; no implementation in this PR
Sibling-analysis input: PR #3407 (/tmp/v2_stage0_independence_analysis.md)

---

TL;DR

• Bootstrap from v2-compiler. v2-compiler is the only working .dag → .rs engine in the repo today; v2.5 starts as a generated crate produced by running v2-compiler against src/v2.5/. No prior v2.5 binary needed.
• Repo shape: src/v2.5/ holds the new tree's .dag files; src/v2.5/stage0/ is a workspace-member crate whose src/*.rs files are all committed regen output (carrying the Generated by … header). The Cargo.toml and a small main.rs are the only hand-maintained files in the seed.
• Regen mechanism: a shell script scripts/regenerate-stage25.sh (model after the working pre-retirement regenerate-stage0.sh) plus a thin Cargo bin regen_stage25 in a new tools/regen_stage25/ workspace member. Either invocation does the same thing; supporting both lets local dev pick the shorter form and CI invoke whichever is more reliable.
• CI gate: scripts/regenerate-stage25.sh --verify invoked directly as a CI step (or cargo run -p regen_stage25 -- --verify). Exit code is the gate. No libtest filter, no --exact, no cross-tree subprocess dependency. This avoids both of the failure modes documented in PR v2 stage0 independence — investigation (analysis-only, no implementation) #3407 (the vacuous --exact filter and the missing emit_method_template_projection v3 bin).
• Hand-maintained seed surface: 2 files (stage0/Cargo.toml, stage0/src/main.rs); everything else generated. Tighter than v2's current 4-file hand-maintained surface and matches the operator's "no edits to stage0" framing.
• Operator's "infinite labour" reframe applied: D is recommended over the more elegant build.rs-on-every-build path (E) because of an architectural property — committed .rs makes fresh checkouts buildable offline without first building v2-compiler — not because of effort. The architectural property is real; offline-buildability is not a downscope.

---

1. Constraints inherited from PR #3407

The v2.5 mechanism must not re-inherit any of these:

1. No retired scripts that lie. scripts/regenerate-stage0.sh and scripts/check-stage0-freshness.sh exit 1 with "retired — v2 stage0 tree no longer exists" messages left over from T-V2-Retirement. BOOTSTRAP.md continues to document them. v2.5 ships a script that actually works, and the doc never gets out of sync with reality (single producer; the script's --verify is the source of truth).
2. No libtest-filter-as-gate. v2's bootstrap_fixed_point -- --ignored --exact invocation matches zero tests (test path is bootstrap::bootstrap_fixed_point; --exact requires the full path). v2.5 invokes the regen check directly, not through a libtest filter.
3. No cross-tree subprocess dependencies. bootstrap.rs shells out to cargo run -p v3-compiler --bin emit_method_template_projection; that bin no longer exists. v2.5's regen depends only on cargo run -p v2-compiler --release (which is in repo and works).
4. No "the harness needs a v3 thing to test a v2 thing" coupling. v2.5's regen runs in process via subprocess to v2-compiler — no library imports across src/v2.x/ boundaries, no v3 or v4 dependencies.

---

2. Where v2.5 lives and what's hand-vs-generated

2.1 Tree layout (proposed)

src/v2.5/                          # the fork
  std/                             # cool-bee-832 / vivid-lynx-807 own
    node.dag
    stage_types.dag
    fold_node.dag
    ...
  01_tokenize.dag                  # nimble-swift-454
  02_parse.dag                     # eager-hawk-661
  03_resolve.dag                   # (new worker)
  03_normalize.dag                 # calm-crane-724
  04_infer.dag                     # swift-swift-37
  05_emit.dag                      # crisp-dove-396
  compile.dag                      # keen-ibex-355
  artifact.dag                     # loyal-dove-547
  trace.dag                        # calm-bat-839
  stage0/                          # bootstrap seed crate (workspace member)
    Cargo.toml                     # HAND-MAINTAINED (~15 lines)
    src/
      lib.rs                       # GENERATED
      main.rs                      # HAND-MAINTAINED (~5 lines, CLI shim)
      <v25_compiler_*.rs>          # GENERATED (output of regen)
      <std_*.rs>                   # GENERATED
      ...
tools/
  regen_stage25/                   # new workspace member
    Cargo.toml                     # HAND-MAINTAINED
    src/main.rs                    # HAND-MAINTAINED (the regen bin)
scripts/
  regenerate-stage25.sh            # HAND-MAINTAINED (~60 lines)

2.2 Hand-maintained surface — exhaustive list

The fork starts with 4 hand-maintained files:

1. src/v2.5/stage0/Cargo.toml — workspace-member manifest, dependencies (stacker, clap, lazy_static, serde, serde_json; mirror v2's stage0 manifest).
2. src/v2.5/stage0/src/main.rs — CLI shim (parse args, delegate to the generated lib.rs::cli_run). v2's stage0 emits its own main.rs from .dag; v2.5 can do the same, in which case this file disappears and the count drops to 3.
3. tools/regen_stage25/Cargo.toml + tools/regen_stage25/src/main.rs — the regen driver. ~100 lines hand-Rust total. This is in scope per the brief ("a small experimental implementation … if it's small enough") but is design-only here pending operator approval.
4. scripts/regenerate-stage25.sh — shell convenience wrapper.

Compare to v2 today: v2's src/v2/stage0/src/ has 4 hand-maintained Rust files (v2_interpreter.rs, cli_run.rs, rest_transport_facts.rs, plus Cargo.toml). v2.5 starts at the same or better count, with the explicit intent of driving these to zero as v2.5's substrate matures (e.g., when cli_run is itself emitted from .dag).

2.3 Cargo workspace integration

Workspace Cargo.toml gains two new members:

members = [
    # existing members ...
    "src/v2.5/stage0",
    "tools/regen_stage25",
]

Both are workspace members, frozen-buildable per the existing convention (only built/tested when their input subgraph is affected; gated in .github/workflows/ci.yml like v2 + v3 are today).

---

3. The regen mechanism

3.1 What it does

input:  src/v2.5/**/*.dag + dsl/**/*.dag
        target/release/v2-compiler binary
process: build v2-compiler (release)
         run: v2-compiler compile \
              --source-root src/v2.5 \
              --source-root dsl \
              --output-dir <tmp>
         normalize: cargo fmt over <tmp>
         action:
           default → copy <tmp>/src/*.rs → src/v2.5/stage0/src/
                     (excluding hand-maintained files)
           --verify → diff <tmp>/src vs src/v2.5/stage0/src/
                      (same exclusion); exit non-zero on drift
output:  src/v2.5/stage0/src/ matches what v2-compiler would produce from .dag

This is literally what regenerate-stage0.sh did before retirement, retargeted at the new tree.

3.2 Shell-script form (recommended primary)

scripts/regenerate-stage25.sh:

• Builds cargo build -p v2-compiler --release (release profile is required for the perf budget v2's compile pipeline already exercises).
• Runs target/release/v2-compiler compile --source-root src/v2.5 --source-root dsl --output-dir /tmp/v2.5-regen-<pid>.
• Runs cargo fmt --all --manifest-path /tmp/v2.5-regen-<pid>/Cargo.toml.
• With no flags: cp -r /tmp/v2.5-regen-<pid>/src/*.rs src/v2.5/stage0/src/ (excluding hand-maintained file list).
• With --verify: diff -r --exclude=… /tmp/v2.5-regen-<pid>/src/ src/v2.5/stage0/src/ and exit $?.
• Always: cleanup /tmp/v2.5-regen-<pid>/.

3.3 Cargo-bin form (mirrors v3)

tools/regen_stage25/src/main.rs does the same thing in Rust, calling out to cargo / v2-compiler via std::process::Command. Roughly the v3 regen_bootstrap.rs shape — argparse for --verify, run, compare/copy, exit.

The Cargo-bin form has two advantages worth carrying:

• cargo run -p regen_stage25 -- --verify is symmetric with cargo run -p v3-compiler --bin regen_bootstrap -- --verify. Developers and CI scripts that already know v3's pattern transfer over.
• No bash portability concerns. The shell script is fine on Linux/macOS but Rust is platform-stable.

The shell script's advantage is review-cost: ~60 lines of obvious shell vs ~150 lines of Rust. Both can coexist — the script calls into the bin, or vice versa, or they're independent.

3.4 REGEN_OUTPUTS registry (mirrors v3)

The list of "which .rs files under src/v2.5/stage0/src/ are regen-owned vs hand-maintained" lives in one place. Two reasonable hosts:

• In tools/regen_stage25/src/main.rs as a const (mirrors v3's REGEN_OUTPUTS in build.rs:657). Single producer; the regen step and the --verify diff enumerate from this constant.
• In src/v2.5/stage0/build.rs (if v2.5's stage0 needs a build.rs anyway for anything else; unlikely at the seed stage).

The first is simpler. Recommend tools/regen_stage25/src/main.rs::REGEN_OUTPUTS as the canonical list.

3.5 What --verify actually checks

Exit 0 if and only if:

• cargo build -p v2-compiler --release succeeds.
• v2-compiler compile --source-root src/v2.5 --source-root dsl succeeds (zero hard diagnostics).
• The set of generated .rs files (filenames + contents, post-rustfmt) under the temp output dir is byte-identical to src/v2.5/stage0/src/<same names> (excluding the hand-maintained file list registered in §3.4).

Anything else is a verify failure with a diff in stderr.

This is the v2.5 analog of v3's regen_bootstrap --verify (src/v3/compiler/src/bin/regen_bootstrap.rs:84-93).

---

4. CI integration

4.1 What we add

A new job in .github/workflows/ci.yml, gated by the v2.5 affected-set:

v2-5:
  if: github.event.pull_request.draft != true && needs.affected.outputs.v2_5 == 'true'
  runs-on: ubuntu-latest
  steps:
    - name: Build v2-compiler (regen host)
      run: cargo build -p v2-compiler --release
    - name: v2.5 regen verify
      run: bash scripts/regenerate-stage25.sh --verify
    - name: v2.5 stage0 build
      run: cargo build -p v2-5-compiler --release

Key properties:

• Direct invocation, not libtest filter. Avoids the --exact bootstrap_fixed_point defect.
• Subprocess to v2-compiler is in-tree. No reach into src/v3/ for a missing bin.
• Affected-set gating identical to v2 / v3. v2.5 is frozen-buildable except when src/v2.5/** (or related deps) change.
• Two distinct gate signals:
  • regen-verify failure = .dag source and committed .rs disagree (worker forgot to regen, or v2-compiler emitter changed)
  • stage0 build failure = generated code doesn't compile (emitter bug)

4.2 What we do NOT add

• No bootstrap_fixed_point test. v2.5's regen-verify IS the fixed-point check at the byte level; libtest is not in this picture.
• No dependency on any v3 bin.
• No "freshness" gate separate from regen-verify; they collapse to one check.

4.3 First-time-bootstrap freshness

There is no chicken-egg. v2.5 starts with src/v2.5/std/ files written by Tier 0/1 workers; the first regen run produces src/v2.5/stage0/src/*.rs from those .dag files; that's the first commit that has both halves. Verify-mode CI passes from that commit forward.

If a worker lands a .dag change without running regen: CI fails on regen-verify, worker re-runs regen locally, force-pushes, CI passes. Conventional Cargo-style workflow.

---

5. Candidate alternatives considered

(E) build.rs-generated stage0 — no committed .rs

src/v2.5/stage0/build.rs runs v2-compiler at every cargo build, writes to OUT_DIR, lib.rs is include!-stitched.

Pros: zero committed .rs; no PR diff overhead; no "did the worker regen?" failure mode (the build always regens).

Cons:

• Every cargo build -p v2-5-compiler triggers a full v2-compiler self-compile (~30-150s wall-clock per the v2 perf ratchet). Local dev feedback is painful unless aggressive caching is set up.
• Fresh checkouts cannot build offline. Without committed .rs, you need v2-compiler built first, which requires its committed .rs (or the network for binary fetch).
• Workspace ordering: cargo build --workspace runs build.rs scripts in dependency order, but v2.5's build.rs depends on the v2-compiler binary, not the v2-compiler library; the dependency is intercrate-tool, not crate-source. Wiring is fragile.

Verdict: Architecturally interesting but operationally costly. Reject for v2.5's launch shape; revisit when v2.5 self-hosts.

(F) v2.5 bootstraps from v3-compiler instead of v2-compiler

v3-compiler exists and parses .dag. Could be the regen host.

Cons:

• v3 is FROZEN per Cargo.toml:12-13 ("frozen pending v4 program"). Tying v2.5's bootstrap to a frozen compiler is unwise.
• v3's emit model is substrate-driven (emits via substrate.dag reflection), not arbitrary .dag → .rs — would require teaching v3 to emit v2.5's target shape, which is a non-trivial v3 change inside a frozen tree.
• v2 already parses v4 cleanly (per CI gate ci.yml:248-252); v2.5 should be a similar v2-parseable subset by construction.

Verdict: Reject. v2-compiler is the right host.

(G) Vendored / fetched binary seed

Tag a v2-compiler binary, vendor it in repo or fetch from artifact storage; regen uses the vendored binary, not a fresh cargo build.

Cons:

• Binary version drift: the vendored binary lags behind v2-compiler .dag/.rs changes. Either auto-update (extra mechanism) or stale-gate failures.
• Cross-platform: vendoring a Linux x86_64 binary breaks macOS / arm64 dev.
• Adds infrastructure (artifact storage, fetch tooling) that doesn't exist for the project today.

Verdict: Reject. cargo build -p v2-compiler --release is already fast on CI (cached per ci.yml:182-184) and is the simpler dependency.

---

6. Preconditions / verifications

Items that need checking before implementation lands. None are blockers in this PR (design-only) but are required before the regen bin merges.

1. v2-compiler can parse v2.5 substrate. Brief asserts "v2.5 syntax is presumably a v2-parseable subset" because v2 parses v4 cleanly. Verification: once vivid-lynx-807 lands the substrate .dag, run target/release/v2-compiler compile --source-root src/v2.5 --source-root dsl --output-dir /tmp/probe and confirm zero hard diagnostics. Owner: sunny-otter-371 at regen-bin implementation time, or vivid-lynx-807 as part of the substrate PR.
2. Cargo workspace can host two new members without re-cycling cargo's resolver. Adding src/v2.5/stage0 + tools/regen_stage25 should be a workspace-edit only; no impact on src/v2/ or src/v3/. Verify by running cargo metadata and cargo build --workspace post-add. Low risk.
3. No name collision in workspace. Proposed package names: v2-5-compiler (or v25-compiler; Rust crate names disallow .), regen_stage25. Confirm no collision with existing crates. Direct grep of workspace Cargo.toml confirms safe.
4. Affected-set detector recognizes src/v2.5/. The affected job in ci.yml:41-49 computes which trees are touched. A new tree means a new output key (v2_5) + a new detection rule. Mechanical addition.
5. src/v2.5/stage0/src/main.rs is hand-maintained vs generated. v2 emits its main.rs from .dag; v2.5 can follow suit by ensuring the compile.dag stage (keen-ibex-355) emits a main function. If that lands, the hand-maintained surface drops to 3 files. Question for substrate workers.
6. The clean shape of stage0/Cargo.toml. Mirrors v2's tiny manifest; nothing novel. Sample text in §7.

---

7. Concrete artifacts (proposed text)

7.1 src/v2.5/stage0/Cargo.toml (hand-maintained, ~15 lines)

[package]
name = "v2-5-compiler"
version = "0.1.0"
edition = "2021"

[dependencies]
stacker = "0.1"
clap = { version = "4", features = ["derive"] }
lazy_static = "1"
serde = { version = "1", features = ["derive", "rc"] }
serde_json = "1"
ureq = { version = "2", features = ["json"] }

7.2 tools/regen_stage25/Cargo.toml (hand-maintained)

[package]
name = "regen_stage25"
version = "0.1.0"
edition = "2021"

[dependencies]
# stdlib only — uses std::process::Command for subprocess to v2-compiler

7.3 tools/regen_stage25/src/main.rs (hand-maintained, ~120 lines)

Shape:

• Parse --verify flag (exit 2 on unknown args; mirrors v3 regen_bootstrap).
• cargo build -p v2-compiler --release.
• target/release/v2-compiler compile --source-root src/v2.5 --source-root dsl --output-dir /tmp/v2.5-regen-<pid>.
• cargo fmt --all --manifest-path /tmp/v2.5-regen-<pid>/Cargo.toml.
• Walk <tmp>/src/, build set of (relative-path, content) tuples; exclude REGEN_OUTPUTS complement (the hand-maintained file list).
• Default mode: copy each tuple over src/v2.5/stage0/src/<relative-path>.
• --verify mode: load committed file; if content differs, print diff hint + path + exit 1.
• Always: cleanup temp dir on success.

7.4 scripts/regenerate-stage25.sh (hand-maintained, ~60 lines)

Thin wrapper around cargo run -p regen_stage25 -- "$@". Or independent shell implementation if no Cargo-bin form lands. Either works.

7.5 CI step (ci.yml)

v2-5:
  if: github.event.pull_request.draft != true && needs.affected.outputs.v2_5 == 'true'
  needs: affected
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: dtolnay/rust-toolchain@stable
    - name: Cache Cargo (v2-5)
      uses: actions/cache@v4
      with:
        path: |
          ~/.cargo/registry
          ~/.cargo/git
          target
        key: cargo-v2-5-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}-${{ hashFiles('src/v2.5/**', 'tools/regen_stage25/**') }}
    - name: v2.5 regen verify
      run: cargo run -p regen_stage25 -- --verify
    - name: v2.5 stage0 build
      run: cargo build -p v2-5-compiler --release

---

8. Open questions for the operator

1. Tree name. src/v2.5/ per the brief, but the brief also flags "operator may rename to src/v2-next/." Crate name follows (v2-5-compiler vs v2-next-compiler). Pick one before implementation.
2. Where the regen bin lives. tools/regen_stage25/ (separate crate) vs src/v2.5/stage0/src/bin/regen.rs (inside the v2.5 crate, hand-maintained). The former is cleaner (v2.5 crate stays 100% generated except for the manifest); the latter is fewer files. Default: tools/.
3. Shell script vs Cargo bin vs both. Both is allowed; if forced to pick one, prefer Cargo bin (cross-platform, Rust-idiomatic, symmetric with v3). Confirm.
4. Whether main.rs is generated. If compile.dag emits a main function in v2.5 (as v2 does), the hand-maintained file count drops from 4 to 3. Wait for compile.dag worker.
5. What we call the affected-set key. v2_5 is the natural shape but underscores in YAML/JSON keys vary. Confirm with the ci.yml convention.
6. Acceptance timing. This PR is design only. Operator-approved + Tier 0/1 substrate landed → implementation lands as a second PR.
7. Does the operator want path (E) build.rs after all? Per "infinite labour" reframe, effort cost isn't a downscope, but the offline-buildable property of D is architectural. Confirm D is the right launch shape.

---

9. Honesty bar / what I did not verify

• Did not run cargo build -p v2-compiler --release in this session (investigation-only mandate).
• Did not attempt to compile any v2.5 .dag — none exist yet.
• Did not run cargo metadata to confirm workspace-add wouldn't disturb cycle detection. Low risk per the existing src/v2/stage0 + src/v3/compiler precedent, but unverified.
• Did not benchmark v2-compiler's self-compile time on the current host; cited the 150s perf ratchet from bootstrap.rs:704 as the reference value. Actual times will vary.
• Did not produce a skeleton implementation in this PR. The brief permits one if small; I chose to stay design-only so Tier 0/1 substrate workers retain full layout discretion. Skeleton is straightforward and ~250 lines total across the four artifacts in §7.
• v3 regen_bootstrap.rs shape facts (lines, structure) are from a Read in this session of src/v3/compiler/src/bin/regen_bootstrap.rs + src/v3/compiler/Cargo.toml + src/v3/compiler/build.rs:640-715.
• PR v2 stage0 independence — investigation (analysis-only, no implementation) #3407 prior-art facts are restated from my own analysis there; the underlying file evidence is in that PR.

---

10. Cross-stage interface notes

You depend on me (sunny-otter-371) for nothing on the critical path. The regen mechanism doesn't gate any other worker's design — they write .dag, I (or whoever implements this) regen.

I depend on you for:

• cool-bee-832 — src/v2.5/std/ layout (does the regen need to know about a special directory structure, or is it just "all .dag under src/v2.5"?). My current design assumes the latter; if there's a directory split I should know about, escalate.
• vivid-lynx-807 — substrate types' parseability by v2-compiler. If anything in the type kit needs syntax v2 doesn't support, escalate so we can either adjust the type kit or extend v2.
• keen-ibex-355 — whether compile.dag emits main.rs (impacts hand-maintained file count).
• All stage workers — your .dag files are the regen input. Once they land, regen-verify can be exercised end-to-end.

Per the brief: cross-stage discussion goes through PM (sunny-wolf-435), not directly worker-to-worker.

---

11. Status

Design complete; implementation deferred pending operator review + Tier 0/1 substrate. Recommended path: D (commit .rs + Cargo-bin + script regen with --verify). Effort cost is not the bottleneck per the "infinite labour" reframe; D is architecturally the right launch shape because of the offline-buildability property, not because it's small.

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: d257235b · Trigger: schedule
• Thinking: 48s wall

---

✅ The provided diff contains no changed files, so there are no PR-introduced concerns to flag.

[briansrls]

Closing per operator wrap-up directive 2026-05-20.