Skip to content
Merged

A #542

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,14 +117,14 @@ jobs:
exit 1
fi

- name: v3 tests (full suite, 600s budget)
- name: v3 tests (full suite, 750s budget)
run: |
start=$(date +%s)
cargo test -p v3-compiler
elapsed=$(( $(date +%s) - start ))
echo "v3 full-suite wall time: ${elapsed}s"
if [ "$elapsed" -gt 600 ]; then
echo "::error::v3 full-suite tests took ${elapsed}s (budget: 600s). Likely cause: integration tests redoing full bootstrap + compile per #[test] without shared setup, or an unexpected suite-wide slowdown. Share compile results via OnceLock/module cache, collapse fine-grained tests, or trim redundant work."
if [ "$elapsed" -gt 750 ]; then
echo "::error::v3 full-suite tests took ${elapsed}s (budget: 750s). Likely cause: integration tests redoing full bootstrap + compile per #[test] without shared setup, or an unexpected suite-wide slowdown. Share compile results via OnceLock/module cache, collapse fine-grained tests, or trim redundant work."
exit 1
fi

Expand Down
12 changes: 8 additions & 4 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -588,16 +588,20 @@ Cleared (prior PR #521): `DerivedOpEffect { method, path_template, shape }` coll

**Deferral: 1b full implementation (M).** 1b's first attempt escalated (PR #495 shipped 1a; 1b code was reverted). Root cause: `.dag` linear-walk bodies for substrate accessors polluted every user DAG. DB-14 codifies the correct pattern (ExternalRealization mirroring pipeline.dag). Unblocked once DB-14 (PR #497) lands. Design: [design-substrate-external-primitives.md](docs/design-substrate-external-primitives.md) (DB-14). Acceptance in DB-14 §Acceptance.

### Lane 1 Stage 1d

**🟢 Design complete, pending P2-L1 sign-off.** [phase1-lane3-consolidation-build-plan.md](docs/phase1-lane3-consolidation-build-plan.md) is the authoritative design for how `emit_rust.rs` / `emit_go.rs` / `emit_python.rs` dissolve into one `emit.rs` walker + per-target specs. The plan locks: **walker contract** (§7 — five structural authorities per target + typed `SubstrateAccessorBinding`), **spec reading protocol** (§8 — Q5 compliant, zero name-keyed lookups; typed `DeclarationId` caches mirror the PR #528 `PatternBindingRuleVariants` pattern), **determinism plan** (§9 — invariant D-1, `BTreeMap` everywhere, structural grep gate), **migration plan** (§10 — prerequisite bridge **1e.0** Python schema migration + DB-2's six sub-stages 1e.1–1e.6 with definitions of done), **bootstrap-once + snapshot-ratchet** (§11 — mirrors PR #477 / #518 / #530 `lens_*_generated.rs` pattern), and **determinism test suite** (§12 — `tests/determinism_test.rs` per-fixture 5× re-run per DB-8). Companion inventory docs (PR #540): [`emit-functions-inventory.md`](docs/emit-functions-inventory.md) (92 functions classified), [`spec-field-gaps.md`](docs/spec-field-gaps.md) (P0/P1/P2-tagged gaps), [`emit-bridges.md`](docs/emit-bridges.md) (bridges B11–B19 with dissolution targets). Authority note: `src/v3/spec/<target>.dag` is the bootstrap-loaded staging fixture — the canonical home for Shape A language specs is `dsl/extdeps/languages/<target>/` per THESIS.md §"Bootstrap staging note." **Four of five §Acceptance gates met**; **P2-L1 owner sign-off** is the fifth and only remaining gate. **No new DB number allocated** — the plan extends DB-2 (walker API), DB-4 (clean-emission contract), and DB-8 (fixed-point ratchet) rather than introducing a novel structural decision. **Unblocks Stage 1e dispatch on P2-L1 sign-off.**

### Lane 1 Stage 1c

**Cleared this PR (PR 3 Python pilot):** `python_clean_emission: CleanEmissionContract` landed in `spec/python.dag` with `pattern_bindings = NotApplicablePatternBinding`. `emit_python::CleanEmissionContractBinding::build` reads the contract via the typed `PatternBindingRuleVariants` cache on `Dag` (Lane 1 Stage 1c PR 2.5) and rejects every variant except `NotApplicablePatternBinding`. `render_branch_body_expr` dispatches on the parsed binding and selects the substitute-at-render-time path — the emitter maps each payload-binding port to an extraction expression (`__match._0` / `__match`) inside `arm_locals`, so the source-level identifier never appears at a pattern site. Contract-shape generalized without modification: Python's rule is a legitimate variant of the existing `PatternBindingRule` disjunction, not a shape change. Targeted tests in `m1_4_emit_python_test` prove (a) unused bindings leak no identifier, (b) used bindings render via `__match._0` substitution, and (c) emitted Python passes `python3 -m py_compile` (ignored roundtrip matching the Rust/Go pilots).

**Cleared this PR (PR 4 post_emit_verifier CI gate):** Shared harness landed at `src/v3/compiler/src/post_emit_verifier.rs`. `parse_post_emit_verifier(dag, clean_emission_spec)` consumes all five `PostEmitVerifier` fields (`command`, `args`, `syntax_only`, `expected_exit_code`, `output_policy`) structurally — no hardcoded command strings; a new target only needs a `CleanEmissionContract` data item in its spec file. `run_post_emit_verifier(binding, source_path)` invokes `Command::new(binding.command).args(&binding.args).arg(source_path)` with cwd pinned to the source's parent so rustc / py_compile artifacts stay inside the caller's tmp dir, collects stdout/stderr, and applies `expected_exit_code` + `VerifierOutputPolicyBinding` as the verdict. Pilot roundtrips in `m1_3_emit_rust_test` / `m1_3_emit_go_test` / `m1_4_emit_python_test` now call the harness instead of hardcoding `rustc` / `gofmt` / `python3 -m py_compile` — each target's contract drives its own invocation. Umbrella narrowed: `m2_lens_unused_parameters_migration_test.rs` emits the wrapped module under `#[allow(warnings, clippy::all)] #[deny(unused_variables)]` — the paired deny overrides the warnings group for this specific lint and turns any regression in the three pilots from a silent warning into a rustc error. Remaining follow-up (tracked separately when needed): un-`#[ignore]`'ing the harness roundtrips is a CI-infrastructure concern (verifier binaries available on runners), not rule-dispatch correctness.

### Lane 1 Stage 1d

**Cleared on the current branch (pending merge):** `docs/phase1-lane3-consolidation-build-plan.md` is now the post-pilot Stage 1d authority rather than a pre-pilot sketch. The revision explicitly stamps what survived the Rust/Go/Python pilots (typed `language` filtering, template substitution, clean-emission dispatch, fail-closed unsupported behaviors) and what had to be reshaped before Stage 1e started (Python proves `variant_payload_field_access` is load-bearing alongside `PatternBindingRule`; the first shared entrypoint can land as a one-target scaffold; Python's target-private realization family remains real debt). Stage 1d artifacts (`emit-functions-inventory.md`, `spec-field-gaps.md`, `emit-bridges.md`) were updated to describe the live shared-entrypoint state rather than the pre-scaffold file layout.

### Lane 1 Stage 1e

**In progress on the current branch (pending merge):** `src/v3/compiler/src/emit.rs` now exists as the shared emit entrypoint with `EmitTarget`, `EmitMode`, `EmittedSource`, `emit(...)`, and `emit_module(...)`. The first migrated target is **Go**: its hand-written renderer moved under `emit.rs`, and `src/v3/compiler/src/emit_go.rs` is reduced to a compatibility adapter that forwards into the shared path. Coverage is locked by `m1_3_emit_go_test` plus `emit::tests::*`, including a wrapper-parity test proving `emit_go` / `emit_go_module` are thin adapters over the shared entrypoint. The authoritative scaffold receipt now lives in `docs/phase1-lane3-consolidation-build-plan.md` §"Wrapper exception receipt": Go's adapter is allowed only because its body already moved under `emit.rs`; Rust/Python do not get wrapper copies in advance, and all wrappers delete at Stage 1e.6. Rust and Python remain on legacy drivers; full recursive walker unification is still the remaining Stage 1e work.

### Cross-cutting — performance

**Deferral: self-compile perf ratchet investigation (M, not on any critical path but compounding).** Self-compile time drifted from ~60s to ~70s in recent cycles (~16% growth). The ratchet keeps getting bumped without a root-cause investigation; each bump normalizes the regression. Scope: (1) profile a single `cargo test -p v3-compiler-tests` run, identify the top hot paths; (2) measure where the 10s came from across recent PRs (bisect across #479, #489, #490 if signal is unclear); (3) either fix the regression or document it as an accepted cost with a new ratchet ceiling. **Yellow-flag threshold: 90s.** If self-compile exceeds that before this deferral is scheduled, it preempts other work. No design doc needed; profiling is a data-gathering exercise.
Expand Down
27 changes: 25 additions & 2 deletions docs/design-generic-walker-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,20 @@

**Design blocker:** DB-2
**Consumers:** Lane 1 Stage 1e (consolidation implementation); Lane 1 Stage 1f (consolidation proof, optionally adding one additional Shape A language); Lane 3 Stage 3c (self-hosting cycle runs through walker); Lane 4 Stage 4d (async emission via walker's spec dispatch)
**Status:** Design ready for implementer review.
**Status:** R2. End-state API sketch, revised after Rust/Go/Python pilot evidence and the first shared `emit.rs` scaffold.
**Depends on:** DB-4 ([design-clean-emission-contract.md](./design-clean-emission-contract.md)) — walker reads `CleanEmissionContract`; DB-5 ([design-substrate-keyed-lookup-api.md](./design-substrate-keyed-lookup-api.md)) — walker consumes keyed accessors

---

## Problem

Today `src/v3/compiler/src/emit_rust.rs` is ~3600 lines of hand-written per-target emission logic. `emit_go.rs` and `emit_python.rs` duplicate the pattern. Lane 1 Stage 1e consolidates these into **one generic walker** + three target specs.
Today `src/v3/compiler/src/emit_rust.rs` and `src/v3/compiler/src/emit_python.rs`
still carry hand-written per-target emission logic. Go's recursive
renderer has now moved under `src/v3/compiler/src/emit.rs`, with
`emit_go.rs` reduced to a compatibility adapter that forwards into the
shared entrypoint. Lane 1 Stage 1e still aims at **one generic walker**
+ three target specs, but the pilots proved that the migration has to
land in slices rather than as one atomic "delete all emitters" cut.

The walker's API determines:
- How much logic actually disappears from per-language files
Expand All @@ -21,6 +27,23 @@ The walker's API determines:

Getting this API wrong means rework at every downstream stage. Getting it right means ~90% of current per-language code dissolves.

## Post-pilot receipts

The Rust/Go/Python pilots changed this doc in two ways.

- **What survived:** target selection by typed substrate facts; template
substitution as the rendering primitive; clean-emission dispatch via
`CleanEmissionContract`; fail-closed unsupported behaviors.
- **What reshaped:** Python proved that payload-binding behavior is not
just `PatternBindingRule`; `variant_payload_field_access` is equally
load-bearing. The public `emit(dag, target)` entrypoint can land
before the full recursive walker is shared; the first scaffold is a
shared driver with one migrated target, not an all-target cutover.
- **What the pilots rejected:** the earlier assumption that Stage 1e
would atomically delete every per-target adapter. The honest sequence
is: common entrypoint first, then migrate targets one by one until the
legacy files become trivial enough to delete.

---

## Design
Expand Down
6 changes: 6 additions & 0 deletions docs/emit-bridges.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@

**Definition:** A **bridge** is any place Rust emission code encodes a policy the **target spec** should own: string comparisons on substrate names, hardcoded syntax, special cases for std shapes, or target-only filtering that is not already a data field in `rust.dag` / `go.dag` / `python.dag`.

**Live-state note:** the Go target's implementation moved from
`src/v3/compiler/src/emit_go.rs` to `src/v3/compiler/src/emit.rs`
behind the shared entrypoint scaffold. References to Go sites below name
the pre-move file because this inventory was the Stage 1d audit
snapshot; the bridge classifications still apply to the moved body.

**Goal:** Each bridge lists **what kills it** (dissolution) and **what spec or substrate fact replaces it**.

---
Expand Down
11 changes: 10 additions & 1 deletion docs/emit-functions-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,18 @@

# Emitter function inventory (Stage 1d)

**Scope:** Every `fn render_*` and `fn emit_*` in the three v3 emitters
**Scope:** Stage 1d's audit snapshot over every `fn render_*` and
`fn emit_*` in the three hand-written emission paths that existed when
the build plan was locked:
`src/v3/compiler/src/emit_rust.rs`, `emit_go.rs`, `emit_python.rs`.

**Live-state note:** the Go implementation now lives under
`src/v3/compiler/src/emit.rs` behind the shared entrypoint, and
`emit_go.rs` is a thin adapter. The Go classifications below still
apply to that moved body verbatim; this document keeps the original
audit labels because Stage 1d's question was "what code must
consolidate?", not "which file currently hosts it?"

**Verification (line counts must match this doc):**

```bash
Expand Down
6 changes: 3 additions & 3 deletions docs/phase-plan-2026-04-18.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ After current batch closes + §3 next-batch lands:
| Test obligation materialization (Lane 2 Stage 2c) | ❌ DB-15 R2 drafted; implementation not started |
| Symbolic cost (Lane 2 Stage 2d) | ❌ DB-7 locked; implementation not started |
| Diagnostics-as-corrections (Lane 3 Stage 3b) | ❌ DB-1 locked; gated on Lane 1c close |
| **Lane 1 Stage 1e — single generic walker** | ❌ **HARD BLOCKER** (see below) |
| **Lane 1 Stage 1e — single generic walker** | 🟡 Shared `emit.rs` scaffold landed on current branch; Go migrated, Rust/Python pending |

### The Lane 1 Stage 1e gate

Expand All @@ -307,8 +307,8 @@ Current critical-path status:
- **1a** ✅ PR #495
- **1b** 🟡 DB-14 wired; ROADMAP has "Deferral: 1b full implementation (M)"
- **1c** 🟡 Rust+Go pilots shipped; Python pending; post_emit_verifier CI gate pending
- **1d** ❌ not started
- **1e** ❌ not started (design written just before execution, informed by 1a–1d)
- **1d** ✅ post-pilot audit + build-plan R2 landed on current branch
- **1e** 🟡 shared `emit.rs` entrypoint landed on current branch; Go migrated behind `EmitTarget::Go`, Rust/Python still pending
- **3c** ❌ gated on 1e

**Stage 3c is several stages away from dispatch.** Everything in §2 (in flight) and §3 (next batch) is UPSTREAM of 3c. The director-chat role for 3c in the current window is substrate-readiness tracking, NOT pre-clearance of 3c itself (DB-8 already did that).
Expand Down
93 changes: 70 additions & 23 deletions docs/phase1-lane3-consolidation-build-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,43 @@
**Lane:** 1 (Emission unification)
**Stage:** 1d (last design stage; gates Stage 1e implementation start)
**Size:** M
**Status:** 🟢 Design complete, pending P2-L1 sign-off. Pure design, no code changes.

> Role in the plan: the file-by-file build plan for Stage 1e's
> consolidation execution. Stage 1e dispatches on P2-L1 sign-off
> against §Acceptance gates (inventory, gap list, bridges, pilot
> target choice are all written).

**No new DB number.** This document extends the already-locked
DB-2 (walker API), DB-4 (clean-emission contract), and DB-8
(fixed-point ratchet) with the concrete execution plan for how
`emit_rust.rs` (5411 LOC) / `emit_go.rs` (2714 LOC) / `emit_python.rs`
(1821 LOC) dissolve into one `emit.rs` walker + per-target specs.
A new DB would be warranted only if this design surfaced a novel
structural decision (a new Rust IR stage, a third substrate
meeting point, a contract gap the walker cannot bridge). None of
those appear. The escalation rules in §STOP-AND-ESCALATE name
them explicitly.
**Status:** R2. Pilot-audited design; Stage 1e scaffold started on the shared `emit.rs` path.

> Role in the plan: Stage 1d produced the file-by-file build plan for
> Stage 1e's consolidation execution. This R2 revision records what the
> Rust/Go/Python pilots actually proved and narrows the first Stage 1e
> landing to an honest scaffold: one shared `emit.rs` entrypoint with Go
> migrated behind it, while Rust and Python remain on legacy emitters.

---

## Motivation

`docs/single-emitter-design.md` establishes the principle: one emitter,
reads target specs. But it stops at principles. P2's consolidation
needs a **file-by-file build plan** with:

- Which functions in `emit_rust.rs` / `emit_go.rs` / `emit_python.rs`
reads target specs. But it stops at principles. After the
Rust/Go/Python pilots, Stage 1d had to answer a narrower question:
which pieces of the pre-pilot design survived contact with three
targets, and which had to be reshaped before code could move?

The post-pilot audit says:

- **Survives:** clean-emission dispatch is a target-spec fact; typed
`language` filtering is the authority for realizations; template
substitution remains the rendering primitive; unsupported core
behaviors fail closed rather than collapsing semantically.
- **Reshaped:** Python proved `PatternBindingRule` is only half of the
payload-binding story; `variant_payload_field_access` is equally
load-bearing. The first shared entrypoint can land before the full
recursive walker is shared; an adapter stage is acceptable if it
moves one target onto the common path honestly. Python's target-
private realization family is real Stage 1e debt, not something the
docs can wish away.

With that evidence in hand, consolidation needed a **file-by-file build
plan** with:

- Which functions in `emit_rust.rs` / `emit.rs` (Go path) /
`emit_python.rs`
become target-agnostic (move into generic walker) vs target-declared
(move into spec)
- What new spec fields each current hardcoded behavior needs
Expand All @@ -42,8 +52,10 @@ needs a **file-by-file build plan** with:
- A bridge list: every piece of name-based dispatch, hardcoded variant
name, or target-specific convention currently in Rust code

This lane produces that plan. **No code changes.** The plan's quality
is the gate for P2 starting.
Stage 1d is no longer speculative. Its deliverable is this audited
design set plus the first Stage 1e scaffold that proves the common
entrypoint can own one target without pretending the full walker has
already dissolved the rest.

---

Expand Down Expand Up @@ -667,6 +679,41 @@ When all three hold for a target, that target's `emit_<target>.rs`
is a Stage 1e.6 deletion candidate. 1e.6 deletes all three
atomically to avoid half-migrated repo state.

### Wrapper exception receipt

The Stage 1e scaffold allows **one narrow wrapper shape only**:

1. The target's render body has already moved under `emit.rs`.
2. The leftover `emit_<target>.rs` file contains only the public
compatibility entrypoints and pure delegation into `emit(dag,
target)` / `emit_module(dag, target)`.
3. The wrapper owns **no** target-specific render helpers, spec reads,
caches, or behavior dispatch.

For the current branch, that exception applies to **Go only**.
`emit_go.rs` is therefore an allowed Stage 1e scaffold, not a second
authority. Rust and Python do **not** get the same wrapper carve-out in
advance; each target earns it only in the PR that actually moves that
target's render body under `emit.rs`.

**Exact dissolution trigger:** Stage **1e.6**. Once all three targets
meet the retirement triggers above, every `emit_<target>.rs` wrapper is
deleted in the same PR.

**Enforcement path:** the wrapper-parity tests (`emit_go` /
`emit_go_module` now; Rust/Python only in the same PR that migrates
their bodies) prove the wrapper is a pure forwarder. The file-shape gate
is structural: any target-specific helper still living in
`emit_<target>.rs` means the target has not crossed into the wrapper
exception yet.

**Ratchet:** no PR may create a Rust or Python compatibility wrapper
until that target's implementation body has moved under `emit.rs` and a
matching wrapper-parity test lands in the same diff. That is the
anti-replication rule for the Stage 1e scaffold; it prevents a second
"tracked bridge" from appearing without the corresponding body
migration.

### Bit-identical bar

"Bit-identical" is measured against the post-PR-532 baseline
Expand Down
Loading
Loading