Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
4e36cc8
docs: update SELF_HOSTING.md with compositional pipeline + prereq sta…
briansrls Apr 16, 2026
22b2de0
docs: §2.2 model every stage's domain to spec before implementation
briansrls Apr 16, 2026
083bca4
docs: add L2.5 per-stage domain modeling to dependency order
briansrls Apr 16, 2026
479d47f
docs: address 3 review blockers on SELF_HOSTING.md §2.1/§2.2
briansrls Apr 16, 2026
7d2fd0d
docs: §2.3 language-agnostic compiler core + emission layering
briansrls Apr 16, 2026
893c79c
docs: resolve single-authority blocker — pipeline supersedes bootstra…
briansrls Apr 16, 2026
7d761d1
docs: fix §2.3 Shape A/B boundary + add ElaborationSpec + reframe Sty…
briansrls Apr 16, 2026
dedf69d
docs: add Prereq 0.5 (implicit generic instantiation) + §4.4 (lens in…
briansrls Apr 16, 2026
7207052
Merge branch 'pr-464' into swift-ram-158
briansrls Apr 16, 2026
68ee747
Advance reflected substrate and runtime emission
briansrls Apr 16, 2026
a82f080
Advance reflected runtime carrier and address review feedback
briansrls Apr 16, 2026
4599818
Extend payload binding and structural recursion checks
briansrls Apr 16, 2026
cf994c6
Make v3 List structural during bootstrap
briansrls Apr 16, 2026
9ef2e53
Canonicalize specialized generic signature shapes
briansrls Apr 16, 2026
1f6d54c
Advance reflected lens execution path
briansrls Apr 16, 2026
94fe557
Migrate unused-parameters lens to emitted shim
briansrls Apr 16, 2026
5cb1ee0
Add freshness gate for emitted lens snapshot
briansrls Apr 16, 2026
796f198
Tighten reflected realization validation
briansrls Apr 16, 2026
8da23c8
Merge origin/main into swift-ram-158
briansrls Apr 16, 2026
a926166
Drop dead syntax binding fields
briansrls Apr 16, 2026
1e0fd5f
Fix v3 clippy regressions
briansrls Apr 16, 2026
711b7bd
Fix remaining v3 clippy targets
briansrls Apr 16, 2026
86c516e
Finish remaining lens migration review fixes
briansrls Apr 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions INVARIANTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,74 @@ well-grounded. Sustainability rules assume facts have single authorities.
If the modeling is unfaithful, the downstream invariants are protecting
the wrong thing.

## Bounded Substrate Seed

The Rust-native seed that exists before any `.dag` declaration loads is
a ratchet, not an escape hatch. The seed may stay flat or shrink; it may
not grow without explicit deletion elsewhere in the seed or a narrowly
argued exception that names why the new primitive is truly indivisible.

At the current reflection boundary the intended seed is minimal:

- Parser and tokenizer substrate for reading source text.
- Resolve / bootstrap machinery needed to load the first declarations.
- Primitive scalar groundings (`Int`, `Bool`, `String`) and the atomic
identity handles (`NodeId`, `PortId`, `DeclarationId`, `SourceSpan`).

Everything else in the compiler substrate must live as a `.dag`
declaration. `Dag`, `Declaration`, `Behavior`, `TypeConnective`,
behavior payload records, and realization metadata are no longer valid
"just keep it in Rust" categories now that the reflection surface
exists. If a concept is structurally representable and used above the
seed boundary, the correct move is to declare it in `.dag` and attach a
realization, not to enlarge the seed.

The enforcement rule is monotonic: count seed primitives in a tracked
ratchet file or equivalent CI gate, and block PRs that increase the
count. The project can tolerate temporary seed holes only when the hole
is explicitly named, bounded, and shrinking.

## Lenses Are Substrate Declarations

The canonical form of a lens is a `.dag` declaration operating over the
reflected substrate, not a permanently hand-written Rust module. Rust
lenses are tolerated only as bootstrap scaffolds while the remaining
execution path for compiled lenses is being finished.

This invariant has two operational consequences:

- New lens features should prefer substrate-level facts and realization
bindings over Rust-side helper APIs.
- The number of Rust bootstrap lenses is itself a ratchet. Deleting a
Rust lens in favor of its `.dag` form is forward progress; adding a
new permanent Rust-only lens is not.

The point is not aesthetic. A lens declared in the substrate can be
type-checked, realized, diffed, and eventually analyzed by other lenses.
A Rust lens is opaque to the substrate and therefore a standing hole in
the self-inspection claim.

## Every Dependency Is A Substrate Fact

Whenever a downstream consumer needs a fact, that fact must have a
structural home in the substrate or in a declared realization. Consumers
must not reconstruct dependencies from names, parallel tables, or hidden
calling convention knowledge.

Applied to the reflection work, this means:

- "Which field does this projection read?" is a substrate fact carried
by the field label and realized via `FieldBinding`.
- "Which port is the primary result of this node?" is a substrate fact
surfaced compositionally as `result_port`, not a five-arm Rust helper.
- "How do I read this reflected field from Rust?" is a realization fact
in `rust.dag`, not a hardcoded emitter branch.

If a consumer cannot answer its question by following typed edges and
declared bindings, the fix is upstream: add the missing substrate fact
or realization binding. Reconstructing the fact locally is a violation
of Single Authority and a precursor to drift.

## Root-Cause Depth Invariant

This codebase is a DAG — both the language it compiles and its own
Expand Down
53 changes: 33 additions & 20 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,30 +363,32 @@ M5: Meta-process modeling (bootstrap, CI, dev process)
old didn't), the debug is manual — no visibility into WHICH
stage diverged or WHY.

bootstrap.dag already models the building blocks (195 lines,
zero consumers): CompilerStage (8 stages), StageInput/Output,
TransformContract (preserved/recomputed fields per transform),
FieldPropagation (per-stage per-field), ChangeClassification,
BootstrapStrategy (SinglePass/TwoPhase/Additive).
**Authority note (2026-04-16):** `bootstrap.dag`'s stage
vocabulary is superseded by the pipeline composition
(`SELF_HOSTING.md` §2.1). The pipeline's typed function
signatures ARE the stage contracts. `bootstrap.dag`'s 195
lines of `CompilerStage`, `TransformContract`, etc. dissolve
into the pipeline declaration — they are not wired as a
separate authority. See Track 17b below for the updated plan.

Three levels, each building on the previous:

Level 1 — Per-stage structural diffs (Track 17b extension):
regen.dag runs old binary + new binary, compares output
at EACH stage boundary (not just the final pass). When
divergence occurs, the report says "Resolve stage output
differs" instead of "fixed-point failed." bootstrap.dag's
CompilerStage and ArtifactKind drive the comparison.
differs" instead of "fixed-point failed." The pipeline's
typed stage signatures drive the comparison.
Directly helps current dark-emu session.

Level 2 — Stage contracts verified during bootstrap:
Each stage declares a TransformContract: "Resolve takes
a ParseTree, produces a ModuleGraph. Preserved fields:
[ident, span]. Recomputed fields: [module_deps, exports]."
Each stage's function signature declares its input/output
types. A lens verifies each contract holds: "does infer
produce a Dag where all ports have Resolved state?"
The bootstrap loop checks: did the contract hold? If a
stage recomputes a field it claims to preserve, the check
fires at the stage boundary with a typed diff — not at the
end as a cryptic fixed-point failure.
stage's output fails its lens check, the error fires at
the stage boundary with a typed diff — not at the end as
a cryptic fixed-point failure.

Level 3 — Automatic strategy selection (M5 "done when"):
ChangeClassification (AddField with default → Additive,
Expand Down Expand Up @@ -1102,22 +1104,33 @@ before/after benefit):**
- **Size:** medium. Each REST op needs a one-line annotation;
the structural test is new infrastructure.

- **PR 17b: `gunbc/bootstrap.dag` → `gunbc/tools/regen.dag`**
- **PR 17b: pipeline composition → `gunbc/tools/regen.dag`**
- **Authority note (2026-04-16):** `bootstrap.dag`'s stage
vocabulary (`CompilerStage`, `TransformContract`,
`ChangeClassification`, `BootstrapStrategy`) is **superseded**
by the pipeline composition declared in
`src/v3/SELF_HOSTING.md` §2.1. The pipeline's typed function
signatures ARE the stage contracts; `bootstrap.dag`'s parallel
vocabulary dissolves into the pipeline declaration. `regen.dag`
reads the pipeline's stage signatures, not a separate
`bootstrap.dag` type set. See `SELF_HOSTING.md` §2.1 for the
single-authority rationale.
- Currently `regen.dag` hardcodes the 5-step sequence in nested
matches (build → compile → copy → check → rebuild → recompile).
- Replace with `let stages: List<CompilerStage> = [...]` driven
from `bootstrap.dag`, folded over.
- Each step's input/output derives from `StageInput`/`StageOutput`,
not bare strings.
- Replace with stage iteration driven from the pipeline
composition's typed stage functions, folded over.
- Each step's input/output derives from the pipeline's function
signatures, not bare strings or a parallel `StageInput`/
`StageOutput` vocabulary.
- **Extension (M5 Phase 4 Level 1):** add per-stage structural
diffs to the bootstrap loop. When old and new binary diverge,
the report names the STAGE where divergence starts, not just
"fixed-point failed." Directly addresses the dark-emu-36-pr3
bootstrap regression (new binary detects a dependency cycle
the old binary didn't — the diff would localize to the Resolve
stage). See M5 Phase 4 for the 3-level plan.
- **Benefit:** adding a new compiler stage is one edit to
`bootstrap.dag`, not edits in regen/freshness/ci. Pipeline
- **Benefit:** adding a new compiler stage is one edit to the
pipeline composition, not edits in regen/freshness/ci. Pipeline
sequence becomes data. Bootstrap failures become diagnosable.
- **Size:** medium (was small — the per-stage diff adds scope,
but it's the most valuable part).
Expand Down
32 changes: 32 additions & 0 deletions THESIS.md
Original file line number Diff line number Diff line change
Expand Up @@ -926,6 +926,38 @@ evolution of layers, one-file edits for new targets, and
compositional modeling that matches the OSI stack's empirical
scalability proof.

### Self-inspection: the substrate is its own subject

The substrate's structural type layer is now declared inside the
substrate itself. `src/v3/std/substrate.dag` names `Dag`,
`Declaration`, `Behavior`, `TypeConnective`, the identity handles,
and the behavior payload records as ordinary `.dag` declarations,
and `src/v3/spec/rust.dag` attaches Rust-side realizations to those
same declarations. There is no separate meta-layer and no
`substrate_query.rs` bridge module. The substrate is described as
data inside itself, and the Rust compiler structs are one
realization of that data.

This matters because self-hosting is not only about compiling user
programs. The compiler has to inspect its own shape, enforce its own
invariants, and eventually express more of its own passes in `.dag`.
Once the substrate's own types are visible as declarations, every
consumer above them can read them through the same structural path:
typed declaration edges, field access, and realization bindings.
That is the fixed point the thesis has been aiming at. A lens over
the compiler substrate is no longer "special tooling" living
outside the model; it is an ordinary `.dag` consumer walking facts
the substrate already carries.

Self-inspection also tightens the two-groundings story. Static
grounding now includes the substrate's own shape, not only user
concepts. Realization grounding now includes the bridge from
reflected `.dag` fields to concrete Rust fields and accessor
methods. The consequence is practical: adding a new reflected
substrate field is a `.dag` declaration change plus a realization
binding, not a bespoke Rust-side query helper. The compiler grows by
surfacing facts structurally, not by accreting meta APIs.

### Two groundings: static validation vs efficient realization

Every concept in gunbc has **two groundings** that must both be
Expand Down
70 changes: 46 additions & 24 deletions docs/lens-library-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,12 +138,21 @@ analyses like cost and depth don't fit the template.
step. This is the thesis's "cost of change = 1 file" applied
to invariant enforcement.

**The missing primitive.** Canonical form requires a **reflection
API**: the substrate's own types (`Behavior`, `NodeId`, `PortId`,
`Dag`) reachable from `.dag`, and a mechanism for a `.dag`
program to receive a compiled `Dag` as its input. v3 does not
have this at M1(3) scope. Until it does, lenses ship as Rust
modules.
**The remaining missing primitive is execution, not reflection.**
The reflection surface now exists in staged form:

- `src/v3/std/substrate.dag` declares the substrate's structural
types (`Dag`, `Declaration`, `Behavior`, behavior payload
records, etc.).
- `src/v3/spec/rust.dag` carries `TypeRealization` entries for the
reflected substrate types plus per-field `FieldBinding`
mappings to Rust fields and accessor methods.
- `emit_rust.rs` can render reflected field access through those
bindings.

What is still missing is the final step: a `.dag` lens receiving a
runtime `Dag` value as input and being invoked as compiled code. Until
that execution path lands, lenses still ship as Rust bootstrap modules.

**The reflection primitive is the subject of
[`docs/substrate-reflection-design.md`](substrate-reflection-design.md)**
Expand All @@ -164,21 +173,17 @@ two ratchets:
`lens_unused_parameters` migrates are tracked in the seed
ratchet and each deletion is a ratchet tick.

Both invariants live in `docs/substrate-reflection-design.md`
§7.2 and will move to `INVARIANTS.md` when the reflection PR
lands.

**Current Rust lenses are M1 bootstrap — explicitly tracked
scaffolds.** `lens_provenance`, `lens_depth`, `lens_cost`,
`lens_unused_parameters`, and any future lens added before the
reflection primitive lands are bootstrap scaffolds covered by
the ratchet above. They provide immediate value (catching real
violations, validating the per-lens algorithm) and are
structurally identical in shape to what the canonical `.dag`
form will be — pure reader, returns lens-specific output. When
the reflection primitive lands, each bootstrap lens migrates to
`.dag` as a thin rewrite; the algorithm stays the same, the
host language changes.
Both invariants are now first-class entries in `INVARIANTS.md`; the
reflection design doc remains the design history and migration plan.

**Current Rust lenses are bootstrap scaffolds with narrower
remaining scope.** Reflection and field-binding realization are now in
place, so the remaining blocker is not "the substrate is invisible";
it is "compiled `.dag` lenses are not yet invocable on a runtime
Dag value." `lens_provenance`, `lens_depth`, `lens_cost`, and
`lens_unused_parameters` therefore remain bootstrap scaffolds, but
their migration target is now mechanically clearer: the substrate
surface and Rust realization bridge they need already exist.

**Reviewer-enforced gate until the ratchet lands.** Before
adding a new lens as Rust, ask: does this thicken the Rust-lens
Expand Down Expand Up @@ -497,6 +502,21 @@ because:
collect referenced port IDs, compare against the function's
parameter ports.

**Migration status after the reflection work.** The reflection
surface and field-binding realization path this lens needs now
exist:

- `src/v3/std/substrate.dag` declares `Dag`, `Behavior`, `BindNode`,
`BranchPath`, and the rest of the walked substrate.
- `rust.dag` carries `TypeRealization.fields` bindings for those
reflected record fields.
- `emit_rust` can render reflected field reads compositionally.

The remaining blocker is execution plumbing: the `.dag` form of the
lens can now be written against real substrate declarations, but the
project still needs the path that feeds a runtime `Dag` value into a
compiled lens and invokes the compiled function from Rust.

**Signature** (as shipped in `src/v3/compiler/src/lens_unused_parameters.rs`):

```rust
Expand Down Expand Up @@ -560,7 +580,8 @@ part of surface syntax (`SurfaceItem::Fn`) and not preserved past
lowering. Indexing makes the violation addressable without
inventing a naming layer the substrate doesn't support.

**Algorithm** (as shipped):
**Algorithm** (as shipped in Rust bootstrap form, and unchanged in
the planned `.dag` migration):

1. For each `Behavior::Bind` in the Dag with non-empty `params`
(function-shape filter — value bindings have no parameters),
Expand Down Expand Up @@ -853,8 +874,9 @@ The new lenses in this document follow the same shape: pure
function over `Dag` + configuration, returns structured output.
They're sibling modules to the existing ones and can be added
without touching existing lens code. All of the above are Rust
bootstrap form (§1.5); canonical `.dag` form awaits the
reflection primitive.
bootstrap form (§1.5); canonical `.dag` form now awaits only the
runtime invocation path for compiled lenses, not the reflection
surface itself.

The substrate already carries enough information for all three
remaining initial lenses. No substrate changes are required —
Expand Down
Loading