Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ Findings from two reflective analyses (integration loop health, `main@b014746` a

Tracked debt not dispatched with the 2026-04-21 lane wave currently in flight (Lane 1e P3.0, SG-3g-b, SG-2c-4, SG-4b-3, cascade cleanup, algebra inhabitance quartet). Opportunistic slots — no blocking dependencies, can be picked up whenever bandwidth frees.

- **SG-2c growth-discipline checkpoint (2026-04-22)**: SG-2c-1…5 have landed; SG-2c-6 is WIP. Decision doc [docs/briefs/sg-2c-proper-capability-gap.md](docs/briefs/sg-2c-proper-capability-gap.md) names the SG-2c-proper capability blocker concretely (recursive `.dag` function bodies over `List<Token>` with cursor threading + `Token` variant match + `Result<(Surface*, Int), Diagnostic>` short-circuit — load-bearing sub-gap is list-body emission per `src/v3/std/list.dag:13-15`), estimates cadence-vs-pivot scope, and recommends: **natural ceiling** — dispatch SG-2c-6 only under a sharper bar (named same-PR consumer block + non-derivable row family, per SG-2c-3 precedent), do not pre-queue SG-2c-7+, and begin capability-pivot design in parallel as a separate C1-scoped lane rather than serially behind more row lanes. Parallel-authority risk from the accumulated table is judged modest-not-accelerating (rows are data, trigger-gated, exempted in the std/-consolidation ratchet under the `parse_tables.dag` precedent-rule bullet).
- **SG-2c growth-discipline checkpoint (2026-04-22)**: SG-2c-1…5 have landed; the **parse_primary** cluster (SG-2c-6 + SG-2c-7) is closed. Decision doc [docs/briefs/sg-2c-proper-capability-gap.md](docs/briefs/sg-2c-proper-capability-gap.md) names the SG-2c-proper capability blocker concretely (recursive `.dag` function bodies over `List<Token>` with cursor threading + `Token` variant match + `Result<(Surface*, Int), Diagnostic>` short-circuit — load-bearing sub-gap is list-body emission per `src/v3/std/list.dag:13-15`); the **next** open question is the pivot-vs-tiny-extractions trade (see that doc) — do not pre-queue a long tail of one-row `parse_tables` extractions; land capability work for SG-2c proper in parallel. Parallel-authority risk from the accumulated `parse_tables` row families remains judged modest-not-accelerating (exempted under the std/-consolidation `parse_tables.dag` precedent-rule bullet).

- **`algebra.dag` signature/comment reconciliation**: `dsl/std/algebra.dag:265-302` — `FreeMonoid<T>`'s comments promise `index(FreeMonoid<T>, Nat) -> T?`, `map(FreeMonoid<T>, fn(T)->U) -> FreeMonoid<U>`, `fold(FreeMonoid<T>, init: U, fn(U,T)->U) -> U`; the declarations drop the carrier/source parameters (`index: fn(Int) -> T`, `map: fn(fn(T)->T) -> FreeMonoid<T>`, `fold: fn(T, fn(T,T)->T) -> T`). `PartialFunction<K,V>` at `:307-325` has the same issue — comment promises `merge(Map<K,V>, Map<K,V>, fn(V,V)->V) -> Map<K,V>`, declaration is `merge: fn(PartialFunction<K,V>) -> PartialFunction<K,V>` (missing fn argument). One of the two (comments or declarations) is wrong; reconcile. P1 Modeling Faithfulness receipt. Owner: unassigned; small scope.
- **`container_template_algebra_rows` string table is duplicate authority**: `dsl/std/types.dag:133-146` declares the renamed map (PR #651 collapsed both consumers onto a single `container_template_algebra` query); rows still duplicate the authoritative type aliases at `:211-213` (`type List<element> = FreeMonoid<element>`, etc.) across Pascal/snake spellings plus algebra self-identities. Dissolution trigger: when `.dag` gains alias reflection, the rows derive from the aliases directly and the hand-maintained table goes away. P2 Boundary Discipline / P1 receipt. Owner: unassigned; small scope.
Expand All @@ -193,10 +193,10 @@ Tracked debt not dispatched with the 2026-04-21 lane wave currently in flight (L
- **Lane E-M — `MethodSemantics` port-or-subsume (S–M)**: transitive carriers (`CollectionSizeEffect` / `CostShape` / `AlgebraFieldTemplate`) are already in `dsl/std/algebra.dag:408-430`, so M-a is a routine port of ~4 carriers (not a promotion-out-of-compiler-internal). M-b is structural subsumption via v3's `TransformTarget` model. Call is "does v3 structural resolution already carry these facts?" — queue after E-T/C/I land.
- **CI ratchet architecture — exemption widening erodes the per-test timeout ratchet**: surfaced during the 2026-04-21 reflective analysis (`53b3110..ae8825a` range). Commits `37cd6128`, `4898983e`, `f84ed355` hardened the ratchet against CI-log instability, but `2d8396df` widened the exemption list. `scripts/slow-test-exemptions.txt` is 83 lines with no monotonic-shrink rule — `feedback_ratchet_only_down` violated at the meta level (the primary ratchet can drift upward through exemption growth). Dissolution: audit the exemption list, categorize each (stale / paydown / structural), delete stale entries, add a meta-ratchet that fails CI on exemption-count growth or per-exempt budget drift. Dispatch brief: `docs/briefs/ci-ratchet-architecture-audit.md`. Owner: unassigned; queue as infrastructure-principal lane, soon.
- **Stale-receipt sweep in `docs/briefs/`**: surfaced during the 2026-04-21 reflective analysis. Briefs under `docs/briefs/` and prose references in adjacent docs still point at debt that the final code has partially or fully dissolved — the `DeclarationLookup` cleanup is the cited example where brief prose names debt the code has largely resolved. Integration drag, not code regression. Dissolution: sweep `docs/briefs/` for superseded claims post-wave, convert closed briefs to historical receipts under `docs/history/` or add a "Closed by #NNN" line, update inline prose in sibling docs that cite the resolved debt. Small scope, cosmetic. Owner: unassigned; do soon after the 2026-04-21 wave lands (most of its outputs will trigger more brief-staleness). **Include:** `docs/design-pure-bootstrap.md`'s hand-maintained count ("78 hand-maintained .rs files") is stale — live `EXPECTED_HAND_AUTHORED` in the SG-0 census has 89+ entries under `src/v3/compiler/`. Either refresh the number or delete it and point at the live census; detected during #642 review.
- **Compiler–`std/` consolidation program — specific migrations**: end-state defined in [docs/thesis/compiler-std-consolidation.md](docs/thesis/compiler-std-consolidation.md). Each of the migrations below is a standalone lane; together they collapse the compiler-specific type surface toward the positive definition (pipeline + regen + lens-specific return-type carriers + accessor). **Ratchet:** count of `type` declarations in `src/v3/compiler/*.dag` AND `src/v3/lenses/*.dag` that are NOT in the positive-def set AND NOT exempted → 0. Positive-def: pipeline/regen types + lens-API return carriers (`Origin`, `UnusedParameter`, `VariantPayloadShapeLookup` 3-variant, `TemplateArgumentBinding` semantic carrier, etc.). Exempted: `parse_tables.dag` (6 types, pending SG-2c-proper per-row classification). In-ratchet: **strict 2-variant Missing|Found Lookup-pattern carriers across lenses (3 today: `CostLookup`, `SymbolicCostLookup`, `TemplateArgumentLookup`)** + **workaround-shaped infer-helper coproducts with named dissolution triggers (`TemplateArgumentsMatch`, `TemplateArgumentCursor`)**. Baseline: 5. See thesis doc table for per-file disposition.
- **Compiler–`std/` consolidation program — specific migrations**: end-state defined in [docs/thesis/compiler-std-consolidation.md](docs/thesis/compiler-std-consolidation.md). Each of the migrations below is a standalone lane; together they collapse the compiler-specific type surface toward the positive definition (pipeline + regen + lens-specific return-type carriers + accessor). **Ratchet:** count of `type` declarations in `src/v3/compiler/*.dag` AND `src/v3/lenses/*.dag` that are NOT in the positive-def set AND NOT exempted → 0. Positive-def: pipeline/regen types + lens-API return carriers (`Origin`, `UnusedParameter`, `VariantPayloadShapeLookup` 3-variant, `TemplateArgumentBinding` semantic carrier, etc.). Exempted: `parse_tables.dag` (7 types, pending SG-2c-proper per-row classification). In-ratchet: **strict 2-variant Missing|Found Lookup-pattern carriers across lenses (3 today: `CostLookup`, `SymbolicCostLookup`, `TemplateArgumentLookup`)** + **workaround-shaped infer-helper coproducts with named dissolution triggers (`TemplateArgumentsMatch`, `TemplateArgumentCursor`)**. Baseline: 5. See thesis doc table for per-file disposition.
- **`tokenize.dag` → `std/tokenize.dag`** — landed: `Token`, `TokenKind`, `KeywordTokenKind`, `PunctTokenKind`, `LocalPunctSpec`, `StringEscapeSpec` now live in `src/v3/std/tokenize.dag`; compiler-local duplicates deleted. Ratchet drop: 25 → 19.
- **`runtime_mirrors.dag` → `src/v3/std/parse_surface.dag` (tranche 2)** — landed: `DagDifference`, `Surface*`, and related carriers (~14 types) now live in `v3.std.parse_surface`; `runtime_mirrors.dag` deleted. Ratchet drop: 19 → 5. Parse-rule/fragment dissolution triggers (`parse_parser_body.txt`, SG-2b/SG-3f) unchanged.
- **`parse_tables.dag`** — decide per-type whether `BinaryOpLevel` / `BinaryOpRow` / `TopLevelItemKwRow` / `SoftKeywordIdentRow` / `BracketRow` / `PrimaryPrefixRow` stay compiler-API (dispatch-row shapes) or move to `std/syntax.dag` (language-level precedence/dispatch facts). Owner: whoever picks up **SG-2c proper** parser cutover; migration-gate: SG-2c proper completion (the `.dag` parser reveals which rows it dispatches on vs which are pure data). Precedent rule: if any individual row needs to move before SG-2c proper lands, the mover-lane sets the classification and the rest follow the same rule when touched.
- **`parse_tables.dag`** — decide per-type whether `BinaryOpLevel` / `BinaryOpRow` / `TopLevelItemKwRow` / `SoftKeywordIdentRow` / `BracketRow` / `PrimaryPrefixRow` / `PrimaryAtomRow` stay compiler-API (dispatch-row shapes) or move to `std/syntax.dag` (language-level precedence/dispatch facts). Owner: whoever picks up **SG-2c proper** parser cutover; migration-gate: SG-2c proper completion (the `.dag` parser reveals which rows it dispatches on vs which are pure data). Precedent rule: if any individual row needs to move before SG-2c proper lands, the mover-lane sets the classification and the rest follow the same rule when touched.
- **`src/v3/std/*.dag` → `dsl/std/*.dag`** — the whole v3-specific std tree collapses when the file-preference-scaffold dissolves (gated on v2 retirement or `dsl/std/` learning v3 grammar). Largest single consolidation; tracked separately in the **"File-preference rank is a ratified-parallel-authority scaffold"** row (earlier in the 2026-04-21 receipt-closure wave post-merge-debt section).
- **`Node` → `std/node.dag`** — already captured in `project_node_to_std` memory as prior pattern. Relevant to this program as a precedent.
- **Generic `Lookup<T>` in `std/` — dissolve per-lens `CostLookup` / `SymbolicCostLookup` / `TemplateArgumentLookup` duplicates** — three lenses currently declare their own 2-variant `Missing | Found(T)` carrier. Collapse to a single `std/Lookup<T>` generic consumed by every lens that returns a possibly-missing result. 3 types dissolve into 1. Scope is strict 2-variant shape only — carriers with additional semantic variants (e.g., `VariantPayloadShapeLookup`'s `NotPayloadProduct`, `TemplateArgumentBinding`'s `Conflict | NoOp | Append`) stay lens-API. Owner: unassigned; migration-gate: **ready-to-dispatch** (no capability gap — pure type-declaration work; generics already work per the earlier ParseStep discussion).
Expand Down
6 changes: 3 additions & 3 deletions docs/thesis/compiler-std-consolidation.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Positive-definition set (NOT counted against the ratchet):
- Substrate reflection accessor declarations

Exempted (pending named trigger, not counted either direction):
- `parse_tables.dag` — 6 types, exempted pending SG-2c-proper per-row classification
- `parse_tables.dag` — 7 types, exempted pending SG-2c-proper per-row classification

Ratchet-tracked (**migrates to `std/`, counted against the baseline**):
- Compiler-side types that duplicate user-facing concepts (tokenize types moved; parse-surface family moved — remaining debt is lens Lookup/coproduct scaffolds listed in the inventory)
Expand All @@ -136,7 +136,7 @@ Baseline (2026-04-22, measured via `grep -cE "^type [A-Z]"` after the tokenizer

| File | Type decls | Disposition |
|---|---|---|
| `src/v3/compiler/parse_tables.dag` | 6 | exempted pending SG-2c-proper |
| `src/v3/compiler/parse_tables.dag` | 7 | exempted pending SG-2c-proper |
| `src/v3/compiler/operators.dag` | 0 | — |
| `src/v3/compiler/pipeline.dag` | 3 | positive-def |
| `src/v3/compiler/regen.dag` | 1 | positive-def |
Expand All @@ -155,7 +155,7 @@ Baseline (2026-04-22, measured via `grep -cE "^type [A-Z]"` after the tokenizer

End state: 0. Each migration lane reduces the count; positive-definition types track growth separately and are not bounded downward by this ratchet.

**`parse_tables.dag` exemption protocol.** The gap analysis above (§"From `src/v3/compiler/parse_tables.dag` → unclear") says some row *shapes* (`BinaryOpRow`, `TopLevelItemKwRow`, `SoftKeywordIdentRow`, `PrimaryPrefixRow`, etc.) may legitimately stay compiler-API as parser-dispatch shapes, while the *data* they carry (operator precedence levels, keyword-to-item mappings) moves to `std/syntax.dag`. That per-row classification is deferred until SG-2c-proper parser cutover. While deferred, `parse_tables.dag` is **exempted from the primary ratchet count** — its 6 types neither count toward the migration target nor against it. When SG-2c-proper classifies:
**`parse_tables.dag` exemption protocol.** The gap analysis above (§"From `src/v3/compiler/parse_tables.dag` → unclear") says some row *shapes* (`BinaryOpRow`, `TopLevelItemKwRow`, `SoftKeywordIdentRow`, `PrimaryPrefixRow`, `PrimaryAtomRow`, etc.) may legitimately stay compiler-API as parser-dispatch shapes, while the *data* they carry (operator precedence levels, keyword-to-item mappings) moves to `std/syntax.dag`. That per-row classification is deferred until SG-2c-proper parser cutover. While deferred, `parse_tables.dag` is **exempted from the primary ratchet count** — its 7 types neither count toward the migration target nor against it. When SG-2c-proper classifies:
- Rows classified as **compiler-API dispatch shapes** → formally added to the positive-definition set (same status as `pipeline.dag` / `regen.dag` types).
- Rows classified as **language-level data** → become ratchet-tracked migrations to `std/syntax.dag`.

Expand Down
1 change: 1 addition & 0 deletions scripts/check-compiler-std-ratchet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ EXEMPT_ROWS=(
"src/v3/compiler/parse_tables.dag:SoftKeywordIdentRow"
"src/v3/compiler/parse_tables.dag:BracketRow"
"src/v3/compiler/parse_tables.dag:PrimaryPrefixRow"
"src/v3/compiler/parse_tables.dag:PrimaryAtomRow"
)

TRACKED_ROWS=(
Expand Down
39 changes: 25 additions & 14 deletions src/v3/compiler/parse_parser_body.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@
// (`parse_logical_or`, `parse_logical_and`, `parse_comparison`, `parse_additive`,
// `parse_term`) used to open-code the token → `OperatorKind` mapping in five separate
// match blocks. That mapping is now authored as `binary_op_*` data rows in
// `parse_tables.dag` and consumed here via `binary_op_at_level`. SG-2c-6 moves the
// `parse_primary` prefix opener set (`if` / `match` / `{` / `[`) into `PrimaryPrefixRow`
// rows, consumed via `primary_prefix_dispatch`. The parser's control flow — cursor
// mechanics, recursive descent, error recovery, span fusion — remains semantic
// authority in this file.
// `parse_tables.dag` and consumed here via `binary_op_at_level`. The SG-2c
// `parse_primary` cluster: SG-2c-6 `PrimaryPrefixRow` + `primary_prefix_dispatch`
// (prefix openers) and SG-2c-7 `PrimaryAtomRow` + `primary_atom_class` (int/bool/
// string/lambda/ident atomic forms). The parser's control flow — cursor mechanics,
// recursive descent, error recovery, span fusion — remains semantic authority in
// this file.
//
// Dissolution trigger (SG-2c proper): delete this file when parse **logic** is
// structurally `.dag`-owned (SELF_HOSTING §6 Phase 4a direct port or Phase 4b
Expand Down Expand Up @@ -1086,27 +1087,37 @@ impl<'a> Parser<'a> {
None => {}
}
let token = self.bump().clone();
match token.kind {
TokenKind::IntLit(value) => Ok(SurfaceExpr::Literal {
let cls = primary_atom_class(&token.kind).ok_or_else(|| Diagnostic::ParseError {
message: format!("expected primary expression, got {:?}", token.kind),
span: token.span.clone(),
fixes: Vec::new(),
})?;
match (cls, token.kind) {
(PrimaryAtomClass::IntLiteral, TokenKind::IntLit(value)) => Ok(SurfaceExpr::Literal {
value: SurfaceLiteral::Int(value),
span: token.span,
}),
TokenKind::KwTrue => Ok(SurfaceExpr::Literal {
(PrimaryAtomClass::True, TokenKind::KwTrue) => Ok(SurfaceExpr::Literal {
value: SurfaceLiteral::Bool(true),
span: token.span,
}),
TokenKind::KwFalse => Ok(SurfaceExpr::Literal {
(PrimaryAtomClass::False, TokenKind::KwFalse) => Ok(SurfaceExpr::Literal {
value: SurfaceLiteral::Bool(false),
span: token.span,
}),
TokenKind::StringLit(value) => Ok(SurfaceExpr::Literal {
(PrimaryAtomClass::String, TokenKind::StringLit(value)) => Ok(SurfaceExpr::Literal {
value: SurfaceLiteral::String(value),
span: token.span,
}),
TokenKind::Pipe => self.parse_lambda(token.span),
TokenKind::Ident(name) => self.parse_ident_expr(name, token.span),
other => Err(Diagnostic::ParseError {
message: format!("expected primary expression, got {other:?}"),
(PrimaryAtomClass::Pipe, TokenKind::Pipe) => self.parse_lambda(token.span),
(PrimaryAtomClass::Ident, TokenKind::Ident(name)) => {
self.parse_ident_expr(name, token.span)
}
(class, other) => Err(Diagnostic::ParseError {
message: format!(
"internal: primary-atom class {class:?} does not match `TokenKind` {other:?} \
(fix `PrimaryAtomRow` in `parse_tables.dag` or `regen_parse_tables` output)"
),
span: token.span,
fixes: Vec::new(),
}),
Expand Down
Loading
Loading