diff --git a/docs/design-reference-resolution-provenance.md b/docs/design-reference-resolution-provenance.md new file mode 100644 index 00000000000..f63fa0d3d6b --- /dev/null +++ b/docs/design-reference-resolution-provenance.md @@ -0,0 +1,185 @@ +> Part of: [post-l15-phase-plan.md](./post-l15-phase-plan.md) | Unblocks: Scheduled deletion of user-range `ResolvedByName` AtomPayload bridges (NOT compiler-internal `declaration_by_name` call sites) | Consumes: compiler-as-dependency-analyzer framing + +# Design DB-17 — Reference-resolution provenance (user-range AtomPayload scope) + +**Design blocker:** DB-17 (substrate amendment that records HOW a user-range resolved identifier was resolved — structural walk vs name-lookup fallback) +**Consumer:** `lens_structural_resolution.dag` — walks user-range AtomPayload edges, reports `ResolvedByName` occurrences reachable from user-range roots. +**Status:** Proposed. Awaiting review. +**Scope is narrow.** DB-17 addresses ONE class of `declaration_by_name`-shaped debt: user-range references whose lowering fell back to name-keyed resolution when a structural walk failed. Compiler-internal call sites of `declaration_by_name` (bootstrap `substrate_markers` caches, pipeline-authority wiring, emitter algebra lookups) are a SEPARATE class with a different dissolution path — see §"What DB-17 does NOT cover" below. +**Origin:** scheduled-deletions reclassification pass (2026-04-17). Codex review on PR #507 correctly flagged that keying the scheduled deletion to the helper name conflates multiple classes; this revision narrows DB-17's scope to one specific class. + +--- + +## Problem + +Today, when lowering resolves an identifier **in user-range code**, the resulting `AtomPayload::ResolvedIdentifier(DeclarationId)` carries the *result* of resolution (the `DeclarationId`) but not the *provenance* — the path by which resolution succeeded. Two very different code paths produce the same substrate shape: + +1. **Structural walk** — lowering follows parent-chain / scope-chain edges from the reference site to a declaration reachable through structural parentage. This is the authoritative path: the reference is grounded in the compiler's dependency DAG. +2. **Name fallback** — lowering calls `declaration_by_name(name)` (or an analogous helper that maps a `String` to a `DeclarationId` via a name table). This works today for std/bootstrap lookups where cross-module structural resolution isn't yet wired, but it's a bridge: the reference reaches a declaration by **string identity**, not by structural edge. + +Post-resolution, both produce `AtomPayload::ResolvedIdentifier(id)`. A lens walking `d.nodes` cannot distinguish them. Any user-range bridge that fell back through name-keyed resolution becomes invisible in the substrate. + +This blocks the scheduled deletion of **user-range name-fallback bridges**: to dissolve them, we need to enumerate them structurally, audit that every legitimate case has a structural-walk alternative, and enforce that no new name-fallback resolutions enter user-range code. Without provenance in the substrate, "enumerate" collapses to grep — the heuristic-over-source failure mode the dependency-analyzer framing rules out. + +More broadly: the **dependency graph is incomplete until every user-range reference is grounded by a structural edge**. Name-keyed resolution is a bridge that bypasses the edge. The compiler-as-dependency-analyzer thesis requires that this bridge be either structurally visible or absent. + +## What DB-17 does NOT cover + +`declaration_by_name` (the Rust helper at `dag.rs:1459`) has ~83 call sites across the compiler. DB-17 covers exactly one class of them — the user-range lowering fallback. The other classes are NOT unblocked by this DB: + +1. **Bootstrap substrate-marker caches** (`dag.rs:1616-1654+`). The compiler looks up `"Int"`, `"Bool"`, `"String"`, `"Value"`, `"Transform"`, `"Branch"`, `"Loop"`, `"Bind"`, `"Main"`, `"DeclarationRef"`, `"TypeRealization"`, `"OperatorRealization"`, `"BehaviorRealization"` by name to populate `substrate_markers`. This is a compiler-internal cache, not a resolution fallback at a user-range declaration. Dissolves via a separate substrate amendment (marker field becoming a typed edge rather than a name-keyed cache) — NOT via DB-17. + +2. **Pipeline-authority wiring** (`bootstrap.rs:216, 362, 448`; `pipeline_authority.rs:35, 65, 162`). Bootstrap walks the Dag by name for `PIPELINE_REALIZATION_META`, `"parse"`, `"PipelineStageBinding"`, `"PipelineSnapshotKind"` to wire realization edges. This is bootstrap-time machinery, not user-range lowering. Dissolves when self-hosting makes the compiler's own pipeline.dag lens-able as ordinary user code. + +3. **Emitter algebra lookups** (`emit_go.rs:2167, 2378`; `emit_python.rs:1233, 1243, 1390, 1522`). Emitters look up algebra declarations (`"OrderedRing"`) or behavior parents by name at emission time. This is emission-layer dispatch, not substrate resolution. Dissolves when Lane 1e consolidates emitters and target-specific dispatch becomes spec-declared. + +**DB-17's lens is NOT a joint dissolver for all of these.** The ROADMAP scheduled-deletions table splits them into separate rows with separate dissolution paths. The "DB-17 covers ALL `declaration_by_name` uses" framing (from an earlier revision of this doc) was wrong. + +--- + +## Design + +### Split `AtomPayload::ResolvedIdentifier` into two variants + +Today (substrate schema, simplified): + +```rust +pub enum AtomPayload { + UnresolvedIdentifier(String), + ResolvedIdentifier(DeclarationId), + TypeParam(String), + Literal(LiteralBits), + // ... +} +``` + +After DB-17: + +```rust +pub enum AtomPayload { + UnresolvedIdentifier(String), + ResolvedByStructure(DeclarationId), // NEW — resolved via structural walk + ResolvedByName(DeclarationId), // NEW — resolved via name fallback + TypeParam(String), + Literal(LiteralBits), + // ... +} +``` + +`ResolvedIdentifier` is removed; every call site that produced it picks which variant applies. Lowering that walks a structural edge produces `ResolvedByStructure`; lowering that calls `declaration_by_name` (or any name-keyed helper) produces `ResolvedByName`. + +### Consumer migration + +Every consumer that currently matches on `ResolvedIdentifier(id)` updates to match both variants. For consumers that don't care about provenance, this is mechanical: + +```rust +// Before +AtomPayload::ResolvedIdentifier(id) => use(id), + +// After +AtomPayload::ResolvedByStructure(id) | AtomPayload::ResolvedByName(id) => use(id), +``` + +A small helper (`resolved_id(payload) -> Option`) can reduce boilerplate for consumers that uniformly ignore provenance. But the default is explicit pattern-matching — the provenance fact is visible at every read site, which is the whole point. + +### Lens enforcement unlocks + +Once DB-17 lands, `lens_structural_resolution.dag` gets a second variant: + +```dag +type StructuralResolutionViolation + = UnresolvedArrowBody { arrow: DeclarationRef } + | NameKeyedReference { site: DeclarationRef, resolved_to: DeclarationRef } + +// A NameKeyedReference violation fires whenever ResolvedByName is reachable +// from a user-range root. Bootstrap/std ranges are exempt until M2 module +// scoping lands (at which point all resolutions are structural). +``` + +The lens walks `d.nodes`, traverses Atom payloads, and reports any `ResolvedByName` found in user-range code. **No grep, no allowlist, no string heuristic** — the substrate fact carries the information the lens queries. + +### Dissolution of `ResolvedByName` + +`ResolvedByName` is itself a scheduled-deletion row in the ROADMAP (the user-range AtomPayload row). Dissolution trigger: M2 module scoping lands. At that point every user-range reference resolves structurally; the variant becomes unreachable; a final PR removes it from the substrate. + +Between DB-17 landing and M2 module scoping landing, `ResolvedByName` is the explicit bridge for user-range fallback — visible, countable, lens-enforced. + +**This does not dissolve compiler-internal `declaration_by_name` call sites.** Those dissolve via separate paths (bootstrap substrate_markers via substrate amendment; pipeline/emitter sites via self-hosting). See §"What DB-17 does NOT cover" above. + +--- + +## Rejected alternatives + +- **Add `provenance: ResolutionProvenance` metadata to a single `ResolvedIdentifier` variant.** + ```rust + ResolvedIdentifier { decl: DeclarationId, provenance: ResolutionProvenance } + ``` + Rejected because the fact belongs at the variant level, not as a metadata field. The substrate's coproduct axis captures "distinct kinds of reference"; folding provenance inside loses structural visibility. Also: consumers that care about provenance must unwrap the metadata; consumers that don't care read `decl` directly, which silently ignores provenance. The variant split makes every read site acknowledge provenance explicitly. + +- **Side table: map of `(Port, DeclarationId) → ResolutionProvenance`.** + Parallel authority (violates D-2). The substrate's Atom is the authority for resolution; a side table forks the fact. Rejected. + +- **Grep over compiler source for `declaration_by_name` call sites.** + Heuristic over source text; can't distinguish sanctioned bootstrap path from user-range bridge; falls to renames and helper indirection. The exact failure mode the dependency-analyzer framing rules out. Rejected. + +- **Leave `ResolvedIdentifier` unified; rely on audit cadence.** + "Audit every few PRs" has the same drift problem as grep — invisible new uses land in the interval. Substrate fact is the only non-heuristic signal. Rejected. + +- **Add provenance at the `Port` level instead of `AtomPayload`.** + Port carries resolved type info, not reference provenance; the fact lives where the resolution happens (the Atom identifier), not where the type lives (the Port). Rejected. + +--- + +## Implementation scope + +1. **Substrate change.** `src/v3/compiler/src/dag.rs` — split `AtomPayload::ResolvedIdentifier` into `ResolvedByStructure` and `ResolvedByName`. Delete the old variant. +2. **Lowering (user-range paths only).** `src/v3/compiler/src/lower.rs` — call sites that currently produce `ResolvedIdentifier` at user-range resolution points update. Structural-walk paths produce `ResolvedByStructure`; user-range name-fallback paths produce `ResolvedByName`. Compiler-internal `declaration_by_name` call sites (bootstrap substrate_markers, pipeline authority, emitter algebra) do NOT produce AtomPayload — they consume the helper directly for their own purposes and are out of scope for this change. +3. **Infer.** `src/v3/compiler/src/infer.rs` — match arms on `ResolvedIdentifier` update to handle both variants. A helper `resolved_id(payload) -> Option` reduces repetition where provenance is uniformly ignored, but match sites are explicit. +4. **Reflection.** `src/v3/std/substrate.dag` — the reflected `AtomPayload` type gains both variants. Existing lenses that match on `ResolvedIdentifier` update. +5. **Lens.** Write `lens_structural_resolution.dag` as the DB-17 consumer. Variants: `UnresolvedArrowBody` (lens-able today against existing substrate), `NameKeyedReference` (DB-17-enabled). Acceptance test: `NameKeyedReference` lens fires on any user-range DAG with `ResolvedByName` AtomPayload reachable from a user-range root. +6. **Scheduled-deletions update.** Flip the user-range `ResolvedByName` AtomPayload row from "Needs DB-17" to the live lens path when the lens lands. The compiler-internal `declaration_by_name` row stays on its separate compiler-source-ratchet path. + +Size: **M** for DB-17 substrate + lowering + consumer migration; **S** additional for the lens itself. Together roughly one lane-stage of work. + +--- + +## Open questions + +1. **Bootstrap range vs user range — how is the boundary represented?** The lens needs to exempt name-keyed resolutions inside std/bootstrap files while firing on user-range references. Today `user_start` / declaration-id comparisons serve this purpose in Rust-side code, but the lens needs a substrate-visible fact. Option: a `Declaration.source: DeclarationSource` edge or similar that distinguishes bootstrap/std/user ranges. This might be a sibling DB (DB-18?) or part of DB-17's scope. Current proposal: separate DB if the fact isn't already visible; otherwise cite the existing substrate edge. + +2. **Does DB-17 need to handle `ResolvedIdentifier` inside type declarations, or just in Behavior-emitted references?** The intuition says "all resolution sites." The implementation scope for "all sites" is larger. Recommendation: handle all, since half-coverage means the lens has blind spots. + +3. **Does the split affect `authored_name_at` or related span-based APIs?** Those resolve names to spans, not to declarations directly. Likely orthogonal — `authored_name_at` is already name-based by design (it's about source position). DB-17's scope is resolved references, not span queries. Confirm at implementation time. + +--- + +## Acceptance + +- [ ] `AtomPayload::ResolvedIdentifier` removed from `dag.rs`; `ResolvedByStructure` and `ResolvedByName` added +- [ ] Every lowering call site updated to produce the appropriate variant +- [ ] Every consumer (infer, lenses, emit) updated to handle both variants +- [ ] `src/v3/std/substrate.dag` reflects the new shape +- [ ] `lens_structural_resolution.dag` written with at least `NameKeyedReference` variant; acceptance test fires on a fixture that triggers `declaration_by_name`-style fallback in user-range code +- [ ] Scheduled-deletions table entry for `declaration_by_name` flipped from "Needs DB-17" to the live lens path +- [ ] Existing bootstrap tests green; the split should be semantically invisible to correct consumers +- [ ] A regression test: a fixture that deliberately uses name fallback in user-range code produces a `NameKeyedReference` diagnostic via the lens + +--- + +## Associations + +- **Compiler-as-dependency-analyzer thesis** (tonight's framing) — DB-17 is the substrate consequence: every reference must be structural; name-fallback bridges are explicitly marked when they exist. +- **Scheduled-deletions ROADMAP section** (same PR as this DB) — DB-17 unblocks the `declaration_by_name` and NameKeyedReference rows. +- **`lens_structural_resolution.dag`** (separate implementation work; other session) — DB-17 is a prerequisite for the second lens variant (`NameKeyedReference`). +- **M2 module scoping** — dissolves `ResolvedByName` entirely; DB-17's new variant is itself scheduled for deletion. +- **`src/v3/compiler/src/dag.rs::AtomPayload`** — the substrate type being split. +- **`src/v3/compiler/src/lower.rs::declaration_by_name`** — the bridge DB-17 makes structurally visible. +- **`src/v3/std/substrate.dag`** — reflected substrate that lenses walk; gains the two new variants. + +--- + +## Why this is worth a DB + +Without DB-17, `declaration_by_name`'s scheduled deletion has **no structural enforcement path**. The scaffold persists because: (a) grep is heuristic and can't tell good fallback from drift, (b) audit cadence misses interval drift, (c) the dependency DAG is incomplete until every reference is structural. DB-17 makes the bridge visible in the substrate so the lens can walk it. Once visible, the scaffold counts down to zero under structural enforcement — not under discipline, not under audit. + +This is the compiler-as-dependency-analyzer thesis cashing out: **every named scaffold gets a structural enforcement path, or the substrate amendment that unlocks one is filed.** diff --git a/src/v3/ROADMAP.md b/src/v3/ROADMAP.md index 1e52a293846..d57a9ca7a95 100644 --- a/src/v3/ROADMAP.md +++ b/src/v3/ROADMAP.md @@ -505,6 +505,56 @@ Sub-stage status (as of 2026-04-17): GitHub issues for this kind of tracking are **closed with a pointer here.** Issues exist for external coordination (user-facing bug reports, security advisories); internal deferrals do not live in issues. +## Scheduled deletions — scaffolds with named dissolution triggers + +**Discipline:** every scaffold in the live substrate lives here with an explicit dissolution trigger, the upstream work it's blocked on, and its enforcement path. When the trigger fires, a PR deletes the scaffold AND removes the row. Unscheduled scaffolds are violations of the scaffold-boundary invariant. + +**Relationship to "Active deferrals":** deferrals name work that is in flight; scheduled deletions name artifacts that will disappear. A deferral may REFERENCE a scheduled deletion (e.g., "this sub-stage dissolves `ArrowBody::Pending` per the scheduled-deletions row"), and a scheduled deletion may reference a design blocker (DB-NN) that unlocks structural enforcement. + +### Enforcement paths — three kinds + +Each scheduled deletion names one enforcement path. **Grep over source code is not an enforcement path** — see §"Grep is not an enforcement path" below. + +1. **Structural lens (preferred).** The substrate already carries the fact; a lens walks user DAGs and reports instances. Example: `ArrowBody::Pending` is a substrate variant; a lens walks `d.nodes` and fires on any `Pending` body reachable from user-range roots. Fits the existing `lens_unused_parameters.dag` / `lens_provenance.dag` shape. Writable today. + +2. **Needs substrate amendment (DB-NN).** The substrate does not yet expose the fact the lens would need. A design blocker proposes the amendment; the lens becomes writable after the DB lands. The scheduled-deletion row carries a `Needs DB-NN` enforcement marker until the DB lands, then flips to a live lens path. + +3. **Compiler-source ratchet (temporary).** For scaffolds that live in the hand-written Rust compiler and can't be lensed until `compiler.dag` self-hosts, a narrow source-level ratchet scoped to `src/v3/compiler/` is acceptable as temporary enforcement. Dissolves automatically when `compiler.dag` self-hosts and the same lens can walk the compiler's own DAG. The row explicitly marks this as temporary. + +### Table + +| Scaffold | Dissolution trigger | Upstream blocker | Enforcement | +|---|---|---|---| +| `ArrowBody::Pending` | M3 ratchet | Every realization arrow bound to `ExternalRealization` | **Lens** — writable now; walk `d.nodes` for Arrow declarations with `body = Pending` reachable from user-range roots | +| `ArrowBody::Unparsed` | M2+ parser surface | Match / pipe / lambda / block-body parsing | **Lens** — writable now; same pattern | +| `ValueBody::Unparsed` | M2+ parser surface | Record / map / list literal parsing | **Lens** — writable now; walk `data` declarations | +| `TransformTarget::Operator` | M2+ parser surface | Operator desugar into algebra-field calls | **Lens** — writable now; walk Transform targets | +| User-range `ResolvedByName` AtomPayload (DB-17 new variant — post-landing, any user-range reference produced via name fallback rather than structural walk) | M2 module scoping | Cross-module structural resolution | **Needs DB-17** (reference-resolution provenance) — once DB-17 lands, lens walks `d.nodes` AtomPayloads for `ResolvedByName` reachable from user-range roots | +| Compiler-internal `declaration_by_name` call sites (bootstrap `substrate_markers` initialization in `dag.rs:1616+`, pipeline-authority wiring in `bootstrap.rs`/`pipeline_authority.rs`, emitter algebra lookups in `emit_go.rs`/`emit_python.rs`) | Self-hosting (most cases) OR specific per-site substrate amendments (e.g., substrate_markers becoming typed edges) | Depends on class — self-hosting for emitter/pipeline sites, specific substrate amendment for marker caches | **Compiler-source ratchet** (temporary, dissolves at self-hosting for most sites) — these are compiler-internal caches/wiring, NOT user-range resolution fallbacks; **DB-17 does not cover them** | +| `Node.name` field (v3 substrate) | `authored_name_at` cross-module span fix + 15 direct reads migrated | Cross-module span resolution via DeclarationId | **Compiler-source ratchet** (temporary, dissolves at self-hosting) | +| `encoding_meet` / `encoding_join` (Rust fns) | Track 8 Phase 2 (user-defined generic emission) | User-defined generic emission for `Lattice` instance | **Compiler-source ratchet** (temporary; becomes lens-able when compiler.dag self-hosts and emission-generated code replaces these hand-written fns) | + +### Notes on specific rows + +- **`declaration_by_name` is a helper name, not a single debt class.** The function at `dag.rs:1459` has 83 call sites that split into distinct classes with separate dissolution paths. [DB-17 (reference-resolution provenance)](../../docs/design-reference-resolution-provenance.md) narrows its scope to **only the user-range AtomPayload fallback class** (lowering produces `ResolvedByName(id)` when a structural walk falls back to name lookup). DB-17's lens walks user-range AtomPayloads; compiler-internal call sites (bootstrap substrate_markers in `dag.rs`, pipeline authority wiring, emitter algebra lookups in `emit_go.rs`/`emit_python.rs`) are a separate compiler-source class that dissolves at self-hosting (or via per-site substrate amendments — e.g., substrate_markers becoming typed edges rather than name-keyed caches). Keying the scheduled deletion to the helper name conflates these. +- **`Node.name` cluster**: 15 direct reads audited in the Node-to-std migration project; each has a replacement via structural edge (declaration lookup with structural path). Compiler-source ratchet suffices until self-hosting because the enforcement surface is one directory (`src/v3/compiler/`) and audit cadence catches drift. +- **`keyword_to_name` (recon outcome 2026-04-17, no row added):** the bare `keyword_to_name` was renamed to `tok_keyword_to_name` during v2 Phase 0 parser restructure — see `src/v2/parser-design.md:403-408`. The new name still carries the scaffold (parser-side keyword-name logic that duplicates facts from the tokenizer's `SyntaxSpec`), but it lives in `src/v2/02_parse.dag:455` and `src/v2/stage0/src/v2_compiler_parse.rs:1321` — **v2 code**, not v3. Grep confirms zero equivalents in `src/v3/`. V2 is the reference-implementation / test oracle per `src/v3/ROADMAP.md` §"Sketch vs Oracle framing"; v2 scaffolds dissolve when v3 supersedes v2 entirely, not individually. The v3 Scheduled Deletions list tracks v3-scope scaffolds only. + +### Grep is not an enforcement path + +Per the compiler-as-dependency-analyzer framing: grep over source text cannot distinguish a real user-range violation from a comment, test fixture, bootstrap path, alias, helper indirection, or trait-dispatched call. It matches strings; the compiler analyzes a graph. Using grep to enforce "the system should be structural" uses a heuristic to enforce the ban on heuristics — the discipline defeats itself on the first move. + +Every time a grep gate is proposed over source code, the correct question is: **what substrate fact would make this a lens?** That fact might need a DB; if so, the grep is a signal for the DB, not a substitute for it. + +**Narrow exception:** the banked-dissolutions ratchet in `docs/post-l15-phase-plan.md` operates on *documentation text* (lane docs can't restate DB-rejected shapes), not on system behavior. Docs don't have a resolved DAG; they're text. Grep over docs for design-consistency is legitimate. System-level scaffolds get structural enforcement. + +### How the scheduled-deletions discipline works + +1. **Adding a scaffold.** The PR that introduces a scaffold opens a row here with: scaffold name (file:line or type path), dissolution trigger, upstream blocker, enforcement path (one of the three kinds above). +2. **Enforcement-path classification happens in the same PR.** Structural lens → write the lens or file `lens_TBD` naming the fact to query. Needs DB → file the DB design doc (or reference an existing one). Compiler-source ratchet → explicit; dissolves at self-hosting. +3. **Dissolution.** When the trigger fires, a PR deletes the scaffold AND removes the row. No lingering row after deletion; audit-traceable via git history. +4. **Reviewer gate.** A PR that introduces a scaffold without a row here — or with an enforcement path classified as "grep source code" — is blocked. + ## What NOT to build yet - **Any fourth per-language emit file** (e.g., `emit_verilog.rs`,