Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
170 changes: 170 additions & 0 deletions docs/design-reference-resolution-provenance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
> Part of: [post-l15-phase-plan.md](./post-l15-phase-plan.md) | Unblocks: Scheduled deletion of `declaration_by_name` + NameKeyedReference bridges | Consumes: compiler-as-dependency-analyzer framing

# Design DB-17 — Reference-resolution provenance

**Design blocker:** DB-17 (substrate amendment that records HOW a resolved identifier was resolved — structural walk vs name-lookup fallback)
**Consumer:** `lens_structural_resolution.dag` (joint enforcement for `declaration_by_name` and NameKeyedReference bridges)
**Status:** Proposed. Awaiting review.
**Origin:** scheduled-deletions reclassification pass (2026-04-17). Every scaffold in the "Needs substrate amendment" column needed a concrete DB; `declaration_by_name` + NameKeyedReference bridges cluster as one class, unlocked by this DB.

---

## Problem

Today, when lowering resolves an identifier, the resulting `AtomPayload::ResolvedIdentifier(DeclarationId)` carries the *result* of resolution (the `DeclarationId`) but not the *provenance* — the path by which resolution succeeded. Two very different code paths produce the same substrate shape:

1. **Structural walk** — lowering follows parent-chain / scope-chain edges from the reference site to a declaration reachable through structural parentage. This is the authoritative path: the reference is grounded in the compiler's dependency DAG.
2. **Name fallback** — lowering calls `declaration_by_name(name)` (or an analogous helper that maps a `String` to a `DeclarationId` via a name table). This works today for std/bootstrap lookups where cross-module structural resolution isn't yet wired, but it's a bridge: the reference reaches a declaration by **string identity**, not by structural edge.

Post-resolution, both produce `AtomPayload::ResolvedIdentifier(id)`. A lens walking `d.nodes` cannot distinguish them. Any bridge that fell back through `declaration_by_name` becomes invisible.

This blocks the scheduled deletion of `declaration_by_name`: to dissolve it, we need to enumerate its uses, audit that every legitimate use has a structural-walk alternative, and enforce that no new name-fallback resolutions enter user-range code. Without provenance in the substrate, "enumerate uses" collapses to grep over compiler source — which is exactly the heuristic-over-source failure mode the dependency-analyzer framing rules out.

More broadly: the **dependency graph is incomplete until every reference is grounded by a structural edge**. Name-keyed resolution is a bridge that bypasses the edge. The compiler-as-dependency-analyzer thesis requires that this bridge be either visible or absent.

---

## Design

### Split `AtomPayload::ResolvedIdentifier` into two variants

Today (substrate schema, simplified):

```rust
pub enum AtomPayload {
UnresolvedIdentifier(String),
ResolvedIdentifier(DeclarationId),
TypeParam(String),
Literal(LiteralBits),
// ...
}
```

After DB-17:

```rust
pub enum AtomPayload {
UnresolvedIdentifier(String),
ResolvedByStructure(DeclarationId), // NEW — resolved via structural walk
ResolvedByName(DeclarationId), // NEW — resolved via name fallback
TypeParam(String),
Literal(LiteralBits),
// ...
}
```

`ResolvedIdentifier` is removed; every call site that produced it picks which variant applies. Lowering that walks a structural edge produces `ResolvedByStructure`; lowering that calls `declaration_by_name` (or any name-keyed helper) produces `ResolvedByName`.

### Consumer migration

Every consumer that currently matches on `ResolvedIdentifier(id)` updates to match both variants. For consumers that don't care about provenance, this is mechanical:

```rust
// Before
AtomPayload::ResolvedIdentifier(id) => use(id),

// After
AtomPayload::ResolvedByStructure(id) | AtomPayload::ResolvedByName(id) => use(id),
```

A small helper (`resolved_id(payload) -> Option<DeclarationId>`) can reduce boilerplate for consumers that uniformly ignore provenance. But the default is explicit pattern-matching — the provenance fact is visible at every read site, which is the whole point.

### Lens enforcement unlocks

Once DB-17 lands, `lens_structural_resolution.dag` gets a second variant:

```dag
type StructuralResolutionViolation
= UnresolvedArrowBody { arrow: DeclarationRef }
| NameKeyedReference { site: DeclarationRef, resolved_to: DeclarationRef }

// A NameKeyedReference violation fires whenever ResolvedByName is reachable
// from a user-range root. Bootstrap/std ranges are exempt until M2 module
// scoping lands (at which point all resolutions are structural).
```

The lens walks `d.nodes`, traverses Atom payloads, and reports any `ResolvedByName` found in user-range code. **No grep, no allowlist, no string heuristic** — the substrate fact carries the information the lens queries.

### Dissolution of `ResolvedByName`

`ResolvedByName` is itself a scheduled-deletion row in the ROADMAP. Dissolution trigger: M2 module scoping lands. At that point every reference resolves structurally; the variant becomes unreachable; a final PR removes it from the substrate.

Between DB-17 landing and M2 module scoping landing, `ResolvedByName` is the explicit bridge — visible, countable, lens-enforced. No silent drift.

---

## Rejected alternatives

- **Add `provenance: ResolutionProvenance` metadata to a single `ResolvedIdentifier` variant.**
```rust
ResolvedIdentifier { decl: DeclarationId, provenance: ResolutionProvenance }
```
Rejected because the fact belongs at the variant level, not as a metadata field. The substrate's coproduct axis captures "distinct kinds of reference"; folding provenance inside loses structural visibility. Also: consumers that care about provenance must unwrap the metadata; consumers that don't care read `decl` directly, which silently ignores provenance. The variant split makes every read site acknowledge provenance explicitly.

- **Side table: map of `(Port, DeclarationId) → ResolutionProvenance`.**
Parallel authority (violates D-2). The substrate's Atom is the authority for resolution; a side table forks the fact. Rejected.

- **Grep over compiler source for `declaration_by_name` call sites.**
Heuristic over source text; can't distinguish sanctioned bootstrap path from user-range bridge; falls to renames and helper indirection. The exact failure mode the dependency-analyzer framing rules out. Rejected.

- **Leave `ResolvedIdentifier` unified; rely on audit cadence.**
"Audit every few PRs" has the same drift problem as grep — invisible new uses land in the interval. Substrate fact is the only non-heuristic signal. Rejected.

- **Add provenance at the `Port` level instead of `AtomPayload`.**
Port carries resolved type info, not reference provenance; the fact lives where the resolution happens (the Atom identifier), not where the type lives (the Port). Rejected.

---

## Implementation scope

1. **Substrate change.** `src/v3/compiler/src/dag.rs` — split `AtomPayload::ResolvedIdentifier` into `ResolvedByStructure` and `ResolvedByName`. Delete the old variant.
2. **Lowering.** `src/v3/compiler/src/lower.rs` — every call site that currently produces `ResolvedIdentifier` updates. Structural-walk code paths produce `ResolvedByStructure`; `declaration_by_name` and analogous helpers produce `ResolvedByName`. This is mechanical per-call-site.
3. **Infer.** `src/v3/compiler/src/infer.rs` — match arms on `ResolvedIdentifier` update to handle both variants. A helper `resolved_id(payload) -> Option<DeclarationId>` reduces repetition where provenance is uniformly ignored, but match sites are explicit.
4. **Reflection.** `src/v3/std/substrate.dag` — the reflected `AtomPayload` type gains both variants. Existing lenses that match on `ResolvedIdentifier` update.
5. **Lens.** Write `lens_structural_resolution.dag` as the DB-17 consumer. Initial variants: `UnresolvedArrowBody`, `NameKeyedReference`. Acceptance test fires on any user-range DAG with `ResolvedByName` reachable.
6. **Scheduled-deletions update.** Flip the `declaration_by_name` + NameKeyedReference rows from "Needs DB-17" to the live lens path when the lens lands.

Size: **M** for DB-17 substrate + lowering + consumer migration; **S** additional for the lens itself. Together roughly one lane-stage of work.

---

## Open questions

1. **Bootstrap range vs user range — how is the boundary represented?** The lens needs to exempt name-keyed resolutions inside std/bootstrap files while firing on user-range references. Today `user_start` / declaration-id comparisons serve this purpose in Rust-side code, but the lens needs a substrate-visible fact. Option: a `Declaration.source: DeclarationSource` edge or similar that distinguishes bootstrap/std/user ranges. This might be a sibling DB (DB-18?) or part of DB-17's scope. Current proposal: separate DB if the fact isn't already visible; otherwise cite the existing substrate edge.

2. **Does DB-17 need to handle `ResolvedIdentifier` inside type declarations, or just in Behavior-emitted references?** The intuition says "all resolution sites." The implementation scope for "all sites" is larger. Recommendation: handle all, since half-coverage means the lens has blind spots.

3. **Does the split affect `authored_name_at` or related span-based APIs?** Those resolve names to spans, not to declarations directly. Likely orthogonal — `authored_name_at` is already name-based by design (it's about source position). DB-17's scope is resolved references, not span queries. Confirm at implementation time.

---

## Acceptance

- [ ] `AtomPayload::ResolvedIdentifier` removed from `dag.rs`; `ResolvedByStructure` and `ResolvedByName` added
- [ ] Every lowering call site updated to produce the appropriate variant
- [ ] Every consumer (infer, lenses, emit) updated to handle both variants
- [ ] `src/v3/std/substrate.dag` reflects the new shape
- [ ] `lens_structural_resolution.dag` written with at least `NameKeyedReference` variant; acceptance test fires on a fixture that triggers `declaration_by_name`-style fallback in user-range code
- [ ] Scheduled-deletions table entry for `declaration_by_name` flipped from "Needs DB-17" to the live lens path
- [ ] Existing bootstrap tests green; the split should be semantically invisible to correct consumers
- [ ] A regression test: a fixture that deliberately uses name fallback in user-range code produces a `NameKeyedReference` diagnostic via the lens

---

## Associations

- **Compiler-as-dependency-analyzer thesis** (tonight's framing) — DB-17 is the substrate consequence: every reference must be structural; name-fallback bridges are explicitly marked when they exist.
- **Scheduled-deletions ROADMAP section** (same PR as this DB) — DB-17 unblocks the `declaration_by_name` and NameKeyedReference rows.
- **`lens_structural_resolution.dag`** (separate implementation work; other session) — DB-17 is a prerequisite for the second lens variant (`NameKeyedReference`).
- **M2 module scoping** — dissolves `ResolvedByName` entirely; DB-17's new variant is itself scheduled for deletion.
- **`src/v3/compiler/src/dag.rs::AtomPayload`** — the substrate type being split.
- **`src/v3/compiler/src/lower.rs::declaration_by_name`** — the bridge DB-17 makes structurally visible.
- **`src/v3/std/substrate.dag`** — reflected substrate that lenses walk; gains the two new variants.

---

## Why this is worth a DB

Without DB-17, `declaration_by_name`'s scheduled deletion has **no structural enforcement path**. The scaffold persists because: (a) grep is heuristic and can't tell good fallback from drift, (b) audit cadence misses interval drift, (c) the dependency DAG is incomplete until every reference is structural. DB-17 makes the bridge visible in the substrate so the lens can walk it. Once visible, the scaffold counts down to zero under structural enforcement — not under discipline, not under audit.

This is the compiler-as-dependency-analyzer thesis cashing out: **every named scaffold gets a structural enforcement path, or the substrate amendment that unlocks one is filed.**
50 changes: 50 additions & 0 deletions src/v3/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,56 @@ Sub-stage status (as of 2026-04-17):

GitHub issues for this kind of tracking are **closed with a pointer here.** Issues exist for external coordination (user-facing bug reports, security advisories); internal deferrals do not live in issues.

## Scheduled deletions — scaffolds with named dissolution triggers

**Discipline:** every scaffold in the live substrate lives here with an explicit dissolution trigger, the upstream work it's blocked on, and its enforcement path. When the trigger fires, a PR deletes the scaffold AND removes the row. Unscheduled scaffolds are violations of the scaffold-boundary invariant.

**Relationship to "Active deferrals":** deferrals name work that is in flight; scheduled deletions name artifacts that will disappear. A deferral may REFERENCE a scheduled deletion (e.g., "this sub-stage dissolves `ArrowBody::Pending` per the scheduled-deletions row"), and a scheduled deletion may reference a design blocker (DB-NN) that unlocks structural enforcement.

### Enforcement paths — three kinds

Each scheduled deletion names one enforcement path. **Grep over source code is not an enforcement path** — see §"Grep is not an enforcement path" below.

1. **Structural lens (preferred).** The substrate already carries the fact; a lens walks user DAGs and reports instances. Example: `ArrowBody::Pending` is a substrate variant; a lens walks `d.nodes` and fires on any `Pending` body reachable from user-range roots. Fits the existing `lens_unused_parameters.dag` / `lens_provenance.dag` shape. Writable today.

2. **Needs substrate amendment (DB-NN).** The substrate does not yet expose the fact the lens would need. A design blocker proposes the amendment; the lens becomes writable after the DB lands. The scheduled-deletion row carries a `Needs DB-NN` enforcement marker until the DB lands, then flips to a live lens path.

3. **Compiler-source ratchet (temporary).** For scaffolds that live in the hand-written Rust compiler and can't be lensed until `compiler.dag` self-hosts, a narrow source-level ratchet scoped to `src/v3/compiler/` is acceptable as temporary enforcement. Dissolves automatically when `compiler.dag` self-hosts and the same lens can walk the compiler's own DAG. The row explicitly marks this as temporary.

### Table

| Scaffold | Dissolution trigger | Upstream blocker | Enforcement |
|---|---|---|---|
| `ArrowBody::Pending` | M3 ratchet | Every realization arrow bound to `ExternalRealization` | **Lens** — writable now; walk `d.nodes` for Arrow declarations with `body = Pending` reachable from user-range roots |
| `ArrowBody::Unparsed` | M2+ parser surface | Match / pipe / lambda / block-body parsing | **Lens** — writable now; same pattern |
| `ValueBody::Unparsed` | M2+ parser surface | Record / map / list literal parsing | **Lens** — writable now; walk `data` declarations |
| `TransformTarget::Operator` | M2+ parser surface | Operator desugar into algebra-field calls | **Lens** — writable now; walk Transform targets |
| `declaration_by_name` (Rust fn) | M2 module scoping | Cross-module name resolution | **Needs DB-17** (reference-resolution provenance) |
| NameKeyedReference bridges (any `ResolvedIdentifier` produced via name fallback) | M2 module scoping + DB-17 | Same as `declaration_by_name` — same class | **Needs DB-17** — dissolves jointly |
| `Node.name` field (v3 substrate) | `authored_name_at` cross-module span fix + 15 direct reads migrated | Cross-module span resolution via DeclarationId | **Compiler-source ratchet** (temporary, dissolves at self-hosting) |
| `encoding_meet` / `encoding_join` (Rust fns) | Track 8 Phase 2 (user-defined generic emission) | User-defined generic emission for `Lattice<Encoding>` instance | **Compiler-source ratchet** (temporary; becomes lens-able when compiler.dag self-hosts and emission-generated code replaces these hand-written fns) |

### Notes on specific rows

- **`declaration_by_name` and NameKeyedReference bridges** are the same underlying class: "reference resolved by name fallback rather than by structural edge." [DB-17 (reference-resolution provenance)](../../docs/design-reference-resolution-provenance.md) makes this a single structural fact in the substrate; landing DB-17 unlocks joint dissolution via one lens.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKING: This collapses two different debts into one: DB-17 would surface name-fallback AtomPayload edges, but it would not cover the other live declaration_by_name call sites (bootstrap caches and downstream name recovery), so "joint dissolution via one lens" misstates the single-authority/semantic-authority fix.

- **`Node.name` cluster**: 15 direct reads audited in the Node-to-std migration project; each has a replacement via structural edge (declaration lookup with structural path). Compiler-source ratchet suffices until self-hosting because the enforcement surface is one directory (`src/v3/compiler/`) and audit cadence catches drift.
- **`keyword_to_name` (recon outcome 2026-04-17, no row added):** the bare `keyword_to_name` was renamed to `tok_keyword_to_name` during v2 Phase 0 parser restructure — see `src/v2/parser-design.md:403-408`. The new name still carries the scaffold (parser-side keyword-name logic that duplicates facts from the tokenizer's `SyntaxSpec`), but it lives in `src/v2/02_parse.dag:455` and `src/v2/stage0/src/v2_compiler_parse.rs:1321` — **v2 code**, not v3. Grep confirms zero equivalents in `src/v3/`. V2 is the reference-implementation / test oracle per `src/v3/ROADMAP.md` §"Sketch vs Oracle framing"; v2 scaffolds dissolve when v3 supersedes v2 entirely, not individually. The v3 Scheduled Deletions list tracks v3-scope scaffolds only.

### Grep is not an enforcement path

Per the compiler-as-dependency-analyzer framing: grep over source text cannot distinguish a real user-range violation from a comment, test fixture, bootstrap path, alias, helper indirection, or trait-dispatched call. It matches strings; the compiler analyzes a graph. Using grep to enforce "the system should be structural" uses a heuristic to enforce the ban on heuristics — the discipline defeats itself on the first move.

Every time a grep gate is proposed over source code, the correct question is: **what substrate fact would make this a lens?** That fact might need a DB; if so, the grep is a signal for the DB, not a substitute for it.

**Narrow exception:** the banked-dissolutions ratchet in `docs/post-l15-phase-plan.md` operates on *documentation text* (lane docs can't restate DB-rejected shapes), not on system behavior. Docs don't have a resolved DAG; they're text. Grep over docs for design-consistency is legitimate. System-level scaffolds get structural enforcement.

### How the scheduled-deletions discipline works

1. **Adding a scaffold.** The PR that introduces a scaffold opens a row here with: scaffold name (file:line or type path), dissolution trigger, upstream blocker, enforcement path (one of the three kinds above).
2. **Enforcement-path classification happens in the same PR.** Structural lens → write the lens or file `lens_TBD` naming the fact to query. Needs DB → file the DB design doc (or reference an existing one). Compiler-source ratchet → explicit; dissolves at self-hosting.
3. **Dissolution.** When the trigger fires, a PR deletes the scaffold AND removes the row. No lingering row after deletion; audit-traceable via git history.
4. **Reviewer gate.** A PR that introduces a scaffold without a row here — or with an enforcement path classified as "grep source code" — is blocked.

## What NOT to build yet

- **Any fourth per-language emit file** (e.g., `emit_verilog.rs`,
Expand Down
Loading