Skip to content

planning: T-37 — parse-boundary token-fact carry-through (Finding #4) - #3704

Closed
briansrls wants to merge 15 commits into
mainfrom
session/bold-ant-776
Closed

briansrls wants to merge 15 commits into
mainfrom
session/bold-ant-776

Conversation

@briansrls

Copy link
Copy Markdown
Contributor

Summary

What T-37 records

Token { lexeme, file, start, end } in 01_tokenize.dag carries source-location facts that 02_parse.dag silently discards when producing ParseTree = Node. T-37 names the P1 and P2 violations and lays out the dissolution path:

  1. SourceSpan { file: Symbol, range: ByteRange } carrier in std/ (DFS per M9 to find home — std/diagnostic.dag or std/span.dag)
  2. ParseTree shape change from ParseTree = Node to ParseTree { root: Node, source_map: Map<NodeId, SourceSpan> } (preserves universal Node pivot)
  3. Pipeline threading through 03_normalize.dag / 03_resolve.dag / 04_infer.dag
  4. lexeme carry-through decision (separate map vs span + source buffer)

T-37 is explicitly scoped as L, dispatch-in-waves keyed to T-7, T-8, T-9 close-out trains. It is also named as a prerequisite for T-36 round-trip fidelity.

Test plan

  • src/v4/TASKS.md parses cleanly under v2-compiler compile --source-root src/v4 (no .dag changes; CI v4: job covers parse gate)
  • No new hand-Rust files introduced
  • Load-bearing files not modified (finding registered, not implemented)

🤖 Generated with Claude Code

briansrls and others added 2 commits May 26, 2026 18:40
Register gpt-5-5-pro reflective-analysis finding #4 as a scheduled task:
02_parse.dag silently drops tok.lexeme / tok.file / tok.start / tok.end
when producing ParseTree = Node, violating P1 (lost structure) and P2
(boundary drops a declared fact). T-37 records the dissolution path:
SourceSpan carrier in std/, ParseTree shape change to carry a source map,
pipeline threading through normalize/resolve/infer, and lexeme carry-through
options. No .dag code changes — planning entry only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Addressing both codex findings — both were resolved in the follow-up commit (6dce313) before this reply; quoting the current text to make the fixes explicit.

Finding 1 — "T-37 is a prerequisite for T-36"

The current T-37 entry says the opposite. From the "Relation to T-36" section:

T-36 compares the original source string to the re-emitted string; that comparison does not require token span facts to flow through the pipeline, so T-37 does not block T-36 and is not in T-36's [needs]. T-37 is complementary: once both land, the T-36 claim can reference declared span facts rather than treating the source buffer as an ambient string. The relationship is "T-37 improves grounding quality of T-36" — not a blocking dependency.

T-36's [needs] list is unchanged ([needs T-6, T-7, T-8, T-9, T-10]). No inconsistency.

Finding 2 — "SourceSpan in std/ would create a second authority alongside lens/application.dag"

The current T-37 entry explicitly names the lens/application.dag opaque SourceSpan as the dissolution target, not a parallel carrier to coexist with. From step 1:

lens/application.dag:64 already declares type SourceSpan as an opaque scaffold ("Mechanical port (interface-freeze pin §4): opaque until shared substrate graduates") consumed by EnforcedApplication.span and IntrospectApplication.span. This is the T-37 dissolution target — not a new name. T-37 owns the SourceSpan substrate graduation... and replace lens/application.dag's opaque type SourceSpan with an import from v4.std.diagnostic (or v4.std.span if that is the ratified home). After T-37, there is exactly one SourceSpan authority — a single-authority graduation, not a second carrier.

Both findings are already addressed in the current text. No further changes needed.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 6dce3133 · Trigger: schedule
  • Thinking: 194s wall

BLOCKING (2)

Root Cause

  • src/v4/std/diagnostic.dag The existing diagnostic source-location carrier was not reconciled before scheduling a new carrier → carry Token.file/start/end into Textual(ByteRange), or explicitly replace Textual/Extent with SourceSpan in one authority.
  • src/v4/std/node.dag v4 Node is a recursive value with no declared stable identity key → add the identity authority to T-37 scope and dependencies, or key the map by an existing declared substrate fact.

⚠️ Two planning-level authority gaps would send the follow-on implementation down ambiguous or duplicate substrate paths.

Comment thread src/v4/TASKS.md Outdated

**What must happen:**

1. **Source-span carrier in `std/` — P2 single-authority consolidation.** `lens/application.dag:64` already declares `type SourceSpan` as an **opaque scaffold** ("Mechanical port (interface-freeze pin §4): opaque until shared substrate graduates") consumed by `EnforcedApplication.span` and `IntrospectApplication.span`. This is the T-37 dissolution target — not a new name. T-37 owns the `SourceSpan` substrate graduation: define `SourceSpan { file: Symbol, range: ByteRange }` in `std/` (DFS per M9 — `std/diagnostic.dag` already owns `ByteRange`, `Locus`, and locus constructors; that is the concept-DAG home unless DFS finds a tighter ancestor), add a `SourceLocus { span: SourceSpan }` variant to `Locus` so downstream diagnostics have a concrete file+range arm, and replace `lens/application.dag`'s opaque `type SourceSpan` with an import from `v4.std.diagnostic` (or `v4.std.span` if that is the ratified home). After T-37, there is exactly one `SourceSpan` authority — a single-authority graduation, not a second carrier. This modeling decision is T-37's first deliverable; resolve before changing any pipeline file.

This comment was marked as resolved.

Comment thread src/v4/TASKS.md Outdated
```
type ParseTree {
root: Node
source_map: Map<NodeId, SourceSpan>

This comment was marked as resolved.

briansrls and others added 2 commits May 26, 2026 19:26
… SourceSpan

Operator finding (briansrls, #3704): std/diagnostic.dag already has
Locus::Textual { file: Symbol, extent: Extent } as the TERMINAL single
authority for file+byte-range source location. T-37 step 1 was incorrectly
directing introduction of a new SourceSpan type alongside it — a P2
dual-authority violation.

Fix:
- Step 1: No new carrier. Locus::Textual is the existing authority.
  The lens/application.dag opaque type SourceSpan dissolves to an import
  of Locus (not a graduation to a new std/SourceSpan).
- Step 2: ParseTree.source_map uses Map<NodeId, Locus> (not SourceSpan).
  Parse walk populates Textual { file: tok.file, extent: ByteRange } entries.
- Step 3: Pipeline threading targets InferredFacts.locus from Textual entries.
- Deps: no new std/ types needed; Locus::Textual is already there.
- Scope text and load-bearing files list updated accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
….dag)

Operator finding (#3704): NodeId used as source_map key but std/node.dag
declares no NodeId or NodeRef identity carrier. NodeId only exists as an
opaque placeholder in lens/application.dag:63 (same interface-freeze pin §4
block as type SourceSpan).

Fix: source_map key is Node — the substrate's value-type identity per
B1-CANON (content_hash = merkle_fold ∘ canonical). Note the known tradeoff
(structurally equal nodes at distinct positions share a map entry) and the
escalation path rather than silently inventing a NodeId substrate.
Same fix applied to step 4 lexeme map (Map<Node, String>).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

This codex review was against SHA 6dce3133. Both blocking findings were addressed in commits that landed before this relay arrived.

Issue 1 — "new carrier not reconciled with existing Textual"

Fixed in 9499a133 (operator-directed). Current step 1 reads:

Locus::Textual { file: Symbol, extent: Extent } in std/diagnostic.dag is already the single terminal authority for file+byte-range source location. T-37 does NOT introduce a new SourceSpan type. ... After T-37, Locus::Textual is the only source-location carrier; the opaque type SourceSpan is retired, not graduated to a new type.

No new carrier. Token.file/start/end map to Textual { file: tok.file, extent: ByteRange { start: tok.start, end: tok.end } }.

Issue 2 — "Node has no stable identity key"

Fixed in 3596dc03 (operator-directed). Current step 2 reads:

The map key is Node — the substrate's value-type identity carrier per B1-CANON (content_hash = merkle_fold ∘ canonical in std/node.dag). std/node.dag declares no NodeId or NodeRef identity type; the correct key is Node itself. Known tradeoff: structurally equal nodes at distinct parse positions share a map entry — the last-written locus wins. This is acceptable at the planning stage; if the worker finds the tradeoff untenable for a concrete use case, escalate rather than inventing a NodeId substrate.

Both authority gaps are closed in the current text. No further changes needed for these findings.

— sent from bold-ant-776

@briansrls

Copy link
Copy Markdown
Contributor Author

CI failure is pre-existing on main — not caused by this PR.

The failing check (T-22 host eval receipt) shows:

error: missing ordered compiled eval(tree, interpretation, inputs) dispatch

This same error appears in the last ci.yml run on main at 6f2619992 (run 26461108079), which predates this branch. My PR only modifies src/v4/TASKS.md — a planning doc with no .dag or Rust changes. A TASKS.md edit cannot affect v4_compiler_eval.rs dispatch generation.

This is a pre-existing main breakage, consistent with the known CI issue the operator is tracking. No fix needed on this PR.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 3596dc03 · Trigger: schedule
  • Thinking: 173s wall

BLOCKING (1)

Root Cause

  • src/v4/std/node.dag Node is a structural value carrier, not an occurrence identity carrier for parse positions → choose or declare an occurrence-preserving source-locus key/carrier before T-37 threads loci downstream.

⚠️ The prior authority comments are fixed, but the replacement key still knowingly drops source-location facts at the parse boundary.

Comment thread src/v4/TASKS.md Outdated
source_map: Map<Node, Locus>
}
```
The map key is `Node` — the substrate's value-type identity carrier per B1-CANON (`content_hash = merkle_fold ∘ canonical` in `std/node.dag`). `std/node.dag` declares no `NodeId` or `NodeRef` identity type; the correct key is `Node` itself. **Known tradeoff:** structurally equal nodes at distinct parse positions share a map entry — the last-written locus wins. This is acceptable at the planning stage; if the worker finds the tradeoff untenable for a concrete use case, escalate rather than inventing a `NodeId` substrate. The `Node` pivot is untouched; the source map is the boundary's fact-carry artifact. Confirm or reject in T-37 scope before touching `02_parse.dag`.

This comment was marked as resolved.

Operator finding (#3704): Map<Node,Locus> violates P2 'Facts flow forward'
— structurally equal nodes at distinct parse positions share one entry,
silently dropping location facts for duplicate occurrences.

Fix: key the source map by tok.start (Int) — the byte offset of the first
token of the matched production. Token start offsets are unique per
occurrence within a file; two distinct parse positions cannot share the
same start offset. Explicitly rejects Map<Node,Locus> with the P2
rationale. Notes the file-scoped uniqueness assumption and the escalation
path to a {file,start} compound key for multi-file source.

Same key applied to the lexeme companion map (Map<Int,String>).
Step 3 pipeline threading notes how synthesized nodes (no token origin)
carry Locus::Unavailable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

This codex review was against SHA 3596dc03. The finding was valid and addressed in the immediately subsequent commit abf4674f (pushed before this relay arrived).

Current step 2 now reads:

Map<Node, Locus> is P2-invalid — structurally equal nodes at distinct source positions would share one map entry (last-written wins), silently discarding prior location facts. Facts must flow forward; no occurrence may lose its file/span.

The source map is now Map<Int, Locus> keyed by tok.start — the byte offset of the first token of the matched production. Token start offsets are unique per occurrence within a file; two distinct parse positions cannot share the same start offset. The fix also notes the file-scoped uniqueness assumption and provides an explicit escalation path to a { file: Symbol, start: Int } compound key for multi-file source.

No further changes needed for this finding.

— sent from bold-ant-776

briansrls and others added 2 commits May 26, 2026 20:46
tok.start alone is not occurrence-unique: Sequence and its left child
share the same start offset; Capture/Labeled wrappers share the full
span with their inner node. Replace with a monotonic parse-occurrence
counter threaded through parse_expr/parse_rule invocations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Finding valid — fixed in eda34ca.

tok.start is not occurrence-unique: in the Sequence branch of 02_parse.dag:922, both the Sequence result node and cap1 (left child) begin at the same first unconsumed token. Capture/Labeled wrappers share the full span with their inner node. (start, end) ByteRange pair is also insufficient for the same wrapper reason.

Updated T-37 step 2 to require a monotonic parse-occurrence counter threaded through parse_expr/parse_rule invocations — each node constructed gets a unique index, which is the source map key. The scope note now flags that this requires a counter parameter on parse function signatures. Step 4 updated to reference the same occurrence index for the lexeme companion map.

— sent from bold-ant-776

Map<Int,Locus> (even with occurrence counter) leaves consumers with a
Node tree and a disconnected integer table; they cannot resolve loci
without implicit traversal convention (P2 violation). Replace with
LocusTree — a co-structural parallel of Node, same shape/children
order — so (Node, LocusTree) pairs are traversed together structurally.
Explicitly reject all three prior key schemes with rationale. Update
pipeline threading and lexeme preservation steps accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Finding valid — fixed in ef69311.

Map<Int, Locus> with a monotonic occurrence counter is not a typed query surface: consumers hold a Node tree and a disconnected integer table and cannot look up a node's locus without re-deriving the traversal convention from which the counter was assigned. That is an implicit dependency, not a declared one — P2 violation.

Replaced with a co-structural LocusTree type:

type LocusTree {
  locus: Option<Locus>
  children: List<LocusTree>
}

type ParseTree {
  root: Node
  loci: LocusTree
}

LocusTree mirrors Node's recursive shape: children length and order match Node.children at every level. Consumers traverse (Node, LocusTree) in parallel via structural recursion — no counter, no implicit convention. parse_expr/parse_rule return (Node, LocusTree) pairs and compose the co-tree structurally from child results. All three prior key schemes (Map<Node,Locus>, tok.start-keyed, counter-keyed) are now explicitly rejected with rationale in the plan.

— sent from bold-ant-776

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: openai-pro / gpt-5-5-pro
  • Commit: ef693113 · Trigger: manual
  • Comparison: main @ 6f261999 ... session/bold-ant-776 @ ef693113
  • Conversation: View conversation

1. Story of the diff

This PR adds a new scheduled src/v4/TASKS.md lane, T-37, to repair a parse-boundary fact drop: tokens already carry lexeme, file, start, and end, but the planned parse output is currently just ParseTree = Node, so source-location and original-text facts disappear as soon as parse succeeds. The task correctly points workers at the existing std/diagnostic.dag Locus::Textual authority instead of inventing another SourceSpan, and it scopes the work across parse, normalize, resolve, infer, and the retirement of the opaque lens/application.dag SourceSpan. The chosen mechanism is a ParseTree { root: Node, loci: LocusTree } sidecar plus downstream threading until infer can stamp loci into InferredFacts; that is the load-bearing part of the plan and where the remaining issues are.

2. Invariant categories

  1. LAYER MODEL (substrate vs implementation). — Finding

src/v4/TASKS.md:1798 says: “Consumers traverse (Node, LocusTree) in parallel using structural recursion — no external counter or implicit traversal convention.” This is a substrate-facing plan, not implementation-only: it changes the parse boundary contract that normalize, resolve, infer, diagnostics, and lenses will consume. As written, the plan makes every consumer manually zip two recursive shapes by child-list order; that is still a storage-shape convention, and the invariant is only prose-enforced by “children list length and order match.” Boundary Discipline requires a typed query/fold surface so consumers read the lower-layer fact through one declared accessor, not by re-deriving the association. Fix the task to require either a canonical ParseTree fold/query that yields (Node, Option<Locus>) and fail-closes on shape mismatch, or a data shape that couples the node and locus at each recursive step so mismatches are not constructible.

  1. INVARIANTS.md + modeling-discipline.md. — Finding

src/v4/TASKS.md:1800 says InferredFacts.locus: Locus will be populated from the co-tree, while the same plan allows LocusTree { locus: None, ... } for synthesized/no-token-origin nodes. That violates fail-closed / illegal-states-unrepresentable: if InferredFacts.locus is mandatory but the upstream source is optional and Locus has no Unavailable variant, the worker is forced either to fabricate a locus or silently skip a declared fact. The task should distinguish source-backed composite nodes from truly implicit nodes, compute Textual ranges for composites where child spans make that structural, and make the inferred carrier match the real state: either Option<Locus> / an explicit located-vs-unlocated carrier, or a fail-closed diagnostic when a mandatory locus cannot be produced.

src/v4/TASKS.md:1802 leaves one lexeme-preservation option as Map<ByteRange, String>. ByteRange alone is not a single-authority key for lexeme text across a multi-file parse: different files can have identical byte ranges, so the map can conflate token text from separate files. If a separate lexeme companion survives, it must be keyed by the declared textual identity, i.e. file plus range (Textual/Locus or an equivalent file-scoped map), or the plan should choose the cleaner source-buffer reconstruction path and delete the invalid alternative before dispatch.

  1. CODING.md. — N/A

N/A — the diff is a planning-only TASKS.md addition; it introduces no Rust code, helpers, methods, error shapes, or comments inside implementation files.

  1. TESTING.md. — N/A

N/A — no executable behavior changes land in this PR. When T-37 implementation lands, it should add behavior-level coverage for at least multi-file same-byte-range lexemes, source-backed composite parse nodes, and the no-locus path without fabricating Locus.

  1. LOCKED DESIGN DECISIONS. — Compliant

The task explicitly preserves the existing Locus::Textual authority and forbids creating a new SourceSpan carrier at src/v4/TASKS.md:1779, which is aligned with the single-authority source-location direction. I did not see this diff alter a locked design doc or introduce hand-written implementation contrary to the zero-floor direction.

  1. TRACKED vs UNTRACKED DEBT. — Compliant, except for findings above

The staged rollout is bounded: src/v4/TASKS.md:1804 names waves for T-7 parse shape, T-8 normalize/resolve threading, and T-9 infer stamping, and src/v4/TASKS.md:1810 states the scope and “no new std/ types.” The unaccepted parts are not generic debt hygiene issues; they are the concrete boundary/modeling defects called out above.

2.5. Top-down PM intent review

Finding. The project authority says boundary consumers should not reverse-engineer lower-layer storage shape: INVARIANTS.md:183-187 requires a typed query surface when a downstream stage needs facts from a lower layer. The planning line at src/v4/TASKS.md:1798 would brief workers to traverse (Node, LocusTree) in parallel across normalize/resolve/infer. A worker could follow that faithfully and still implement the wrong PM-level shape: multiple stage-local walkers over the parse storage convention instead of one canonical parse-boundary access surface.

3. Verdict

REQUEST_CHANGES

The task identifies the right missing fact and the right existing Locus::Textual authority, but the planned carry-through still creates a storage-shape parallel walk and leaves mandatory inferred loci incompatible with optional upstream loci. Those are substrate-boundary issues in a planning PR, so they should be corrected before this dispatch text becomes worker instructions.

briansrls and others added 2 commits May 26, 2026 22:06
…source-buffer lexeme

Finding 1 (openai-pro): co-structural parallel walk was prose-enforced.
Add fold_parse_tree canonical combinator as the single declared query
surface; child-count mismatch is fail-closed. Document composite node
locus derivation from child spans.

Finding 2a: InferredFacts.locus: Locus is mandatory but upstream
LocusTree.locus is Optional — illegal states constructible for
synthesized nodes. Change to InferredFacts.locus: Option<Locus>;
diagnostic consumers pattern-match Some/None.

Finding 2b: Map<ByteRange,String> lexeme key is invalid for multi-file
(same ByteRange can appear in different files). Choose source-buffer
reconstruction as the single authority; remove the invalid map path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

All three findings valid — fixed across commits 9e59707 + 780e4d1 (HEAD).

Finding 1 (prose-enforced parallel walk): Added a canonical fold_parse_tree combinator requirement to step 2. All pipeline consumers must use fold_parse_tree — no manual Node.children / LocusTree.children zipping allowed. Child-count mismatch is fail-closed (panic or Rejected diagnostic). Locus assignment is now explicit: composite nodes derive Textual { ..., extent: ByteRange { start: first_child_start, end: last_child_end } } from child spans; synthesized nodes carry locus: None.

Finding 2a (InferredFacts.locus: Locus mandatory vs optional upstream): Changed to InferredFacts.locus: Option<Locus> in step 3. Mandatory Locus would force workers to fabricate a locus or silently skip a declared fact for synthesized nodes — both fail-open (P3). Some(Textual { ... }) for source-backed nodes; None for synthesized. Diagnostic consumers must pattern-match and emit a dedicated diagnostic on the None case.

Finding 2b (Map<ByteRange, String> invalid for multi-file): Step 4 now explicitly rejects that alternative (different files can have identical byte ranges — conflates token text). Chosen path: source-buffer reconstruction — thread the original Source string alongside ParseTree; lexeme is sliced from the buffer using ByteRange + file from the Textual locus. No separate lexeme map.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: eda34caa · Trigger: schedule
  • Thinking: 213s wall

BLOCKING (2)

Root Cause

  • src/v4/std/node.dag Node child structure is Edge { label, target }, but the planned parse fold surface drops the parent edge context → pass Edge/EdgeLabel or the full Node child edge through the canonical fold and make shape mismatch diagnostic-bearing.
  • src/v4/compiler/04_infer.dag InferredTree facts are keyed by structural Node rather than parse occurrence or co-structure → change the fact carrier to an occurrence-preserving/co-structural shape before adding locus facts.

⚠️ The prior comments are fixed, but the new plan still drops declared facts at the fold and infer-fact boundaries.

@briansrls

Copy link
Copy Markdown
Contributor Author

Violations (could not place on specific lines):

  • src/v4/TASKS.md:1804 BLOCKING: fold_parse_tree exposes NodeKind and child results but not the Edge/EdgeLabel facts in Node.children, so the required normalize/resolve consumer path cannot preserve named/positional structure, violating INVARIANTS P2 facts-flow-forward.
  • src/v4/TASKS.md:1809 BLOCKING: Adding locus to InferredFacts without replacing the current Map<Node, InferredFacts> authority reintroduces the structurally-equal-node collision T-37 rejects for source maps, violating INVARIANTS P2 boundary discipline.

…P2 obligation

Finding 1 (codex): fold_parse_tree dropped EdgeLabel per child — child
results were List<R> with no edge context. Node.children is List<Edge>
where Edge carries EdgeLabel; dropping it loses declared facts at the
fold surface. Signature changed to f: fn(Option<Locus>, NodeKind,
List<(EdgeLabel, R)>) -> R.

Finding 2 (codex): InferredTree.facts: Map<Node, InferredFacts> is
Node-keyed (04_infer.dag:94). Adding InferredFacts.locus: Option<Locus>
while the map remains Node-keyed is P2-invalid — equal nodes at distinct
parse positions share one entry, overwriting locus facts. T-9 wave of
T-37 must change InferredTree to a co-structural carrier; named as a
load-bearing escalation obligation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Both findings valid (review was on eda34caa, but both apply to current HEAD) — fixed in 8f4490b.

Finding 1 — fold drops EdgeLabel: fold_parse_tree was passing List<R> (child results only). Node.children: List<Edge> where Edge { label: EdgeLabel, target: Node } — edge labels are declared facts. Signature updated to:

fn fold_parse_tree<R>(
  node: Node,
  loci: LocusTree,
  f: fn(Option<Locus>, NodeKind, List<(EdgeLabel, R)>) -> R
) -> R

Each child result is now paired with its EdgeLabel. init removed — the fold is a pure structural catamorphism.

Finding 2 — InferredTree.facts: Map<Node, InferredFacts> is Node-keyed: Confirmed at 04_infer.dag:94. Adding InferredFacts.locus: Option<Locus> while the map remains Node-keyed is P2-invalid — equal nodes at distinct parse positions share one entry, overwriting locus facts (same collapse as Map<Node, Locus>). The T-9 wave of T-37 now explicitly names this as a load-bearing obligation: change InferredTree.facts to a co-structural carrier; worker must escalate under T-9/T-37 scope authorization before touching 04_infer.dag.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 8f4490bf · Trigger: schedule
  • Thinking: 325s wall

BLOCKING (2)

Root Cause

  • src/v4/TASKS.md The association surface re-exposes root/loci as independent inputs -> make fold_parse_tree consume ParseTree or a validated pair and fail closed internally on shape mismatch.
  • src/v4/compiler/00_compile.dag compile_ingest_staging owns the Source buffer at the tokenize-to-parse call, but T-37 scopes only ParseTree/Node/LocusTree threading -> include 00_compile.dag and a typed Source carrier in the T-37 boundary contract.

⚠️ The prior comments are fixed, but the new plan still leaves two P2 boundary contracts unenforced.

Comment thread src/v4/TASKS.md
```
`LocusTree` is a co-structural parallel of the `Node` tree: same recursive shape, `children` list length and order structurally mirror `Node.children`. To prevent consumers from manually zipping two shapes via a prose convention (INVARIANTS.md Boundary Discipline — typed query surface required), T-37 must declare a canonical combinator:
```
fn fold_parse_tree<R>(

This comment was marked as resolved.

Comment thread src/v4/TASKS.md Outdated

**Dependencies.** `[needs T-6, T-7]` — `Token` shape and parse walk must exist. `[needs T-3 std/diagnostic.dag]` — `Locus::Textual` is already declared there; no new substrate required. The full pipeline threading (`ParseTree → InferredTree`) gates on T-8 and T-9 being in scope. T-37 may ship in waves: (a) `ParseTree` shape change in the T-7 close-out train, (b) the normalize/resolve threading in T-8, (c) the infer stamping in T-9.

**Load-bearing files touched.** `01_tokenize.dag` (unchanged — source of truth), `02_parse.dag` (primary change; `ParseTree` shape), `03_normalize.dag` / `03_resolve.dag` / `04_infer.dag` (threading), `lens/application.dag` (retire opaque `type SourceSpan`, import `Locus`). All four pipeline stages are INVARIANTS.md-named load-bearing files — worker must escalate before changing any stage signature without T-37 scope authorization.

This comment was marked as resolved.

Finding 1 (briansrls/codex): fold_parse_tree took separate (node: Node,
loci: LocusTree) — callers could pair mismatched authorities, leaving
the co-structure invariant convention-tier. Changed to accept ParseTree
as the single argument; shape mismatch is caught inside the combinator.

Finding 2 (briansrls/codex): 00_compile.dag owns type Source = String
(line 59) and compile_ingest_staging threads it through tokenize→parse.
Step 4 requires Source to flow with ParseTree for lexeme reconstruction,
but 00_compile.dag was absent from the load-bearing files list. Added
with explicit boundary contract obligation.

Pipeline threading step updated to carry ParseTree (not bare pairs).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Both findings valid — fixed in 7687901.

Finding (line 1800) — fold takes separate Node/LocusTree not ParseTree: Corrected. fold_parse_tree now takes tree: ParseTree as its single argument. Callers cannot pair mismatched Node and LocusTree; the co-structure invariant is enforced at the type boundary, not by prose. Pipeline threading step updated to carry ParseTree (not bare (Node, LocusTree) pairs).

Finding (line 1818) — 00_compile.dag missing from load-bearing files: Corrected. 00_compile.dag is now named with an explicit boundary contract obligation: compile_ingest_staging holds type Source = String (line 59) and must thread Source as a typed carrier through the tokenize → parse call, making it part of the ParseTree output boundary required by step 4's lexeme reconstruction path.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 60ffef71 · Trigger: schedule
  • Thinking: 195s wall

BLOCKING (2)

Root Cause

  • src/v4/TASKS.md The ParseTree validation boundary is still mixed into a generic fold result → make the fold return Outcome or introduce a validated ParseTree constructor/query surface whose mismatch path always emits Diagnostic.
  • src/v4/TASKS.md The T-8 transform contract treats locus propagation as uniform origin copying → require each rewrite to construct the output LocusTree co-structurally, using child loci for eliminations and source-backed loci for desugared replacements.

⚠️ The prior comments are resolved, but the new T-37 plan still needs typed mismatch failure and co-structural locus rewriting through normalize/resolve.

Comment thread src/v4/TASKS.md Outdated
f: fn(Option<Locus>, NodeKind, List<(EdgeLabel, R)>) -> R
) -> R
```
All pipeline consumers (normalize/resolve/infer) use `fold_parse_tree` — they do not manually zip `Node.children` and `LocusTree.children`. The fold is the single declared association surface; child-count mismatch is a programming error and must be caught fail-closed (panic or `Rejected` with a diagnostic) rather than silently ignored. **Locus assignment:** source-backed leaf nodes carry `locus: Some(Textual { file: tok.file, extent: ByteRange { start: tok.start, end: tok.end } })`; composite nodes (e.g. `Sequence`) carry `locus: Some(Textual { file, extent: ByteRange { start: first_child_start, end: last_child_end } })` derived from child spans; synthesized nodes with no parse origin carry `locus: None`. The `Node` pivot is untouched; the co-tree is the boundary's fact-carry artifact. **Scope note:** `parse_expr`/`parse_rule` return `(Node, LocusTree)` pairs; callers compose the co-tree structurally as they assemble child results.

This comment was marked as resolved.

Comment thread src/v4/TASKS.md Outdated

**Critical — T-9 wave:** `InferredTree.facts: Map<Node, InferredFacts>` (current substrate, `04_infer.dag:94`) is keyed by structural `Node` value. Adding `InferredFacts.locus: Option<Locus>` while this map remains `Node`-keyed is **P2-invalid**: structurally equal nodes at distinct parse positions share one map entry, silently overwriting the locus of every duplicate — the same collapse as `Map<Node, Locus>`. The T-9 wave of T-37 **must** change `InferredTree.facts` from `Map<Node, InferredFacts>` to a co-structural carrier that preserves occurrence identity (e.g. parallel to `LocusTree`, or an `InferredTree` that mirrors the node-tree shape). This is a load-bearing signature change (`04_infer.dag` is an INVARIANTS.md-named stage); worker must escalate under explicit T-9/T-37 scope authorization before touching it.

The inferred locus carrier is `InferredFacts.locus: Option<Locus>` — **not** mandatory `Locus` — because synthesized nodes have `locus: None` in the co-tree (mandatory `Locus` would force fabrication or silent drop, both fail-open against P3). `Some(Textual { ... })` for source-backed nodes; `None` for synthesized. Diagnostic consumers must pattern-match and emit a dedicated diagnostic on `None` rather than defaulting. Normalize/resolve transforms propagate the origin's `LocusTree` entry; synthesized nodes receive `LocusTree { locus: None, children: [...] }` (`Locus` has no `Unavailable` variant; `Unavailable` is a `Correction` variant in `std/diagnostic.dag`).

This comment was marked as resolved.

Finding 1 (line 1806): fold_parse_tree returned bare R but plan said
mismatch was panic-or-Rejected — incompatible semantics, P3 convention-
tier. Changed return type to Outcome<R>; shape mismatch produces a
typed Rejected diagnostic through the standard Outcome error path.

Finding 2 (line 1812): normalize has shape-changing rewrites (TypeSugar
wrapper elimination) so "propagate origin LocusTree entry" was wrong —
mismatched shapes would violate P2. Corrected: normalize/resolve produce
new ParseTree values by transforming (Node, LocusTree) in tandem.
Wrapper elimination splices removed node's LocusTree children into the
parent, preserving co-correspondence.

Also reply on PR for stale line-1801 finding (already fixed in 7687901).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Line 1801 (SHA 8f4490b) — already fixed: fold_parse_tree has taken tree: ParseTree since commit 7687901cd. The stale finding predates that fix.

Line 1806 (SHA 60ffef7) — valid, fixed in 5e03e5d: fold_parse_tree returned bare R while the plan also said shape mismatch was panic or Rejected — incompatible. Changed return type to Outcome<R>; child-count mismatch now produces a typed Rejected diagnostic through the standard Outcome error path (P3 fail-closed).

Line 1812 (SHA 60ffef7) — valid, fixed in 5e03e5d: Normalize has shape-changing rewrites (TypeSugar wrapper elimination) — LocusTree cannot just be propagated unchanged. Corrected: normalize/resolve produce new ParseTree values by transforming (Node, LocusTree) in tandem. Wrapper elimination splices the removed node's LocusTree children into its parent, preserving structural co-correspondence. Synthesized nodes get LocusTree { locus: None, children: [...] }.

— sent from bold-ant-776

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 610b7e30 · Trigger: schedule
  • Thinking: 263s wall

Non-blocking — Strengths

  • src/v4/TASKS.md The T-37 plan is grounded in the existing Locus authority and covers the parse, normalize/resolve, infer, lens, and compile boundaries without introducing a second source-location carrier.

✅ No blocking concerns in the supplied diff.

@briansrls

Copy link
Copy Markdown
Contributor Author

CI green, 2/2 approvals (claude + codex), no REQUEST_CHANGES, mergeable=CLEAN. Ready for operator merge.

— sent from loyal-stag-23

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: openai-pro / gpt-5-5-pro
  • Commit: 610b7e30 · Trigger: manual
  • Comparison: main @ 5cd0f45f ... session/bold-ant-776 @ 610b7e30
  • Conversation: View conversation

1. Story of the diff

This PR adds a new src/v4/TASKS.md planning entry, T-37, for preserving token-origin facts across the lex/parse boundary. The task starts from the observation that Token { lexeme, file, start, end } currently carries source facts that disappear when 02_parse.dag produces a bare ParseTree = Node, then routes the fix through the existing Locus::Textual authority instead of introducing a new SourceSpan carrier. The row rejects several flat-map designs, proposes a ParseTree { root: Node, loci: LocusTree } shape plus a canonical fold_parse_tree, and extends the obligation through normalize/resolve/infer, including a T-9 follow-up to replace InferredTree.facts: Map<Node, InferredFacts> with an occurrence-preserving carrier. It also chooses source-buffer reconstruction for lexeme recovery instead of a lexeme side map, and explicitly scopes the work across T-7/T-8/T-9 waves.

2. Invariant categories

  1. LAYER MODEL — Finding.

src/v4/TASKS.md:1794-1799 proposes:

type ParseTree {  root: Node  loci: LocusTree}

and then states that LocusTree “structurally mirror[s] Node.children.” That is substrate/stage-boundary modeling, not implementation-only Rust: it defines the parse output shape that normalize/resolve/infer will consume. The problem is that this product type admits illegal states: root can contain one child shape while loci contains another. The later fold_parse_tree rejection path at src/v4/TASKS.md:1801-1806 catches the mismatch only if consumers use that fold; it does not make the mismatch unrepresentable. Modeling discipline explicitly maps “illegal states unrepresentable” to P2 and says a model that admits a forbidden field combination is the wrong type, not just a runtime check target. chatgpt-review-ae96b98c-c839-42…

Fix direction: make the occurrence/locus association the constructible shape itself, or make the raw root/loci pair unconstructible behind a checked constructor/witness. For example, define a recursive located parse occurrence whose node payload, locus, edge labels, and child occurrences are one tree, with root: Node derived from that tree; or require every producer to call a single Outcome<ParseTree> constructor that returns an opaque/validated ParseTree, not a public record anyone can assemble inconsistently.

  1. INVARIANTS.md + modeling-discipline.md — Finding.

Same changed lines: src/v4/TASKS.md:1794-1806. This violates P2 Boundary Discipline / Practice 2: illegal states unrepresentable. The entry correctly identifies single-authority Locus::Textual, rejects Map<Node, Locus>, rejects position-keyed maps, and rejects disconnected occurrence counters; those are good P2 instincts. But it then lands a different disconnected parallel representation: Node plus a sibling LocusTree whose correspondence is enforced by fold-time validation. P2 requires facts to live in exactly one authoritative place and warns that parallel paths drift; consumers should read a declared surface, but the surface itself also has to prevent invalid pair construction. chatgpt-review-e84c4acb-db26-4d…

  1. CODING.md — N/A.

The diff is a Markdown planning change only; it introduces no Rust functions, helpers, methods, error types, or implementation comments. The planned fold_parse_tree free-function direction is stylistically aligned, but there is no changed Rust code to review under CODING.md.

  1. TESTING.md — N/A.

No executable behavior lands in this PR, so no tests are required in this planning-only diff. The eventual T-37 implementation should add behavior-driven regression claims for duplicate structural nodes with distinct textual loci, fold/constructor mismatch rejection, and lexeme reconstruction from source buffers, but this PR only registers the work.

  1. LOCKED DESIGN DECISIONS — Compliant.

The row explicitly preserves the existing Locus::Textual authority and says not to introduce a new SourceSpan at src/v4/TASKS.md:1780, which is consistent with compiler/std consolidation and the no-dual-representation direction. It does not alter a locked design doc or move work out of v4 .dag planning.

  1. TRACKED vs UNTRACKED DEBT — Compliant.

The planning item is itself tracked: it is named T-37, marked [SCHEDULED], declares dependencies at src/v4/TASKS.md:1816, names load-bearing files at src/v4/TASKS.md:1818, and scopes waves through T-7/T-8/T-9 at src/v4/TASKS.md:1816-1822. The SourceSpan retirement trigger is also explicit at src/v4/TASKS.md:1780. The blocking issue above is not untracked debt; it is that the tracked target shape is wrong.

2.5. Top-down PM intent review

Finding. The highest-level intent is that correctness is structural, not behavioral: the thesis says correctness dimensions are structural facts carried by the program model, and Tier 1 is “impossible to write the bug.” chatgpt-review-03678390-3c02-49…

The T-37 brief currently lowers that bar for parse loci: src/v4/TASKS.md:1794-1799 would let workers implement a parse result where root and loci disagree, and src/v4/TASKS.md:1801-1806 makes the canonical fold catch that later. A faithful worker following this plan could therefore ship a representation where the bug is writeable and only conditionally detected. That is semantic dilution of the work’s purpose; the plan should require a structurally paired/opaque validated parse-tree authority before dispatch.

3. Verdict

REQUEST_CHANGES

The task correctly identifies the lost-token-fact problem and routes it through the existing Locus authority, but the proposed ParseTree { root, loci } shape makes the critical node↔locus correspondence a runtime/prose invariant. For a substrate boundary task, that needs to be structurally unrepresentable or enforced by an opaque validated constructor before this planning row directs implementation.

@briansrls

Copy link
Copy Markdown
Contributor Author

Review (root session, May 27): T-37 planning entry for TASKS.md — well-reasoned, no issues.

Grounding:

  • P1 and P2 violations are correctly identified: Token { lexeme, file, start, end } is the single authoritative carrier of source-location facts; 02_parse.dag silently drops them when producing ParseTree = Node.
  • The dissolution target is correctly the existing Locus::Textual { file: Symbol, extent: Extent } in std/diagnostic.dag (already declared terminal) rather than a new SourceSpan type — no new std/ carrier required.
  • Three alternative approaches (Map<Node,Locus>, position-keyed, counter-keyed) are explicitly rejected with sound rationale (node collision, position collision, disconnected integer table).

Design:

  • The LocusTree co-structural parallel approach is the right pick — preserves Node pivot (A1), has a typed combinator (fold_parse_tree), and Outcome<R> return on the fold makes child-count mismatch P3 fail-closed.
  • Escalation discipline is correct: load-bearing pipeline stage signatures (04_infer.dag InferredFacts map keying) require explicit scope authorization before touching — this is called out explicitly.
  • T-37 correctly declared as NOT blocking T-36.
  • Wave-based dispatch strategy (T-7 train → T-8 → T-9) is sensible.

No issues found. Approve.

@briansrls briansrls closed this May 27, 2026
@briansrls briansrls reopened this May 27, 2026
@briansrls briansrls closed this Jun 1, 2026
@briansrls
briansrls deleted the session/bold-ant-776 branch June 1, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant