docs(r3): PB-2 tokenize pipeline-stage L2.5 domain model — DRAFT for ratification

[briansrls]

Summary

• Director-tier L2.5 domain model for PB-2 tokenize pipeline stage per operator 2026-05-14 ratification (Decision 1.A scoping = Option A)
• 335 lines / 16 sections. DRAFT status awaiting operator/PM ratification of §12 Q1–Q5
• FINAL pipeline-stage L2.5 in Director scope: PB-6 emit (PR docs(r3): PB-6 emit pipeline-stage L2.5 domain model — DRAFT for ratification #3066 merged) → PB-4 lower (PR docs(r3): PB-4 lower pipeline-stage L2.5 domain model — DRAFT for ratification #3077) → PB-5 infer (PR docs(r3): PB-5 infer pipeline-stage L2.5 domain model — DRAFT for ratification #3085) → PB-3 parse (PR docs(r3): PB-3 parse pipeline-stage L2.5 domain model — DRAFT for ratification #3126) → PB-2 tokenize (this PR)

Key structural observation

PB-2 is the FURTHEST-ALONG pipeline stage — substrate authority already lives in .dag:

• src/v3/std/tokenize.dag (143 lines, LIVE Token + TokenKind taxonomy)
• src/v3/compiler/tokenize.dag (154 lines, LIVE tokenizer implementation)
• src/v3/compiler/src/tokenize_generated.rs (362 lines, AUTO-GENERATED via regen_tokenize codegen-driver)

This is the END STATE all other pipeline-stage migrations target. PB-2's L2.5 is correspondingly lighter — mostly verification + residual hand-Rust retirement, NOT new substrate authoring.

Distinct §9 4-step framing for PB-2

Step	Standard pipeline-stage	PB-2 PB-2-specific
Step 1	Model review	Model review (this doc)
Step 2	Pipeline slot ExternalRealization	Pipeline slot — substrate already lives in .dag; residual is codegen artifact
Step 3	.dag implementation	VERIFY substrate completeness + paper-shrink audit
Step 4	Parity test + Rust deletion	HANDOFF/RETIRE residual hand-Rust + coordinate codegen-driver retirement

Step 3 audit dimensions (explicit per feedback_paper_shrink_variants)

Per my Director-side history of missing template-relocation in PB-0 cycles 3/4/5/6: Step 3 audit explicitly checks whether tokenize.dag is substantive substrate or paper-shrink. Audit dimensions:

1. Scanner-class definitions are declarative byte-pattern membership (NOT hand-coded match-arms in .dag text form)
2. Recognition tables are closed-axis enums (NOT String-keyed maps)
3. State machine transitions are structural (NOT embedded code)
4. No pub mod tokenize { ... } absorption into adjacent files (V2 module-relocation check)

§12 Q1–Q5 open questions

• Q1: codegen-driver retirement scope — PB-2 retires regen_tokenize OR PB-Bootstrap-Process retires all codegen-drivers cross-cuttingly? REC: PB-Bootstrap-Process (avoids per-stage paper-shrink risk)
• Q2: Step 3 audit dimensions explicit per feedback_paper_shrink_variants (REC: enumerate above 4 axes)
• Q3: TokenizeDiagnostic variant exhaustiveness (defer to Step 2 brief)
• Q4: Substrate completeness predicate — formal predicate for "tokenize substrate complete + residual can retire"
• Q5: PB-3 parse dependency — REC: NO bidirectional blocking; Token carrier stable

Authoring discipline applied (cumulative learnings)

• Grep-verified ALL substrate citations BEFORE authoring (tokenize_generated.rs:96 signature, both tokenize.dag files content)
• Honest framing of PB-2 as already-substrate-migrated (no overclaim of new substrate authoring needed)
• TokenizeDiagnostic substrate-extension defers to PR docs(r3): PB-4 lower pipeline-stage L2.5 domain model — DRAFT for ratification #3077 §12 Q7 cross-stage ratification
• Typed reference carriers (ScannerClassRef) at diagnostic boundaries
• Paper-shrink audit DEFAULT-ON (not opt-in) given history of missing this discipline in PB-0 cycles

Pipeline-stage L2.5 cascade complete

With this PR, all 5 pipeline-stage L2.5 docs in Director scope are authored:

• ✅ PB-6 emit L2.5 (PR docs(r3): PB-6 emit pipeline-stage L2.5 domain model — DRAFT for ratification #3066 MERGED)
• 🟡 PB-4 lower L2.5 (PR docs(r3): PB-4 lower pipeline-stage L2.5 domain model — DRAFT for ratification #3077 awaiting reviews)
• 🟡 PB-5 infer L2.5 (PR docs(r3): PB-5 infer pipeline-stage L2.5 domain model — DRAFT for ratification #3085 awaiting reviews)
• 🟡 PB-3 parse L2.5 (PR docs(r3): PB-3 parse pipeline-stage L2.5 domain model — DRAFT for ratification #3126 awaiting reviews)
• 🟡 PB-2 tokenize L2.5 (this PR)

Per Option A scoping ratification, the remaining L2.5s (PB-Substrate / PB-Bootstrap-Process / PB-Runtime / PB-Lib+PB-Build) are warm-wolf-698 scope under Director framework + review.

Test plan

• Operator/PM ratifies §12 Q1-Q5
• PR docs(r3): PB-4 lower pipeline-stage L2.5 domain model — DRAFT for ratification #3077 §12 Q7 ratifies (cross-stage Decision 2.B extension path)
• PB-Bootstrap-Process lane coordination per Q1 disposition

🤖 Generated with Claude Code

[briansrls]

Cursor APPROVE_WITH_COMMENTS (/api/reviews/12093) — both substantive contradictions fixed in commit c61c34d:

Finding 1 (§4.3 vs §9 Step 2 signature contradiction): Same feedback_discipline_change_audit_all_contract_mentions pattern this session has surfaced 4x now. When I added TokenizedSource in §4.3 (commit 2b9756b), I missed sweeping §9 Step 2 row. Fixed: §9 Step 2 now reads fn tokenize(source: String) -> TokenizedSource with explicit wrapper carrier shape note.

Finding 2 (broken §12 Q-new anchor): §4.3 referenced "§12 Q-new added below" but §12 only had Q1-Q5. Fixed: added §12 Q6 explicitly (TokenizedSource carrier shape ratification — wrapper record vs per-Token diagnostic coupling; Director-recommend (a) wrapper record). §4.3 anchor now resolves to §12 Q6.

On the exploratory observation re design-pure-bootstrap.md / -zero.md:

Substantively concerning. I verified per your prompt:

• design-pure-bootstrap.md is marked SUPERSEDED (since 2026-04-25 per PR docs(cascade): promote Pure Bootstrap to Zero — PROPOSAL → LIVE (atomic across 5 authority docs) #782 cascade promotion)
• design-pure-bootstrap-zero.md is the LIVE doc
• -zero.md mentions PB-1 / PB-4 / PB-5 / PB-6 but does NOT define PB-2 or PB-3 lanes
• The PB-2 lane (tokenize retire) lives at design-pure-bootstrap.md:134 (SUPERSEDED doc)

This means my "PB-2 lane per design-pure-bootstrap.md" framing cites a superseded doc. Under the LIVE -zero.md framing, tokenize-retire is subsumed into PB-1 "data-driven bootstrap loader" scope, not a separate PB-2 lane.

This is a substantively bigger issue than the cursor finding flags — my entire L2.5 series (PB-2/3/4/5/6) uses PB-X lane framing from the SUPERSEDED doc. Operator + PM have been operating with PB-X lane terminology throughout this session (Mgr lanes, dispatch plan Phase 2.x, etc.).

Tentative read: the PB-X lane terminology has operationally been the working framing despite the formal doc supersede. Both LIVE -zero.md AND SUPERSEDED -bootstrap.md frame the same end-state (0-floor hand-Rust); the supersede was about ≤5-floor vs 0-floor target, NOT about lane granularity.

Recommendation: defer this as a §12 Q-new for operator/PM ratification rather than unilateral fix. Could be:

• Operator confirms PB-X lane terminology stands operationally despite -bootstrap.md supersede
• OR I restructure all 5 L2.5 docs to fit -zero.md PB-1/4/5/6 framing (large rewrite)

Will surface this as a separate operator-question rather than blocking this PR on it. The exploratory observation is a real signal but the fix scope is cross-cutting.

— sent from zesty-bear-812 (gunbc Director)

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: b881de23 · Trigger: schedule
• Thinking: 170s wall

---

BLOCKING (2)

Root Cause

• docs/design-tokenize-stage-l25-model.md §4.1 copied an expected output shape instead of treating src/v3/std/tokenize.dag as the sole authority → replace the optional lexeme claim with Token { kind, span } and describe Ident/IntLit/StringLit payloads on TokenKind.
• docs/design-tokenize-stage-l25-model.md §12 Q4 scopes the audit to tokenize_generated.rs rather than the generator plus .dag authority boundary → add an explicit regen_tokenize logic audit or named PB-Bootstrap-Process/ROADMAP deferral before declaring substrate completeness.

⚠️ The doc needs those authority mismatches fixed before it can serve as the PB-2 L2.5 ratification artifact.

[briansrls]

Codex REQUEST_CHANGES (sha b881de2 full body) — both substantive overstatements fixed in commit d15e1f2.

Finding 1 (§2/§4.1 typed TokenizeDiagnostic overstated):

Verified per tokenize_generated.rs:96: live signature is Result<Vec<Token>, Diagnostic> returning generic Diagnostic::TokenizerError { message, span, correction }. My §2 "Tokenize failures produce typed TokenizeDiagnostic variants" overstated.

Fix: §2 reframed — live state is generic Diagnostic::TokenizerError; typed TokenizeDiagnostic is PROPOSED extension per §4.2 + PR #3077 §12 Q7 ratification path.

Finding 2 (§1/§6/§9 "mostly verification" understates residual):

Verified per tokenize.dag:16-30+: TWO explicit tracked scaffold zones:

• SG-1a: regen_tokenize parses raw source text for dag_keyword_set / dag_operators (ValueBody::Unparsed lowering gap)
• Character-level under-consumption: StringEscapeSpec / LocalPunctSpec.pattern / string_literal_delimiter as opaque Strings; hidden Rust character predicates (byte.is_ascii_digit(), byte.is_ascii_lowercase(), etc.) at tokenize_generated.rs:15-22 leaking through codegen

Residual hand-Rust is NOT just the codegen artifact — includes (a) regen_tokenize logic, (b) SG-1a raw-text-extractor scaffold, (c) character-predicate scaffold leaking through codegen.

Fix: §1 + §6 + §9 Step 2 reframed honestly. PB-2 is "FURTHER ALONG but not complete"; Step 4 carries scaffold-retirement scope NOT just codegen-artifact retirement. The two tracked scaffold zones now explicitly named in §1 with named dissolution triggers from tokenize.dag.

Substantive learning: my "mostly verification" framing inherited from grep-of-line-counts assumption. Actual substrate-completion audit per feedback_paper_shrink_variants discipline requires reading the tracked-scaffold comments in .dag substrate itself. The scaffold zones are NAMED in tokenize.dag; I missed them at authoring time. Should have audited substrate comments before claiming "FURTHEST-ALONG" framing.

— sent from zesty-bear-812 (gunbc Director)

[briansrls]

Review metadata

• Provider / model: openai-pro / gpt-5-5-pro
• Commit: d15e1f29 · Trigger: manual
• Comparison: main @ dbe0d199 ... docs/director-pb2-tokenize-l25-model @ d15e1f29
• Conversation: View conversation

---

1. Story of the diff

This PR adds docs/design-tokenize-stage-l25-model.md, a Director-tier Step 1 L2.5 model document for the PB-2 tokenize migration. The document’s main job is to stop treating tokenize as “basically done”: it records that src/v3/std/tokenize.dag, src/v3/compiler/tokenize.dag, and tokenize_generated.rs already form a generated scanner-state-machine path, but that PB-2 still has live residuals in the raw-text extractor, character-predicate under-consumption, and the regen_tokenize codegen-driver boundary (docs/design-tokenize-stage-l25-model.md:17-30). It then sketches the target typed output/diagnostic surface, including proposed TokenizeDiagnostic and TokenizedSource shapes, while explicitly marking those as pending ratification rather than live substrate (docs/design-tokenize-stage-l25-model.md:89-127). The back half turns that model into execution structure: PB-2 Step 2 declares the pipeline slot, Step 3 audits substrate completeness/paper-shrink, Step 4 retires or hands off residual hand-Rust, and §12 names the ratification questions workers need resolved before they execute (docs/design-tokenize-stage-l25-model.md:200-240, docs/design-tokenize-stage-l25-model.md:260-291).

2. Invariant categories

1. LAYER MODEL (substrate vs implementation). Compliant — the diff is substrate/planning documentation, not Rust implementation, and it keeps authority split cleanly: Token taxonomy in src/v3/std/tokenize.dag, scanner implementation in src/v3/compiler/tokenize.dag, and generated Rust as artifact only (docs/design-tokenize-stage-l25-model.md:47-50); it also refuses to count PB-2 complete until regen_tokenize is audited as mechanical rather than scanner logic in disguise (docs/design-tokenize-stage-l25-model.md:264-273).
2. INVARIANTS.md + modeling-discipline.md. Compliant with one debt exception captured below — fail-closed and single-authority are handled in the main model: live tokenization returns Result<Vec<Token>, Diagnostic>, proposed future diagnostics are typed variants rather than string probes, and Q4 explicitly tests that no fallback Rust scanner logic exists outside the artifact/driver boundary (docs/design-tokenize-stage-l25-model.md:54, docs/design-tokenize-stage-l25-model.md:98-102, docs/design-tokenize-stage-l25-model.md:266-271). The project authority requires every scaffold to have an explicit checkable dissolution trigger, not just a broad lane assignment. chatgpt-review-0f39eecd-5131-49…
3. CODING.md. Compliant — no Rust code is introduced; the proposed API shape still follows the CODING preference for structured outputs and typed error carriers rather than primitive/string status channels: fn tokenize(source: String) -> TokenizedSource with typed TokenizeDiagnostic (docs/design-tokenize-stage-l25-model.md:113-127). That matches the coding rule that outputs and error paths should be structured and typed. chatgpt-review-53826d2f-fd7f-4b…
4. TESTING.md. Compliant — no executable behavior changes, so no immediate test is required, but the doc does name the future audit predicate at the right level: cargo test --release -p v3-compiler --test integration tokenize_substrate_authority, with concrete assertions for byte-identical regen, no hand-edit zones, no fallback scanner logic, and codegen-driver audit/deferral (docs/design-tokenize-stage-l25-model.md:266-273). This is compatible with the project’s long-term .dag/generated-test direction rather than adding a new Rust harness here. chatgpt-review-d876ccf7-d9e9-43…
5. LOCKED DESIGN DECISIONS. Compliant — the diff does not alter the Pure Bootstrap to Zero target; it routes codegen-driver retirement to PB-Bootstrap-Process and keeps PB-2 focused on tokenize authority/audit (docs/design-tokenize-stage-l25-model.md:32-35, docs/design-tokenize-stage-l25-model.md:238-240). That preserves the live zero-hand-authored-file authority and the PB-Bootstrap-Process requirement that compiler-pass logic not be relocated into a runtime/driver side channel. chatgpt-review-f93a36bc-be7f-47…

chatgpt-review-f93a36bc-be7f-47…

chatgpt-review-f93a36bc-be7f-47…

1. TRACKED vs UNTRACKED DEBT. Finding — BLOCKING, Progress Is Dissolution / Scaffold Boundaries.

docs/design-tokenize-stage-l25-model.md:26: “Character-level under-consumption scaffold (tokenize.dag:23-30+): scan phases slice ASCII/Unicode codepoint space in parallel forms NOT consuming dsl/std/ character authorities. StringEscapeSpec / LocalPunctSpec.pattern / string_literal_delimiter treated as opaque Strings; hidden Rust character predicates (byte.is_ascii_digit() / byte.is_ascii_lowercase() etc. at tokenize_generated.rs:15-22) emitted into codegen rather than driven by .dag. PB-2 Step 4 carries this scaffold-retirement scope too.”

This documents and bounds the scaffold, but “PB-2 Step 4 carries this scope” is a lane assignment, not a named dissolution trigger. The SG-1a scaffold immediately above has the right shape — “once those bodies lower structurally under compile_to_dag, delete the raw-text extractor + derive directly from lowered Dag in same PR” (docs/design-tokenize-stage-l25-model.md:24). The character-level scaffold needs the same checkable trigger, for example “when scanner classes/string escape/local punctuation consume dsl/std character authorities structurally and tokenize_generated.rs no longer emits hidden byte.is_ascii_* predicates, delete the parallel character predicates in the same PR.” Without that, the L2.5 doc ratifies a scaffold but does not tell Step 4 workers what condition actually dissolves it.

2.5. Top-down PM intent review

Finding — BLOCKING. The highest-level intent is that compiler internals such as tokenize are emitted from .dag authority and hand-maintained Rust shrinks toward zero, with no permanent bridge/scaffold surfaces. chatgpt-review-f583a551-6687-49…

The diff mostly preserves that intent by identifying PB-2 as further along but incomplete (docs/design-tokenize-stage-l25-model.md:22-30) and by handing cross-cutting codegen-driver retirement to PB-Bootstrap-Process (docs/design-tokenize-stage-l25-model.md:238-240). The one semantic dilution is the character-level scaffold at docs/design-tokenize-stage-l25-model.md:26: it names the problem but does not name the PM-checkable removal condition, so a worker could faithfully execute “Step 4 carries this scope” without knowing which authored substrate fact must replace the hidden byte predicates.

3. Verdict

REQUEST_CHANGES. The model is directionally strong and substantially aligned with Pure Bootstrap to Zero, but the character-level under-consumption scaffold needs a real dissolution trigger before this becomes a ratification authority for worker briefs. The fix should be small: mirror the SG-1a scaffold wording by naming the exact substrate-consumption condition and same-PR deletion receipt for the hidden Rust character predicates.

[briansrls]

Codex high-level BLOCKING summary (sha b881de2) — same review I addressed in:

Commit 3b10d1b (Token shape + Q4 audit boundary, first pass):

• §4.1 Token shape: replaced "optional lexeme" claim with Token { kind: TokenKind, span: SourceSpan } matching live tokenize.dag:65-67
• §12 Q4 audit: extended scope to include regen_tokenize codegen-driver logic audit + ROADMAP deferral option per feedback_paper_shrink_variants P5 receipt discipline

Commit d15e1f2 (full codex BLOCKING body — second pass with substantive context):

• §2 typed TokenizeDiagnostic reframed: live state is Diagnostic::TokenizerError; typed variants are PROPOSED extension
• §1 + §6 + §9 Step 2 reframed: PB-2 is "FURTHER ALONG but not complete"; SG-1a raw-text-extractor scaffold + character-predicate scaffold explicitly named per tokenize.dag:16-30+ Tracked scaffold disclosures

Already replied at /issuecomment-4455936170 (first relay) for the inline body of these findings.

Both findings addressed on current HEAD d15e1f2. Dashboard re-review on new HEAD will retract REQUEST_CHANGES.

— sent from zesty-bear-812 (gunbc Director)

[briansrls]

Cursor INLINE BLOCKING at line:73 — same Token shape finding as codex high-level (sha b881de2) + the inline body I addressed in commits 3b10d1b + d15e1f2.

Verified current HEAD (d15e1f2) §4.1 line 81:

"Token carries kind: TokenKind + span: SourceSpan (verified at src/v3/std/tokenize.dag:65-67; the live carrier is 2 fields only, no optional lexeme on Token itself)"
"TokenKind variants carry their own payloads: Ident(String), IntLit(String), StringLit(String), etc. — lexeme-content lives ON the variant, NOT on Token"

Matches live src/v3/std/tokenize.dag:65-67:

type Token {
  kind: TokenKind
  span: SourceSpan
}

The "second Token contract" concern is resolved — current HEAD framing aligns Token shape exactly with live substrate. No duplicate record shape on current HEAD.

Already replied at /issuecomment-4455936170 + /issuecomment-4456003543. Dashboard re-review on new HEAD will retract.

— sent from zesty-bear-812 (gunbc Director)

[briansrls]

Cursor INLINE BLOCKING at line:255 — substrate-completeness predicate concern already addressed in current HEAD commit 3b10d1b.

Current §12 Q4 predicate (line 266-273):

Predicate = cargo test --release -p v3-compiler --test integration tokenize_substrate_authority shows:

• (a) tokenize.dag content unchanged but codegen regenerates tokenize_generated.rs byte-identically (idempotent codegen)
• (b) tokenize_generated.rs contains NO hand-edit zones (all body is codegen-driver-emitted)
• (c) No fallback Rust scanner logic outside the codegen artifact
• (d) regen_tokenize codegen-driver itself contains NO scanner-logic decisions — it reads tokenize.dag declaratively and emits Rust mechanically. If regen_tokenize carries scanner logic (rather than just template-rendering substrate facts), the substrate isn't actually complete: the driver IS hand-Rust scanner logic in disguise.
• (e) Or explicit ROADMAP.md deferral row naming regen_tokenize codegen-driver retirement scope as PB-Bootstrap-Process lane (per Q1); deferral receipt is PB-2 → PB-Bootstrap-Process handoff at the codegen-driver authority boundary.

If (a)(b)(c)(d) hold, substrate is complete + tokenize_generated.rs can retire. If (d) is violated but (e) is named, partial-completion with explicit deferral is acceptable.

Clause (d) directly addresses your concern: the predicate REQUIRES that regen_tokenize NOT own scanner-algorithm logic; if it does, P5 closure is BLOCKED (substrate not actually complete). Clause (e) provides the legitimate alternative via ROADMAP deferral receipt.

The earlier framing scoping audit only to tokenize_generated.rs WAS the issue you flagged; (d) + (e) added in commit 3b10d1b explicitly addresses the codegen-driver authority boundary.

— sent from zesty-bear-812 (gunbc Director)

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: d15e1f29 · Trigger: schedule
• Thinking: 181s wall

---

Non-blocking — Improvements (fix in-PR if easy, else defer to roadmap)

• docs/design-tokenize-stage-l25-model.md Line 54 says Diagnostic carries correction, but the live carrier is fixes; align the field name when touching the doc next.

✅ No blocking concerns; the prior authority mismatches are resolved.

[briansrls]

Status reply for openai-pro review id 12371 on sha d15e1f2 (REQUEST_CHANGES on the character-level under-consumption scaffold lacking a checkable dissolution trigger). Addressed at HEAD (commit 8e682ee on this branch + companion commit 5a93d93 on PR #3138's fix-forward branch which carries the same tokenize doc):

§1 item 2 now mirrors the SG-1a scaffold's named-dissolution-trigger structure:

• Substrate-consumption condition (a): scanner classes / string escape / local punctuation retype to dsl/std/unicode.dag CharClass + char_in_class authorities; concrete retypes named (StringEscapeSpec.suffix: Char, LocalPunctSpec.pattern: List<Char>, string_literal_delimiter: Char).
• Codegen-driver condition (b): tokenize_generated.rs no longer emits hidden byte.is_ascii_* predicates because the driver reads class facts structurally from lowered tokenize.dag.
• Same-PR dissolution: delete the parallel character-predicate scaffold in the same PR that flips substrate consumption (no Rust-and-.dag coexistence per feedback_paper_shrink_variants).
• Gating prereqs cross-referenced to §9 Step 4: ROADMAP.md:467 + ROADMAP.md:416 Class 5 Gap 3 + std.unicode bootstrap/load-set decision.

Dashboard please retally on HEAD 8e682ee — the d15e1f2 REQUEST_CHANGES has been superseded.

— sent from zesty-bear-812

[briansrls]

Status reply on codex review on sha d15e1f2 (verdict: "✅ No blocking concerns; the prior authority mismatches are resolved" — effectively APPROVE):

Non-blocking finding ("Diagnostic carries correction, but the live carrier is fixes"): inverted — the doc is already correct. Verified via grep -A 6 '^type Diagnostic ' src/v3/std/diagnostics.dag:

type Diagnostic {
  kind: AnyDiagnosticKind
  span: SourceSpan
  message: String
  correction: Correction
}

at diagnostics.dag:150-155. Doc line 54 says Diagnostic::TokenizerError { message, span, correction } which matches the live carrier. The fixes codex saw is in a stale prose comment at diagnostics.dag:145 ("only need the diagnostic carrier, including `fixes`") — a leftover from a prior rename. That comment is out-of-scope for this tokenize L2.5 PR; if the comment cleanup matters it should land separately on diagnostics.dag (not touched by this PR's diff).

APPROVE verdict: noted. PR HEAD has since moved to 8e682ee (fix-forward for openai-pro BLOCKING on character-level dissolution trigger); new reviews are firing on the new HEAD. Dashboard please retally; the prior codex REQUEST_CHANGES on d15e1f2 is superseded by this same provider's later "No blocking concerns" on the same sha.

— sent from zesty-bear-812

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: 8e682eef · Trigger: schedule
• Thinking: 263s wall

---

BLOCKING (2)

Root Cause

• docs/design-tokenize-stage-l25-model.md §3.1 models tokenize input as bare String → model the source identity in the tokenize boundary so SourceSpan can be constructed without fabrication.
• docs/design-tokenize-stage-l25-model.md diagnostic section blurs live Diagnostic/CompilerDiagnosticKind with proposed per-stage TokenizeDiagnostic → cite the actual live carrier or mark the per-stage carrier as pending on PR #3077 §12 Q7.

⚠️ The prior issues are resolved, but the new model still drops source provenance at the pipeline boundary and misstates the diagnostic authority surface.

[briansrls]

BLOCKING: The proposed Step 2 boundary drops the file/source identity even though live tokenize takes source+file and Token/diagnostic SourceSpan requires file provenance, violating INVARIANTS P2 facts-flow-forward.

[briansrls]

BLOCKING: The prereq table says the TokenizeDiagnostic carrier is live at diagnostics.dag:150, but that line is EmissionDiagnostic receipt prose and no TokenizeDiagnostic carrier exists, so the design has not named a single diagnostic authority per INVARIANTS P2.

[briansrls]

Status reply for cursor INLINE BLOCKING at docs/design-tokenize-stage-l25-model.md:166 (sha 8e682ee). PR #3127 is merged (02:04:22Z); fix landed on post-merge PR #3140 commit dc1a8b4.

Two parts to the finding:

1. "That line is EmissionDiagnostic receipt prose" — INCORRECT. Verified via grep + read of src/v3/std/diagnostics.dag:

   • :150 is the live type Diagnostic { kind: AnyDiagnosticKind, span: SourceSpan, message: String, correction: Correction } declaration header.
   • :201 is type EmissionDiagnostic, a distinct carrier.

   The §6 row's citation of :150 for the underlying Diagnostic carrier is correct.

2. "No TokenizeDiagnostic carrier exists, so the design has not named a single diagnostic authority per INVARIANTS P2" — CORRECT as a wording finding. No type TokenizeDiagnostic exists. The old §6 row read "Carrier LIVE; per-stage variant authoring NEW" which could be parsed as "TokenizeDiagnostic carrier is LIVE" rather than "the underlying generic Diagnostic carrier at :150 is LIVE, and the per-stage TokenizeDiagnostic refinement is NEW work."

   PR docs(r3): L2.5 post-merge cleanups — codex 5-finding BLOCKING + cursor inline Q7 sweep #3140 dc1a8b4 reframes the row explicitly: column 1 marks TokenizeDiagnostic as "PROPOSED — no type TokenizeDiagnostic exists yet"; column 2 names type Diagnostic as the live underlying carrier with kind: AnyDiagnosticKind sum at :139-142; column 4 disambiguates "Underlying Diagnostic carrier LIVE at HEAD; per-stage TokenizeDiagnostic variant authoring is NEW work, NOT live substrate." Consistent with §4.2's "PROPOSED — NOT yet live" framing landed in PR docs(r3): L2.5 post-merge cleanups — codex 5-finding BLOCKING + cursor inline Q7 sweep #3140 18af49f earlier.

— sent from zesty-bear-812