[codex] Add v2.5 tokenizer stage

[briansrls]

Summary

Adds the v2.5 fork lexer as .dag. This PR is stacked on Tier 1 session/vivid-lynx-807 so the tokenizer consumes canonical v2.5 boundary carriers instead of defining local result/token authorities.

Current signature: tokenize(source: Source) -> Outcome<TokenStream> at src/v2.5/01_tokenize.dag:538. Outcome / Diagnostic come from v2_5.std.diagnostic and Token / TokenShape / TokenStream come from v2_5.std.token at src/v2.5/01_tokenize.dag:3 through src/v2.5/01_tokenize.dag:13.

The Tier 1 stage kit now has a matching TokenizeStageOut { tokens: Outcome<TokenStream> } carrier at src/v2.5/std/stage_types.dag:1033; the pipeline header was reconciled to point at the typed stage-port records instead of the stale bare SourceText -> TokenStream shape.

No Rust files are hand-edited.

Stage Shape

• Input: local Source at src/v2.5/01_tokenize.dag:15, carrying source text plus both path text and file identity for SourceSpan and diagnostic Locus.
• Output: canonical TokenStream from v2_5.std.token, populated by make_token at src/v2.5/01_tokenize.dag:138.
• Failure: canonical Diagnostic through Outcome::Rejected, constructed at src/v2.5/01_tokenize.dag:130.
• Token category: canonical TokenShape; this replaces the former local TokenKind coproduct. Keyword and punctuation scans produce ShKeyword, ShIdent, ShLit*, ShStr*, punctuation shapes, and ShEof directly.

Cross-Stage Interfaces Consumed

• v2_5.std.node.Symbol at src/v2.5/01_tokenize.dag:3.
• v2_5.std.identity.SourceSpan at src/v2.5/01_tokenize.dag:4.
• v2_5.std.diagnostic.{Diagnostic, Outcome, Textual, ByteRange, Unavailable, UserInputBoundary} at src/v2.5/01_tokenize.dag:5.
• v2_5.std.token.{Token, TokenShape, TokenStream} at src/v2.5/01_tokenize.dag:13.
• v2_5.std.stage_types.TokenizeStageOut at src/v2.5/std/stage_types.dag:1033 records the fail-closed tokenizer stage output shape.

Coproduct Receipts

The closed coproduct structures are the receipts; no in-code prose tags are used.

• CharacterClass at src/v2.5/01_tokenize.dag:21 is terminal lexer-local input classification consumed by char_in_class at src/v2.5/01_tokenize.dag:160.
• LexerState at src/v2.5/01_tokenize.dag:44, ScanOutcome at src/v2.5/01_tokenize.dag:73, and TokenizeExpectation at src/v2.5/01_tokenize.dag:77 are closed state/result sums for the tokenizer fold.
• ClassScanState, LineScanState, StringBodyState, and StringTokenRole at src/v2.5/01_tokenize.dag:48, src/v2.5/01_tokenize.dag:52, src/v2.5/01_tokenize.dag:56, and src/v2.5/01_tokenize.dag:61 are closed lexer-local scan modes.

What Dissolved

• Local Outcome<T> and local Diagnostic dissolved into v2_5.std.diagnostic imports at src/v2.5/01_tokenize.dag:5; local diagnostic reason constructors became Symbol rows at src/v2.5/01_tokenize.dag:87.
• Local TokenKind, Keyword, LiteralKind, Punctuation, Token, and TokenStream dissolved into canonical TokenShape / Token / TokenStream imports at src/v2.5/01_tokenize.dag:13.
• The prior bare tokenizer signature in stage_types.dag dissolved into the typed TokenizeStageOut carrier at src/v2.5/std/stage_types.dag:1033; the retained pipeline header now points at the stage-port records.
• Tokenization is driven by fold_source_chars at src/v2.5/01_tokenize.dag:142; the top-level tokenize supplies the fold algebra at src/v2.5/01_tokenize.dag:542.
• ShUnknown-style fallback is fail-closed: unexpected characters reject at src/v2.5/01_tokenize.dag:487, unterminated strings reject at src/v2.5/01_tokenize.dag:372, and unterminated interpolation rejects at src/v2.5/01_tokenize.dag:450 and src/v2.5/01_tokenize.dag:565.

Syntax Authority Status

Keyword and punctuation spelling tables remain local bridges because v2.5 does not yet have a canonical SyntaxSpec.keyword_set / operator-spec carrier on the Tier 1 base.

• keyword_table: src/v2.5/01_tokenize.dag:92.
• single_punct: src/v2.5/01_tokenize.dag:108.
• double_punct: src/v2.5/01_tokenize.dag:117.

Dissolution trigger: replace these rows with the canonical v2.5 syntax-spec projection when that substrate lands.

Regression Coverage

Co-located .dag regression claims remain for the interpolation edge cases:

• nested_brace_interpolation_regression at src/v2.5/01_tokenize.dag:594 checks that "x {foo({bar: 1})} y" is accepted after nested interpolation braces decrement correctly.
• unterminated_interpolation_regression at src/v2.5/01_tokenize.dag:628 checks that "x {name rejects with unterminated_interpolation.
• The two Bool evaluations at src/v2.5/01_tokenize.dag:638 and src/v2.5/01_tokenize.dag:643 are not executable CI coverage yet. PM confirmed this is blocked on a separate v2.5 verification substrate worker, not local tokenizer scope.

Yellow List

• Source is still local at src/v2.5/01_tokenize.dag:15; it should move to a canonical v2.5 source-text carrier when one lands.
• Keyword/operator spelling tables are local tracked bridges pending canonical v2.5 syntax spec.
• Regression claims are machine-readable but not runner-enforced until the routed v2.5 verification substrate lands.
• The current fold-driven scanner can rescan source slices inside each driver step (scan_class, scan_to_eol, and scan_string_body all fold over source text). This is accepted for the fork slice and should dissolve into a cursor/span traversal primitive or indexed source view when performance becomes a v2.5 concern.
• Unknown string escapes currently preserve v2 behavior by retaining the backslash plus escaped character (escaped_char at src/v2.5/01_tokenize.dag:282; v2 process_escapes_loop uses the same fallback at src/v2/01_tokenize.dag:459). Fail-closed escape validation should move to the canonical string-literal syntax policy when that syntax carrier lands.

Validation

Passed locally after conflict resolution against session/vivid-lynx-807 and stage-contract reconciliation:

rm -rf /tmp/v25-tokenize-conflict-smoke && target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-conflict-smoke --target rust

Result: indexed 7 modules, resolved 7 sources, compiled: 11 files emitted, 0 diagnostics.

[briansrls]

Addressed the EOF observation in commit 1883386: the in-fold EOF scan now advances the cursor sentinel-style while keeping the EOF token span at the real end of input, so it cannot trip the progress guard if reached after trivia scanning. Verified again with target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-smoke --target rust (0 diagnostics).\n\nThe InterpolationEnd => StringMiddleToken case is intentional for a }...{ segment: the resumed string segment is not terminal if it opens another interpolation. The drop_last O(n) rebuild is also accepted for this fork slice because interpolation depth is lexer-local state and the stack is bounded by source nesting, not a Node side table.\n\n— sent from nimble-swift-454

[briansrls]

Verified against current head f9efd942750e6fbf09b12439507a3336c0967a67: the finding is correct. nested_brace_interpolation_regression_holds and unterminated_interpolation_regression_holds are plain Bool data declarations, and this branch has no v2.5 TestClaim/TestSuite runner or other consumer that would make them fail CI.

I am not pushing a fake local TestClaim wrapper here because it would still be unconsumed. I am also not adding hand-written Rust tests or importing v3/v4 verification, because the fork brief forbids both for this stage. Adding a v2.5 verification substrate or runner looks like a separate routed dependency rather than an in-scope 01_tokenize.dag edit, so I surfaced that blocker to the PM. The tokenizer bug fixes remain in the PR, but the regression coverage is not executable yet.

— sent from nimble-swift-454

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: 1883386f · Trigger: schedule
• Thinking: 243s wall

---

BLOCKING (3)

Root Cause

• src/v2.5/01_tokenize.dag Standalone compilation is being achieved by cloning std diagnostic/node carriers → consume the canonical carriers or add an explicit bounded bridge with a dissolution trigger.
• src/v2.5/01_tokenize.dag The tokenizer introduces a fresh token boundary shape instead of reusing/classifying the existing TokenShape/LexRules surface → reuse the canonical carrier or add valid disposition tags with gates.
• src/v2.5/01_tokenize.dag The tokenizer owns .dag lexical facts as local maps/enums rather than consuming the declared syntax model → route keyword/operator recognition through SyntaxSpec or a tracked LexRules projection.

⚠️ The new tokenizer stage needs to remove or explicitly track these parallel syntax/result/token authorities before it sets the v2.5 boundary shape.

[briansrls]

BLOCKING: Outcome redefines the compiler result carrier locally instead of importing the canonical std diagnostic Outcome, creating a parallel stage-boundary authority in violation of MODELING M6 and INVARIANTS P2.

[briansrls]

BLOCKING: TokenKind is a new stage-boundary coproduct carried on Token without the required 🟢/🟡/🔴 dissolution tag, violating docs/modeling-discipline Practice 4 and INVARIANTS P5.

[briansrls]

BLOCKING: keyword_table re-authors the .dag keyword set instead of deriving from SyntaxSpec.keyword_set/dag_keyword_set, so keyword classification can drift from the canonical syntax authority under MODELING M7 and INVARIANTS P2.

[briansrls]

Addressed the boundary-carrier review on current head 1a4e94e39:

• Retargeted [codex] Add v2.5 tokenizer stage #3423 onto the Tier 1 base session/vivid-lynx-807 and merged that interface kit, so this tokenizer now imports v2_5.std.diagnostic.{Diagnostic, Outcome} and v2_5.std.token.{Token, TokenShape, TokenStream} instead of defining local result/token carriers.
• Removed the local TokenKind / Keyword / LiteralKind / Punctuation boundary coproducts; tokenizer output now uses canonical TokenShape.
• Added in-file 🟢 coproduct tags for the remaining local lexer state coproducts.
• Kept keyword_table, single_punct, and double_punct as explicit 🟡 feature:v2_5-syntax-spec-* bridges because this Tier 1 base does not yet expose a canonical SyntaxSpec.keyword_set / operator-spec carrier to consume.
• Restored non-tokenizer paths to match the Tier 1 base; the PR diff is back to just src/v2.5/01_tokenize.dag.

Validation: target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-smoke --target rust passes with 7 modules, 11 emitted files, 0 diagnostics.

— sent from nimble-swift-454

[briansrls]

Addressed the stage-boundary mismatch on head 69bd04653:

• Removed the stale prose pipeline signature block from src/v2.5/std/stage_types.dag.
• Added typed TokenizeStageOut { tokens: Outcome<TokenStream> } in stage_types.dag, importing Outcome from v2_5.std.diagnostic, so the phase kit now matches the fail-closed tokenizer result.
• Removed the in-file 🟢/🟡 receipt comments from src/v2.5/01_tokenize.dag; coproduct receipts and bridge status now live in the PR body with line citations.

Validation: target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-smoke --target rust passes with 7 modules, 11 emitted files, 0 diagnostics.

— sent from nimble-swift-454

[briansrls]

Rebased/merged #3423 onto the reconciled Tier 1 head dfb926f8c after the THESIS-faithful substrate cascade. The only conflict was the already-known stage_types.dag tokenizer boundary: I kept the reconciled 2-variant substrate comments, removed the stale prose pipeline signature, and preserved the typed TokenizeStageOut { tokens: Outcome<TokenStream> } carrier.

Tokenizer remains NodeKind-agnostic; no TypeNode/ComputationNode/DeclarationKind/CompositeRole usage was present. ByteRange now comes from the new substrate shape, and the current v2 compiler accepts the tokenizer cursor positions there with no diagnostics.

Validation: target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-smoke --target rust passes with 7 modules, 11 emitted files, 0 diagnostics.

— sent from nimble-swift-454

[briansrls]

Addressed the latest Claude comments on head 6a7a12a12:

• Removed the unused lexeme parameter from diagnostic(...) and all call sites. Validation still passes with target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-smoke --target rust (7 modules, 11 emitted files, 0 diagnostics).
• Verified the O(n²) scan note. It is correct for the current fold-driven implementation; I added it to the PR body yellow list as implementation debt for a future cursor/span traversal primitive or indexed source view.
• I did not restore the prose pipeline contract block in stage_types.dag: that block was the prior source of a split stage-boundary authority. The typed contract is now TokenizeStageOut { tokens: Outcome<TokenStream> }, and the PR body carries the narrative.

— sent from nimble-swift-454

[briansrls]

Verified the Cursor approval observations against current head 0586ed7e5:

• nested_brace_interpolation_regression only asserts ExpectProduced; that is intentional for this pre-runner slice. The paired negative regression checks the fail-closed EOF/interpolation path. Token-shape assertions should move to the v2.5 verification surface once that lands.
• keyword_table remains a local authority; this is already tracked in the PR body yellow list as pending canonical v2.5 syntax spec / keyword-set projection.
• The _holds: Bool declarations are not hard-failure CI assertions today. That blocker was verified earlier and PM routed it to the separate v2.5 verification substrate worker; the PR body now states these are machine-readable but not runner-enforced yet.

No code change needed for these approval-only observations. Current GitHub head is mergeable, but dashboard readiness still has only one counted approval on the current head.

— sent from nimble-swift-454

[briansrls]

Verified the latest Claude observations against the current branch.

• The deleted prose pipeline block is intentional: the durable contract is now the typed TokenizeStageOut { tokens: Outcome<TokenStream> } carrier in src/v2.5/std/stage_types.dag. Restoring the old tokenize : SourceText -> TokenStream prose would recreate the split-boundary issue from the earlier review.
• The fold-driven scanner cost is real; it was already tracked in the PR body's yellow list with a dissolution trigger to move to a cursor/span traversal primitive or indexed source view.
• Unknown escapes are also real, but this preserves v2 tokenizer behavior: v2's process_escapes_loop keeps unknown escapes as backslash plus character. I added this to the yellow list as deliberate compatibility pending a canonical string-literal syntax policy carrier.

No code changes were needed for these non-blocking observations; PR body citations were refreshed.

— sent from nimble-swift-454

[briansrls]

Resolved the merge conflict with session/vivid-lynx-807 in commit 82a7533 by merging the updated base branch into session/nimble-swift-454-v25-tokenize.

The only manual conflict was the stage_types.dag pipeline header. I kept the base branch's durable contract block but corrected it to reference the typed stage-port records, including TokenizeStageOut { tokens: Outcome<TokenStream> }, instead of the stale bare SourceText -> TokenStream tokenizer shape.

Validation after the merge:

rm -rf /tmp/v25-tokenize-conflict-smoke && target/debug/v2-compiler compile --source-root src/v2.5 --output-dir /tmp/v25-tokenize-conflict-smoke --target rust

Result: indexed 7 modules, resolved 7 sources, compiled: 11 files emitted, 0 diagnostics.

GitHub now reports PR #3423 head 82a753312f238a75f0e04f2970dc66b995036f1b as MERGEABLE; dashboard review cache still shows the previous head while it catches up.

— sent from nimble-swift-454

[briansrls]

Closing per operator wrap-up directive 2026-05-20.