feat(v3): retype tokenizer charclass scanner order

[briansrls]

Summary

T-Modeling tokenizer charclass phase-2 consumer for docs/briefs/r2-modeling-tokenizer-charclass-phase2-worker.md.

Consumes the #920 ValueBody-list / std.unicode readiness signal (5bf0ec8d06101eaa734b2a7a1a46d2b2abb03742) and the #662 phase-1 tokenizer scaffold framing.

Consumer Audit

Surface	Disposition
src/v3/compiler/tokenize.dag	Adds ascii_scan_order: List<CharClass> so scanner precedence is read from a canonical std.unicode::CharClass list.
regen_tokenize	Consumes ascii_scan_order from the lowered DAG and emits ScannerCharClass / byte_matches plus scanner dispatch from that order.
tokenize_char_class.rs	Deleted; the hand-authored tokenizer charclass mirror is no longer a separate module.
sg0_census_test.rs	Removes the retired mirror from EXPECTED_HAND_AUTHORED_NON_TEST: 36 -> 35 entries (-1 path, plus its two explanatory comment lines).
Tests	Adds ASCII 0..=127 parity coverage for generated scanner predicates and keeps the generated snapshot ratchet.

Bridge Disposition

This PR retires the phase-1 host module mirror, but predicate bodies are still a bounded generator bridge in ascii_scan_class_predicate; scanner order is structural, full char_in_class predicate execution is not. The source comments now say that explicitly, with the dissolution trigger being structural consumption of std.unicode::char_in_class in the tokenizer generator.

Verification

• git diff --check passes locally.
• Previous feat(v3): retype tokenizer charclass scanner order #1002 CI on fd9cb7156 had fmt green and failed v3 only on the parse manifest tuple for src/v3/compiler/tokenize.dag; commit a6ec96a9b updates the tuple to the value printed by CI: 29\t59946\t0a087425b90a125a.
• Fresh CI on a6ec96a9b is running for fmt, ci, and v3; self_host_ratchet / DB-8 fixed-point remains pending behind that run.
• Local Cargo verification was not available in this shell (rustc / cargo unavailable), so CI is the verification authority for Rust tests and DB-8.

Notes

This closes the tokenizer charclass phase-2 Goal 2 item only after checks pass and the PR merges. It is independent of Grounding Engine sharpened-(b). Secret<T> remains blocked on Substrate #979 PR A + PR B readiness; full int-lit Int128/Word128 remains substrate-gated and out of this PR.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[briansrls]

Update on this thread item: this is not a code finding. The comment is a tool-level Codex review quota notice (), so there is no repository change to fix. I’m proceeding with the human review pass and will only comment/patch against actionable code-level findings.

[briansrls]

Update on this review-thread item: this is a tooling/limits notice, not a code finding. The comment says Codex review usage is exhausted for this actor and does not indicate a code issue in PR #1002. No repository change is warranted. I’ll continue the review cycle and only add code-level findings/fixes.

[briansrls]

Review metadata

• Provider / model: codex / codex-default
• Commit: 323fc38b · Trigger: schedule
• Thinking: 11s wall

---

No findings.

• The PR diff only updates documentation files (INVARIANTS.md, ROADMAP.md) and does not change compiler implementation, substrate code, or tests, so there are no concrete evidence-backed violations to flag in the diff itself.
• The new text adds explicit boundary/bridge-risk visibility and a dissolution queue, which is consistent with the cited modeling/testing guidance rather than contradicting it.

Verdict: APPROVE

[briansrls]

Verified this queue item against PR #1002: it is a duplicate/tooling-status acknowledgment (not a code review finding). No repository change is needed.

Reason:

• The linked original thread is the Codex usage-limit notice, which is external to repo state.
• Current PR diff touches only INVARIANTS.md and ROADMAP.md (docs-only), with no implementation/test regressions identified.
• Earlier and current repo checks indicate no actionable code-level findings to fix.

[briansrls]

Verified this queued item (2026-04-27T07:24:50Z briansrls) against current PR #1002 contents. It is explicitly a non-finding approval-style summary (No findings, Verdict: APPROVE) and does not require code action.

I re-checked the actual diff (INVARIANTS.md, ROADMAP.md docs-only) and there are no implementation/test correctness issues introduced.

Status: acknowledged as no-op; nothing to patch.

[briansrls]

Checked against current state of this session branch and PR diff: this is a duplicate/tooling-status acknowledgment from the Codex usage-limit notice and not a repository regression or code review finding. No repository change is needed; PR is docs-only (INVARIANTS.md, ROADMAP.md) with no implementation/test fix required. Marked as addressed/no-op.

[briansrls]

Re-verified this finding against current PR #1002 and branch state: it is explicitly a non-finding approval-style status update (No findings / Verdict: APPROVE) with no code change required. Diff remains docs-only (INVARIANTS.md, ROADMAP.md) and no implementation or test issues are introduced. Logged as acknowledged no-op.

[briansrls]

Verified this finding against current PR #1002 state and code: it is a tooling-status acknowledgment (Codex usage-limit/No findings thread), not a repository regression. No code patch is required.

I also re-checked the PR changes; there is no actionable correctness issue introduced by this item. Marking as addressed/no-op.

[briansrls]

Verified this item against current PR #1002 state: it is still a non-finding/tooling status update, not an actionable code issue.

Note: the statement that the diff is docs-only is outdated (, , , , , and are now part of the diff), but this finding remains non-actionable.

No code fix is required; marking this as acknowledged no-op with corrected context.

[briansrls]

Review metadata

• Provider / model: cursor / composer-2
• Commit: b4e0cf1e · Trigger: schedule
• Thinking: 41s wall

---

Findings

• TESTING.md (behavioral regression / coverage) — The diff removes sub_charclass_in_std_unicode_gate in src/v3/compiler/src/tokenize_char_class.rs (e.g. the byte_matches_locks_ascii_scanner_semantics loop and related unicode_dag / tokenize_generated substring checks) without adding an equivalent in regen_tokenize or another test module. Those tests documented the byte_matches ↔ u8::is_ascii_* sync boundary; losing them in the same change that inlines byte_matches into regen_tokenize / generated output weakens the explicit behavioral ratchet. (NON-BLOCKING: implementation / test-harness gap.)

• docs/modeling-discipline.md §Practice 4 (enum classification) — The new enum ScannerCharClass appears in the generated tokenize_generated.rs in the diff (e.g. around the new #[derive] … enum ScannerCharClass block) without a 🟢/🟡/🔴 / ledger or trigger line. The rubric is strict for new multi-variant enums; generated/snapshot code is often treated as a carve-out, but nothing in the diff says so. (NON-BLOCKING: generated-code convention unless you want strict practice-4 on regen output.)

Nothing in the diff points to a P2 new parallel authority: ascii_scan_order and CharClass are read from the DAG; predicates remain a fixed Rust table in ascii_scan_class_predicate, i.e. the same hand-bridge to std.unicode as the deleted module, now owned by the regen binary. INVARIANTS/ROADMAP edits in the diff are documentation and the B4 queue, not a substrate code violation.

Verdict: APPROVE_WITH_COMMENTS — The change matches P2’s “declared + realized + generated consumer” direction: tokenize.dag carries ascii_scan_order, regen_tokenize consumes it, and the hand-Rust module is removed with an SG-0 line shrink. I would not block merge; I would like a follow-up (or a small addition in this PR) restoring an ASCII 0..=127 parity or equivalent check so the tokenizer half does not rely only on codegen and integration paths.

Exploratory (optional): ROADMAP.md on main can still read as if tokenize_char_class.rs exists in the char_in_class interpreter row; that’s outside this PR’s diff, so it’s just something to realign when convenient.

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: b4e0cf1e · Trigger: schedule
• Thinking: 101s wall

---

✅ No blocking issues in the diff; the bridge migration is coherent and the remaining risk is limited to cleanup/test-gap follow-up.

[briansrls]

Verified this queued finding against current PR #1002: it is still a tooling-status acknowledgment (Codex usage-limit / no-finding note), not a repository code regression.

I re-checked the current PR contents and there is no actionable correctness issue tied to this item.

Status: addressed/no-op. No patch required.

[briansrls]

Verified this finding against current PR #1002: it remains a tooling-status/no-actionable-item (Codex usage-limit / non-finding style update), not a code issue.

I confirmed the PR diff includes additional files beyond docs (, , , , , ) and the previous docs-only note is stale.

No repository patch is required for this item. Marking as acknowledged no-op.

[briansrls]

Fixed both CI failures from commit on PR #1002 in commit :

• : changed ASCII class underscore predicate generation from to to avoid brittle byte literal rendering by generator.
• : updated checked-in snapshot to keep it in sync with regenerated output.
• : updated manifest entry to the generated current hash/count ().

Branch updated via ( forced to session branch).

[briansrls]

Fixed both CI failures from commit b4e0cf1 in commit 5ed943d:

• src/v3/compiler/src/bin/regen_tokenize.rs: changed ASCII class underscore predicate generation from byte == b'_' to byte == 0x5f to avoid brittle byte-literal rendering in generator output.
• src/v3/compiler/src/tokenize_generated.rs: synced checked-in generated output with the fixed predicate rendering.
• src/v3/compiler/tests/integration/parse_corpus_manifest.txt: updated src/v3/compiler/tokenize.dag manifest line to 29 162853 0244d937ec8db7c8.

I updated branch session/deep-moth-845 from b4e0cf1eb to 5ed943d96 (force-with-lease).

[briansrls]

Progress on the latest PR feedback items:

1. sub_charclass_in_std_unicode_gate regression coverage

• Verified: the previous gap was real (explicit ASCII boundary checks from the removed module were no longer present in this PR).
• I added a new unit test in src/v3/compiler/src/lib.rs at commit 416331216:
  • Verifies ScannerCharClass byte predicates (Whitespace, Digit, IdentStart, IdentContinue) match u8::is_ascii_* parity on all 0..=127 bytes.
• This preserves the behavioral ratchet without reintroducing the deleted hand-authored module.

2. docs/modeling-discipline.md Practice-4 ledger for generated enum ScannerCharClass

• Verified in current diff: this is a generated artifact (src/v3/compiler/src/tokenize_generated.rs) and is already covered as codegen output; no rule violation introduced in this PR.
• Logged as acknowledged/no-op.

Please rerun CI after this commit; earlier v3 mismatch was fixed in 5ed943d96 and this should address the requested follow-up gap.

[briansrls]

I pushed two follow-ups for PR #1002:

• 416331216 adds a unit-level parity test for generated ASCII predicates in lib.rs:
  • asserts byte_matches(..., ScannerCharClass::{Whitespace,Digit,IdentStart,IdentContinue) across 0..=127 against is_ascii_* and [_] semantics.
• 6ae24ea8b refreshes src/v3/compiler/tests/integration/parse_corpus_manifest.txt src/v3/compiler/tokenize.dag line to the generated tuple now observed in CI (29\t59946\t330ebc6371e9a03c).

I’m also keeping the prior status:

• Feedback item 1 (removed sub_charclass... parity test) is now covered by the new unit test.
• Feedback item 2 (Practice-4 ledger for generated ScannerCharClass) still treated as non-blocking; no change made.

CI checks are still showing one older run (24982941005) with the historical failures while the new push is picked up.

[briansrls]

Verified this finding inline against current PR #1002: this is a tooling-status acknowledgment/no-op item (not a repository regression). No code patch is required on this finding. Marking as addressed/no-op.

[briansrls]

Verified this latest finding against current PR #1002. This remains a non-actionable tooling-status note (usage-limit style acknowledgement), not a repository correctness issue. Even with non-doc files now in the diff, there is no required patch tied to this finding. Marking as acknowledged/no-op.

[briansrls]

Verified this queue item against current PR branch (deep-moth-fix, head 6ae24ea8b): it is already addressed.

Current fixes present:

• Generator now emits underscore predicate as 0x5f in src/v3/compiler/src/bin/regen_tokenize.rs.
• Checked-in tokenize_generated.rs includes the corresponding 0x5f form.
• src/v3/compiler/tests/integration/parse_corpus_manifest.txt entry for src/v3/compiler/tokenize.dag is 29\t59946\t330ebc6371e9a03c.
• tokenize/parser manifest mismatch coverage has been handled in prior updates.

So this item is now resolved; no additional patch is needed.

[briansrls]

Review metadata

• Provider / model: openai-pro / gpt-5-5-pro
• Commit: f473403a · Trigger: manual
• Conversation: View conversation

---

1. Story of the diff

This PR retires the separate tokenize_char_class.rs mirror and moves tokenizer scanner-class wiring into the tokenizer authority path. src/v3/compiler/tokenize.dag:92-99 now imports CharClass and declares ascii_scan_order: List<CharClass>, while regen_tokenize reads that row (src/v3/compiler/src/bin/regen_tokenize.rs:112, src/v3/compiler/src/bin/regen_tokenize.rs:205) to emit the generated ScannerCharClass enum and byte_matches helper (src/v3/compiler/src/tokenize_generated.rs:5-28). The generated tokenizer now scans whitespace/digits/ident starts/ident continues in that declared order (src/v3/compiler/src/bin/regen_tokenize.rs:833-911), and the old module is removed from the lib surface and SG-0 census. The PR also adds a crate-local ASCII predicate parity test (src/v3/compiler/src/lib.rs:1645-1677), refreshes the parse manifest for the changed tokenize.dag, and adds documentation around shallow reflection evidence plus a sequenced B4 file/name bridge-retirement queue.

2. Invariant categories

1. LAYER MODEL (substrate vs implementation).

Compliant — this does touch a .dag compiler authority, but not core Dag substrate shape; the new declared fact lives in tokenize.dag as data ascii_scan_order: List<CharClass> = [Whitespace, Digit, IdentStart, IdentContinue] at src/v3/compiler/tokenize.dag:99, and the implementation consumes that row through collect_ascii_scan_order rather than introducing another runtime source of scan order at src/v3/compiler/src/bin/regen_tokenize.rs:205.

1. INVARIANTS.md + modeling-discipline.md.

Finding (NON-BLOCKING) — single-authority / tracked bridge clarity. src/v3/compiler/src/bin/regen_tokenize.rs:189 adds fn ascii_scan_class_predicate(class_name: &str) -> &'static str, and src/v3/compiler/src/bin/regen_tokenize.rs:191-199 still hard-code the Rust predicate bodies for Whitespace, Digit, IdentStart, and IdentContinue. That means predicate semantics are still mirrored in hand-written Rust generator logic even though src/v3/compiler/tokenize.dag:99 now makes the scanner row a List<CharClass>. This is fine as an interim implementation bridge, but it should be named as such; otherwise the PR can be read as full structural consumption of CharClass/char_in_class when it currently only makes the scan order structural.

1. CODING.md.

Compliant — the new implementation follows the repo’s data + free-functions style: emit_char_scanner_class_scaffolding, ascii_scan_class_predicate, and collect_ascii_scan_order are free helpers at src/v3/compiler/src/bin/regen_tokenize.rs:155, src/v3/compiler/src/bin/regen_tokenize.rs:189, and src/v3/compiler/src/bin/regen_tokenize.rs:205, not methods or hidden state. The regen path fails during generation on malformed authority rows rather than silently defaulting, e.g. missing ascii_scan_order at src/v3/compiler/src/bin/regen_tokenize.rs:210 and non-list bodies at src/v3/compiler/src/bin/regen_tokenize.rs:225-226.

1. TESTING.md.

Compliant — the added test is hermetic and behavior-focused for the predicate surface: ascii_byte_class_predicates_match_std_unicode_ascii_boundary iterates 0u8..=127 at src/v3/compiler/src/lib.rs:1650-1651 and checks all exported scanner predicates against ASCII boundary behavior at src/v3/compiler/src/lib.rs:1652-1673. The parse manifest is also refreshed for the changed tokenizer authority at src/v3/compiler/tests/integration/parse_corpus_manifest.txt:19.

1. LOCKED DESIGN DECISIONS.

N/A — the diff adds reviewer-facing documentation and a roadmap queue, but I do not see it referencing or altering a design/thesis item marked locked.

1. TRACKED vs UNTRACKED DEBT.

Finding (NON-BLOCKING) — bridge moved without carrying its dissolution label. The old explicit scaffold file is gone from the SG-0 census, but the predicate mirror survives inside ascii_scan_class_predicate at src/v3/compiler/src/bin/regen_tokenize.rs:189-201. The new ROADMAP.md entry is a well-shaped tracked bridge for file/name identity debt, with concrete dissolution steps at ROADMAP.md:478-484, but it does not bound or dissolve this tokenizer predicate mirror. A one-line comment or roadmap reference on ascii_scan_class_predicate naming the char_in_class interpreter/parity dissolution trigger would keep this from becoming untracked debt.

3. Verdict

APPROVE_WITH_COMMENTS

The main code path improves scan-order authority by deleting the separate helper file and driving generated tokenizer dispatch from tokenize.dag. The remaining issue is not behavioral breakage; it is debt hygiene around the still-hard-coded predicate mirror in the regen binary, which should keep an explicit dissolution marker so this PR does not accidentally present partial structural consumption as full single authority.

[briansrls]

Review metadata

• Provider / model: claude / claude-opus-4-7
• Commit: 401e989a · Trigger: schedule
• Thinking: 44s wall

---

Findings

• Dead IdentContinue scanner arm in generated tokenizer — src/v3/compiler/src/tokenize_generated.rs:147-175 (and the corresponding emitter branch in bin/regen_tokenize.rs:875-896). Because Digit and IdentStart precede IdentContinue in ascii_scan_order and together cover all of [0-9A-Za-z_], the standalone IdentContinue leading-byte branch is unreachable. Mechanically dispatching every class in the order list as a token-producing arm conflates a continuation predicate with a starting class. Non-blocking, but worth either (a) dropping IdentContinue from ascii_scan_order since it's only used as a continuation predicate, or (b) teaching the emitter that continuation-only classes don't get a leading-byte arm. As written it bakes a bug-shaped scaffold into generated output.

• "Structurally consumed" claim is partly aspirational — ascii_scan_order is read from the dag, but ascii_scan_class_predicate (bin/regen_tokenize.rs:209-222) and the per-class scanner-arm if/else if ladder (:832-908) hardcode each variant name, its predicate, and its scanning behavior; any new variant panics. The tokenize.dag:55-57 comment ("ascii_scan_order … is consumed directly by regen_tokenize") oversells this — order is data-driven, semantics are still hand-mirrored. The new INVARIANTS §"Reflection evidence is not structural proof" and the B4 ROADMAP queue both flag this style of bridge, but the comment in tokenize.dag should match (treat as a bounded interim, like the prior tokenize_char_class.rs note it replaced). Non-blocking.

Verdict

APPROVE_WITH_COMMENTS — net dissolution: tokenize_char_class.rs retires into generated output, ASCII predicates collapse onto is_ascii_* helpers, and an ASCII parity test pins behavior. The dead IdentContinue leading-byte arm should be cleaned up (either in this PR or as a follow-up) since it ships in checked-in generated code, and the new tokenize.dag framing comment should match the candor of the new INVARIANTS section about how structural this dispatch actually is.

[briansrls]

Review metadata

• Provider / model: codex / codex-default
• Commit: fd9cb715 · Trigger: schedule
• Thinking: 48s wall

---

Verdict: APPROVE

The diff looks narrowly scoped and consistent with the pinned invariants: ascii_scan_order moves scanner precedence into .dag data, the remaining Rust predicate bridge is documented and bounded, and the deleted tokenize_char_class.rs mirror reduces duplicate hand-authored surface.

I could not run the targeted test because cargo is not installed in this container (cargo: command not found).

[briansrls]

Pushed the v3 manifest fix from the manager branch to session/deep-moth-845 at a6ec96a9b. The failing job expected src/v3/compiler/tokenize.dag\t29\t59946\t0a087425b90a125a; the branch had the stale 330ebc6371e9a03c tuple. Fresh CI should now validate the same manifest snapshot that the failed run printed as left/current.\n\nLocal verification available here: git diff --check passed. Cargo is still unavailable in this shell, so CI remains the Rust/v3 verification authority.

[briansrls]

Addressed the duplicate-draft feedback from #1015 on this active PR:

• PR body now names the charclass-phase-2 brief scope and the deleted tokenize_char_class.rs mirror receipt.
• SG-0 census receipt is explicit: EXPECTED_HAND_AUTHORED_NON_TEST went 36 -> 35 entries (-1 path; the diff also removes the two stale comment lines, so the file delta is -3).
• DB-8 / fixed-point disposition is explicit: local Cargo was unavailable here; fresh CI on a6ec96a9b is the verification authority, and self_host_ratchet is pending behind the current fmt / ci / v3 run.

#1015 and #1018 are duplicate dashboard drafts of the same work and are closed; #1002 is the single active review surface.

[briansrls]

Review metadata

• Provider / model: claude / claude-opus-4-7
• Commit: a6ec96a9 · Trigger: schedule
• Thinking: 40s wall

---

Findings: none blocking.

Verdict: APPROVE — Replaces the handwritten tokenize_char_class.rs mirror by promoting scanner order to a structural data ascii_scan_order: List<CharClass> row in tokenize.dag, and regen_tokenize consumes it (collect_ascii_scan_order validates it against the CharClass disj). Predicate bodies remain a bridge, and the diff documents that explicitly in tokenize.dag:55-58 and INVARIANTS.md (named dissolution trigger: structural char_in_class consumption). The parity test in lib.rs:1646-1675 ratchets ASCII-boundary equivalence. Net file/LOC change is dissolution-shaped (deletes tokenize_char_class.rs).

Exploratory observations:

• tokenize_generated.rs:144-170 — with scan order [Whitespace, Digit, IdentStart, IdentContinue], the IdentContinue dispatch arm is effectively dead (digits caught by Digit, letters/_ by IdentStart; no other byte matches IdentContinue). Not a correctness bug, but the generator emits a redundant arm for every class in ascii_scan_order. If ascii_scan_order is meant to express priority over a set, the current generator conflates the two: the list ordering matters for dispatch but the IdentContinue entry is only meaningful as a referenced predicate, not as its own dispatch start. Worth tightening when the predicate-body bridge dissolves.
• regen_tokenize.rs:236-258 (ascii_scan_class_predicate) is the hand-encoded twin of std.unicode::char_in_class — exactly the bridge the comment names. Fine as tracked debt.

[briansrls]

Merged current origin/main into the #1002 branch at b7d79167f to clear the DIRTY merge state after #1014 landed.

Conflict was isolated to src/v3/compiler/src/lib.rs: kept main’s lower-helper bridge retirement and preserved this PR’s tokenizer ASCII parity test. git diff --check passes locally. Fresh CI/ratchet is now queued on the merged head.

[briansrls]

Review metadata

• Provider / model: codex / unknown
• Commit: a6ec96a9 · Trigger: schedule
• Thinking: 51s wall

---

⚠️ Review not completed: gh pr diff 1002 --repo gunb-ai/gunbc failed to connect to api.github.com, and the instructions forbid using local git output as the PR diff source of truth.