docs(tokenize): reframe character-level scaffold as consumption gap

[briansrls]

Summary

Discovery pass found that the character-level authorities tokenize.dag would consume already exist in dsl/std/ and are imported cross-tree today (src/v3/compiler/regen.dag:3 imports std.types):

• std.types::Char = Int — Unicode scalar, U+0000–U+10FFFF
• std.string_type::String = FreeMonoid<Char>
• std.unicode — DisplayWidth, UnicodeBlock, block/width classification
• std.encoding — ASCII | UTF8 | Latin1 | Text | Binary | Unknown lattice with ASCII <: UTF8 <: Text (literally the ASCII→Unicode causal chain)
• std.bit::Byte

The gap is not substrate — it is consumption. tokenize.dag encodes reserved individual codepoints as opaque strings (StringEscapeSpec fields, LocalPunctSpec.pattern, string_literal_delimiter, line_comment_prefix) and reserved codepoint classes as hidden Rust predicates (is_ascii_whitespace / is_ascii_digit / is_ascii_alphabetic / is_ascii_alphanumeric) push_str'd into regen_tokenize.rs, rather than consuming the canonical Char / std.unicode authorities. Parallel misuse in dsl/extdeps/languages/dag/syntax.dag (OperatorSpec.symbol, dag_keyword_set keys).

The only missing substrate piece is a CharClass taxonomy for tokenization (Whitespace | Digit | IdentStart | IdentContinue), with natural home as a sibling to DisplayWidth in std.unicode.

What this PR changes

Docs-only. Two edits:

• src/v3/compiler/tokenize.dag header — reframes the character-level scaffold note as a consumption gap. Enumerates the two parallel non-canonical forms (opaque-string codepoints, hidden Rust class predicates), names the existing authorities to consume, names the single missing substrate delta (CharClass taxonomy), and carves the three-step follow-up lane.
• ROADMAP.md post-merge debt (2026-04-22) — matching debt row pointing at the scaffold note, naming the existing authorities, enumerating the misuse points in both tokenize.dag and syntax.dag, and listing the follow-up lane steps (add CharClass + predicates; retype opaque-string fields; rewire regen_tokenize).

No structural rewrite in this PR. The follow-up lane is marked ready-to-dispatch in ROADMAP.

Test plan

• grep for existing std.types::Char and std.unicode references confirms the authorities exist and are imported cross-tree
• Scaffold note + ROADMAP row are self-consistent (both name the same authorities, misuse points, and dissolution steps)
• No code changes — no test run needed

🤖 Generated with Claude Code

[briansrls]

Review metadata

• Provider / model: claude / claude-opus-4-7
• Commit: 5b99ea5e · Trigger: schedule
• Thinking: 27s wall

---

Verdict: APPROVE — Docs-only PR that labels an existing borrow (ASCII character-class predicates hardcoded in regen_tokenize.rs:700,727,750,752) as a tracked scaffold. The entry satisfies all three tracked-debt properties: documented (ROADMAP + tokenize.dag header), bounded (four specific ASCII classes + _), and has a named dissolution trigger (structural CharClassInterface in syntax.dag or a borrowed-predicate carrier in src/v3/std/). Cited file:line references all verify against the current tree. Nothing in the diff violates the rubric.

[briansrls]

Review metadata

• Provider / model: codex / gpt-5.4
• Commit: 5b99ea5e · Trigger: schedule
• Thinking: 41s wall

---

Verdict: APPROVE

Doc-only diff, and it stays honest against the live code. The new scaffold note in src/v3/compiler/tokenize.dag matches the existing hardcoded ASCII predicate borrow in src/v3/compiler/src/bin/regen_tokenize.rs, and the new ROADMAP.md row is documented, bounded, and names a concrete dissolution trigger, so I don't see a violation of the pinned invariants, coding discipline, or testing guidance in this PR.

[briansrls]

Review metadata

• Provider / model: codex / gpt-5.4
• Commit: 5b99ea5e · Trigger: schedule
• Thinking: 212s wall

---

Non-blocking — Strengths

• ROADMAP.md This is real tracked debt under P5: the new row documents the missing authority, bounds it to the tokenizer's borrowed ASCII classes, and names a concrete dissolution trigger.
• src/v3/compiler/tokenize.dag Mirroring the receipt at the authority site is the right shape because the borrow is now visible to readers of the tokenizer spec instead of living only in roadmap prose.

ROADMAP — Verified

• Character-class borrow in tokenize: The cited hardcoded predicates are present in regen_tokenize.rs, and dsl/extdeps/languages/dag/syntax.dag currently exposes keywords/operators but no structural character-class carrier.

✅ This looks clean; it makes an implicit tokenizer borrow explicit without changing behavior and the scaffold is tracked the way INVARIANTS.md expects.

[briansrls]

Review metadata

• Provider / model: claude / claude-opus-4-7
• Commit: 62963775 · Trigger: schedule
• Thinking: 28s wall

---

Verdict: APPROVE — docs-only change. The scaffold note is tracked correctly: documented (both in tokenize.dag:22-68 header and ROADMAP.md:202), bounded (enumerates the specific opaque-string fields in tokenize.dag/syntax.dag and the hidden Rust predicates in regen_tokenize.rs), and has a named three-step dissolution trigger (add CharClass to std.unicode, retype fields to consume Char/List<Char>, rewire regen_tokenize). Reframing from "substrate gap" to "consumption gap" is accurate — std.types::Char, std.string_type::String = FreeMonoid<Char>, std.unicode, and std.encoding do already exist. Manifest hash refresh in parse_corpus_manifest.txt:20 matches the only changed file (node-count 33 and serialized-size 95348 unchanged, content hash updated — consistent with a header-comment-only edit). No invariant, modeling, coding, or testing discipline violations in the diff.

[briansrls]

Review metadata

• Provider / model: codex / gpt-5.4
• Commit: 62963775 · Trigger: schedule
• Thinking: 86s wall

---

APPROVE

Diff is docs-only, and the new tokenize.dag scaffold note is documented, bounded, and carries a concrete dissolution trigger rather than normalizing the debt. The manifest refresh is consistent with a header-comment-only change, and I don’t see a concrete violation of the pinned invariants, coding, or testing docs in this diff.