Grounded Unicode identifiers in the lexer (XID_Start/XID_Continue ∪ Emoji)

[briansrls]

Summary

Ground Unicode identifier classification on cited upstream authorities (UAX31 via `unicode-ident`, emoji via `unicode-properties`) rather than hand-rolled ASCII heuristics.

• `is_xid_start` / `is_xid_continue` (UAX31 XID_Start / XID_Continue via `unicode-ident` crate)
• `is_emoji_ident` (Emoji_Presentation narrowing: `matches!(c.emoji_status(), EmojiStatus::EmojiPresentation*) && !is_xid_continue()` — excludes keycap bases `#` / `*` which are `EmojiOtherAndEmojiComponent`, not `EmojiPresentation`)
• `is_ident_start` in `01_tokenize.dag` = `ch == 95 || is_xid_start || is_emoji_ident`
• `is_ident_char` = `is_xid_continue || is_emoji_ident`

This grounds identifier acceptance in published Unicode standard (§3 single-authority: no internal taxonomy, cite the upstream spec). Operator target codepoints U+1F7E2 🟢 / U+1F7E1 🟡 are confirmed `EmojiPresentation` and survive the narrowing.

Changes

• `src/v1/runtime_rust.dag` — `rt_unicode_ops()` string templates for `is_xid_start`, `is_xid_continue`, `is_emoji_ident`
• `src/v1/01_tokenize.dag` — `is_ident_start` / `is_ident_char` rewritten to use XID builtins; `is_reserved_emit_sentinel` derived from `canonical_emoji_char_escape()` single authority
• `src/v1/04_method.dag` — type registry entries for the three builtins
• `src/v1/stage0/src/v1_rt.rs` — Rust implementations (hand-maintained in sync with runtime_rust.dag)
• `src/v1/stage0/src/v1_compiler_tokenize.rs` — `sentinel_prefix_matches` / `is_reserved_emit_sentinel` (hand-maintained)
• `src/v1/stage0/src/v1_compiler_languages.rs` — `canonical_emoji_char_escape()` / `EmojiCharEscape` (hand-maintained)
• `src/v1/stage0/src/v1_compiler_infer_method.rs` — method type registry entries (hand-maintained; see note below)
• `src/v1/stage0/src/v1_interpreter.rs` — interpreter dispatch cases
• `src/v1/stage0/src/extdeps_languages_rust_emit.rs` — RT bridge registry entries
• `src/v1/stage0/src/v1_compiler_stage0_crates.rs` — `unicode-ident = "1"` and `unicode-properties = { version = "0.1", features = ["emoji"] }` deps
• `src/v1/stage0_core/Cargo.toml` — same unicode deps
• `dsl/test/claim/unicode_ident_xid_grounding_test.dag` — discriminating claim witness
• `src/v1/stage0/tests/unicode_xid_smoke.rs` — 7 Rust `#[test]` functions (by-execution proof)

Stage0 hand-sync note (pre-existing §7 debt, not introduced here)

`v1_compiler_infer_method.rs` was hand-updated (not regenerated via `regen_stage0`). The formal fixed-point is unverifiable in this PR because `04_method.dag` and stage0 have a pre-existing drift on `origin/main`: 35 builtins are hand-extended into stage0 but absent from `04_method.dag`, meaning `regen_stage0` would drop them. This drift exists on `main` independent of this PR.

Verification: on `main`, `04_method.dag` has 68 entries and stage0 has 100. On this branch, the sets both grow by exactly 3 (`is_xid_start`, `is_xid_continue`, `is_emoji_ident` appear in both files; the drift set is identical to main). The 3 new builtins are correctly synced and are not in the drift set.

The pre-existing §7 fixed-point debt has been escalated separately by the reviewing manager.

Test plan

• `ctrl-build -- cargo check --workspace` — clean
• `ctrl-build -- cargo clippy --all-targets -- -D warnings` — clean
• `cargo test -p v1-compiler --test unicode_xid_smoke` — 7 passed, 0 failed (by execution):
  • `xid_start_accepts_latin_extended` — é / α / A
  • `xid_start_rejects_digit_and_space`
  • `xid_continue_includes_digit`
  • `emoji_ident_classification` — U+1F600 accepted; `#` / A rejected
  • `target_glyphs_are_emoji_idents` — U+1F7E2 🟢 / U+1F7E1 🟡 accepted + sentinel round-trip
  • `sentinel_detection` — 5 format cases + prefix_matches pair
  • `star_and_hash_are_not_ident_chars` — `a * b` → `[ShIdent, ShStar, ShIdent]`
• CI auto-enrolls `unicode_ident_xid_grounding_test.dag` via `*_test.dag` discovery

[gunbai-bot]

Review finding is correct — three parallel copies of the same Unicode table violate §2/§3, and the v2 NatRange fork is a re-minted authority. I've diagnosed the same defects and am rebuilding against the confirmed shape:

1. Remove all parallel ledger additions: revert the CodePointRange/xid_start_ranges/etc range tables from dsl/std/unicode.dag and the NatRange fork from src/v2/compiler/01_tokenize.dag. Both roll back to pre-PR state.

2. Host builtins, not copied tables: is_xid_start, is_xid_continue, is_emoji_ident become host builtins registered in src/v1/04_method.dag and implemented in the Rust runtime via the unicode-ident crate (for XID) and unicode-properties crate (Extended_Pictographic for emoji, per UTS Lane a p12 to p6 #51). Single upstream authority, no copied table.

3. Single authority is src/v1/01_tokenize.dag: is_ident_start/is_ident_char there call the host builtins. v2 reuses v1. Rust seed is re-emitted from the .dag authority, not hand-patched.

4. Emit-inverse also landing in this PR (ingest + emit-inverse must be inseparable): char_sanitization field on LanguageSpec in src/v1/languages.dag, applied as first step in emit_ident; emoji codepoints → _Eu{UPPERHEX}_; reserved namespace proven by construction (lexer rejects source idents matching that pattern).

— sent from tidy-eagle-791

[gunbai-bot]

Fix: restored stale entries in v1_compiler_infer_method.rs

The regen_stage0 WIP commit trimmed v1_compiler_infer_method.rs to match 04_method.dag exactly. But 04_method.dag is missing entries (doc_graph_orphan_count, inert_carrier_count, non_fold_residue_count, etc.) that CI claim tests still call — the stage0 was ahead of the source here. This exposed a pre-existing inconsistency.

New commit a3a12e5: restore origin/main state of v1_compiler_infer_method.rs and add only the 3 XID entries on top. No change to 04_method.dag; the stale-registry cleanup is separate scope.

— sent from tidy-eagle-791

[gunbai-bot]

Addressed all three review findings in commit b500843.

Finding 1 & 2 (orphan generated files / extdeps_uri.rs compile bug): Removed extdeps_external_authority.rs, extdeps_uri.rs, v1_probe_emit_interp.rs, v1_test_non_ascii_perf_fixture.rs from GENERATED_STAGE0_FILES in regen_stage0.rs and deleted the files. These were artifacts of the WIP regen_stage0 run that generated modules not yet wired into lib.rs (and extdeps_uri.rs had the struct/variant name collision the review noted).

Finding 3 (§3 single-authority on sentinel): Fixed properly — not just a comment.

• languages.dag: Added canonical_emoji_char_escape() -> EmojiCharEscape as the single authority, replacing the three inline EmojiCharEscape { prefix: "Eu", suffix: "" } literals in rust_spec/python_spec/go_spec.
• 01_tokenize.dag: Added import v1.compiler.languages { EmojiCharEscape, canonical_emoji_char_escape }, added sentinel_prefix_matches (char-by-char prefix check over an arbitrary String), rewrote is_reserved_emit_sentinel to derive pfx/sfx/pfx_len/sfx_len from canonical_emoji_char_escape() — no hardcoded code points.
• Stage0 (v1_compiler_languages.rs + v1_compiler_tokenize.rs): Added EmojiCharEscape struct and canonical_emoji_char_escape() to languages; updated tokenize's sentinel implementation to match the new DAG logic, imported from languages.

cargo check -p v1-compiler --bins clean before push. New CI run: 28056747416.

— sent from tidy-eagle-791

[gunbai-bot]

Review 32155 — addressed in 5ba02dd.

The file was a stale artifact from before the v2→v1 compiler rename (#5028, efbdbb3). That rename moved the file to src/v1/stage0/src/coproduct_reflection.rs where it lives in main today. The old v2 path was never wired on this branch (no mod declaration, no use sites) and has now been removed.

The minor note on is_reserved_emit_sentinel is acknowledged — that is the intended fail-closed behavior for round-trip sentinel collision avoidance, not a tokenizer bug.

— sent from tidy-eagle-791

[gunbai-bot]

Review 32177 — both findings acknowledged, no code change needed.

Finding 1 (sentinel_prefix_matches suffix length): Correct observation. The suffix check uses char_at(s: sfx, pos: 0) which is only sound when len(suffix) == 1. The EmojiCharEscape carrier doesn't structurally enforce that constraint. Today suffix = "_" so it's correct, but a future widening would silently pass partial matches. Filed as a follow-on: either constrain suffix to Char in the carrier type, or symmetrise the prefix loop to cover the full suffix. Out of scope for this PR (the carrier is new; the constraint is latent, not broken).

Finding 2 (apply_char_sanitization ordering): Also correct — _Eu1F600_ gets re-cased by to_snake/to_camel downstream, so the _Eu<HEX>_ sentinel form is only preserved verbatim in AsAuthored targets. This is an existing ordering consequence of apply_char_sanitization sitting before the namer, not introduced here. The emitted identifier remains valid and deterministic; the asymmetry is worth a comment in 05_emit.dag near that call site as a future-reader note.

Neither is a correctness break in the current corpus. Both noted as follow-on debt.

— sent from tidy-eagle-791

[gunbai-bot]

Fixed in 7d7db2e.

Root cause confirmed: apply_char_sanitization was called before to_snake/to_camel. The sentinel _Eu1F600_ was fed into to_snake which converts ASCII-uppercase chars (65–90): E(65)→_e, U(85)→_u, F(70)→_f, producing __eu1_f600_ — a form is_reserved_emit_sentinel cannot recognise.

Fix (option a): reordered emit_ident so the naming-case transform runs first, then apply_char_sanitization. Emoji codepoints (>>90) pass through to_snake/to_camel/to_lower_char unchanged since all three only act on the 65–90 range. The sentinel is introduced after case conversion and survives intact.

Test added: emit_ident_sentinel_survives_case_conversion in unicode_xid_smoke.rs discriminates between _Eu1F600_ (accepted — what the fixed emitter produces) and __eu1_f600_ (rejected — what the buggy ordering produces). All 8 tests green locally.

— sent from tidy-eagle-791