diff --git a/INVARIANTS.md b/INVARIANTS.md index 63b8e3c29f3..2827200ed4b 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -150,6 +150,13 @@ A downstream stage reads a lower layer not through its declared accessors but by - **E-9: External Realization Lives On Arrow.body** — declares the single authority for external semantics - **DB-14: Substrate External Primitives Materialize Through Declared Arrow.body Plus Target Bindings** — same single-authority principle for external primitives +### Reflection evidence is not structural proof + +`reflect_program_dag_nodes_in_file` in `lens_apply.rs` currently emits a **shallow behavior spine** (`result_port` plus limited tags) and intentionally drops structural fields (`target`, `inputs`, `input`, `paths`, `source`, `init`, `body`, `bound`, etc.). Consumers that rely on this view — in practice `LensOutputEquals` and `AlgebraicLaw` runners in `test_runner.rs` — therefore only gain **regression evidence** from the current reflection path, not structural self-inspection parity with +`src/v3/std/substrate.dag`. + +**Confidence tag for these gates:** treat these paths as `StructuralEvidence::Shallow` until full substrate reflection is landed. PRs using them should state that status explicitly in their acceptance notes and route dissolution via the dedicated lossy-reflection closure row (`ROADMAP.md`), not by claiming by-construction proof. + --- ## P3: Fail-Closed diff --git a/ROADMAP.md b/ROADMAP.md index 7af55775833..110b88ac6b5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -475,6 +475,14 @@ Three distinct reflective/exploratory analyses ran against `main@7f74f09` and `m - **Filename / sentinel bridges in `test_runner.rs`** (NOVEL — class-of-pattern row): `src/v3/compiler/src/test_runner.rs` carries multiple file-path / fixture-name / sentinel bridges as identity references: fixture filename → bind name mapping; `PROGRAM_INPUT_SENTINEL = "r1_lens_output_input_from_program"`; `include_str!("../lenses/named_function_count.dag")` as canonical lens side-channel; `EXTDEPS_BOOTSTRAP_FIXTURES` manually listing currently-loadable extdeps surface; `fold_site_skips_d1_monomorph_list_fold_path` using a path suffix. Each instance is locally explainable; together they show **the same missing substrate fact: identity and role references not yet carried structurally enough for consumers**. **Dissolution trigger**: replace fixture-filename routing + `PROGRAM_INPUT_SENTINEL` with structural references (`DeclarationRef`; explicit input value carriers; bind/output identity in the claim itself; resource/materialization records). Surfaced by Reflective Analysis 2026-04-25 (Pattern 3 + Risk 4) + Exploratory 2026-04-25. P2 Boundary Discipline (string/name/path as temporary identity). Owner: unassigned; M scope. +- **B4 bridge-retirement queue: file/name bridge class (sequenced)** (NOVEL + consolidation): retire file/name bridge sites in a single identity-carrier queue under `docs/briefs/b4-identity-carrier-substrate-pass.md`: + 1. **`SourceSpan.file` participation checks** (`lens_apply.rs:340-344`, `lower.rs:848`, `lower.rs:1637-1738`, `lower.rs:1954+`, `emit.rs:1082-1094`, `dag.rs:2914`): dissolve via structural fold-shape / emit-helper carrier landing and migration. + 2. **Canonical lens-name dispatch / fixture-name routing** (`test_runner.rs`): consume `DeclarationRef` and claim-role carriers for lens/input identity; delete `PROGRAM_INPUT_SENTINEL` and fixture-name bridges. + 3. **`include_str!` side channels** (`test_runner.rs:23`, `:33`): dissolve in favor of canonical declaration references once `apply_lens_declaration` uses same-id carriers. + 4. **Exact-string patching in generated helper code** (`lib.rs::patch_lower_helpers_generated_type_alias_refinement`): remove once generated helper output carries refinement fields natively and bridge-patch is no longer needed. + +**Sequencing:** this queue is an owner-scoped B4 sub-brief slice; each item follows `B4.1` (`DeclarationRef`) then the relevant carrier-landing rows in B4 order, and records closure only when the previous row proves its successor consumer is structurally routed. This is not a new program, just a single sequence inside the existing B4 identity-carrier lane. Surfaced by 5c bridge sweep request. P2 Boundary Discipline + P5 Progress Is Dissolution. Owner: unassigned; M scope. + ### Reviewer-noise class — a practice, not a debt - **Integration-reflection cadence**: every ~few days, run a reflective + exploratory analysis pair. The 2026-04-15 and 2026-04-18 passes caught items individual PR reviews missed (authority split across PRs, silent cross-PR name-based lookups, the `CreateComment` drift, the `repeat_string` bug). Worth institutionalizing. **Velocity-tripwire reporting (added 2026-04-25 per [PR #810](https://github.com/gunb-ai/gunbc/pull/810) §4)**: each cadence pass also reports introduction:dissolution PR ratio for the window. Heuristic match: PR titles containing `retire`/`dissolve`/`delete`/`remove`/`collapse` count as dissolution; substantial-scaffold-introducing PRs (new hand-Rust file, new sentinel, new Rust mirror of a `.dag` authority) count as introduction. **Calibration caveat**: this is a lower-bound heuristic — feature PRs that ship dissolution work in the same diff (e.g., a T-LensAPI PR adding a lens and retiring three legacy paths) are undercounted. Before a ≥3:1 reading triggers Director review (per INVARIANTS.md §P5 "Dispatch-Discipline Mechanisms" (c)), the cadence pass MUST do a manual sweep for dissolution-bearing feature PRs in the window to avoid false positives. Current 2026-04-18→2026-04-25 baseline: ~1.6:1 (8 dissolution-shaped + 13 substantial introductions; lower bound). diff --git a/src/v3/compiler/src/bin/regen_tokenize.rs b/src/v3/compiler/src/bin/regen_tokenize.rs index 08b196e6b27..7a846fb3add 100644 --- a/src/v3/compiler/src/bin/regen_tokenize.rs +++ b/src/v3/compiler/src/bin/regen_tokenize.rs @@ -109,6 +109,7 @@ fn read_shared_syntax_source(manifest_dir: &std::path::Path) -> String { fn generate(dag: &Dag, shared_syntax: &SharedSyntaxAuthority) -> String { let keywords = collect_keyword_rows(dag, shared_syntax); + let ascii_scan_order = collect_ascii_scan_order(dag); let puncts = collect_punct_rows(dag, shared_syntax); let line_comment_prefix = string_data_named(dag, "line_comment_prefix"); let string_delim = string_data_named(dag, "string_literal_delimiter"); @@ -125,7 +126,7 @@ fn generate(dag: &Dag, shared_syntax: &SharedSyntaxAuthority) -> String { let mut out = String::new(); out.push_str("use crate::diagnostics::{Diagnostic, SourceSpan};\n"); - out.push_str("use crate::tokenize_char_class::{byte_matches, TokenizerCharClass};\n\n"); + out.push_str(&emit_char_scanner_class_scaffolding(&ascii_scan_order)); out.push_str(&emit_token_kind_enum(dag)); out.push_str( r#"#[derive(Debug, Clone)] @@ -145,11 +146,120 @@ pub struct Token { &diag_int_pre, &diag_int_suf, &escapes, + &ascii_scan_order, )); out.push_str(&emit_punctuation_token(&puncts)); out } +fn emit_char_scanner_class_scaffolding(scan_order: &[String]) -> String { + assert!( + scan_order.len() == 4, + "`ascii_scan_order` in `tokenize.dag` must list exactly 4 class names" + ); + assert!( + scan_order + .iter() + .collect::>() + .len() + == scan_order.len(), + "`ascii_scan_order` in `tokenize.dag` must not contain duplicates" + ); + + let mut out = String::new(); + out.push_str("\n#[derive(Debug, Clone, Copy, PartialEq, Eq)]\n"); + out.push_str("pub(crate) enum ScannerCharClass {\n"); + for class in scan_order { + out.push_str(&format!(" {class},\n")); + } + out.push_str("}\n\n"); + + out.push_str("#[inline]\n"); + out.push_str("pub(crate) fn byte_matches(byte: u8, class: ScannerCharClass) -> bool {\n"); + out.push_str(" match class {\n"); + for class in scan_order { + let expr = ascii_scan_class_predicate(class); + out.push_str(&format!(" ScannerCharClass::{class} => {expr},\n")); + } + out.push_str(" }\n}\n\n"); + + out +} + +fn ascii_scan_class_predicate(class_name: &str) -> &'static str { + // Interim bridge: `ascii_scan_order` supplies structural scanner order, but + // class predicate bodies remain here until `std.unicode::char_in_class` is + // structurally consumable by the tokenizer generator. + match class_name { + "Whitespace" => "matches!(byte, b'\\t' | b'\\n' | b'\\x0c' | b'\\r' | b' ')", + "Digit" => "byte.is_ascii_digit()", + "IdentStart" => "byte.is_ascii_lowercase() || byte.is_ascii_uppercase() || byte == 0x5f", + "IdentContinue" => "byte.is_ascii_alphanumeric() || byte == 0x5f", + _ => panic!("unsupported scanner class `{class_name}` in `ascii_scan_order`"), + } +} + +fn collect_ascii_scan_order(dag: &Dag) -> Vec { + let scan_decl = dag + .declarations() + .iter() + .find(|d| d.name.as_deref() == Some("ascii_scan_order")) + .unwrap_or_else(|| panic!("missing `ascii_scan_order` in `{TOKENIZE_AUTHORITY_FILE}`")); + + let char_class_decl = dag + .declarations() + .iter() + .find(|d| d.name.as_deref() == Some("CharClass")) + .unwrap_or_else(|| panic!("missing `CharClass` in `{TOKENIZE_AUTHORITY_FILE}`")); + + let TypeConnective::Disj { + variants: char_class_variants, + } = &char_class_decl.connective + else { + panic!("`CharClass` should be a disj declaration"); + }; + + let Some(ValueBody::List(values)) = scan_decl.value_body.as_ref() else { + panic!("`ascii_scan_order` in `{TOKENIZE_AUTHORITY_FILE}` must be a list"); + }; + + let mut out = Vec::new(); + for value in values { + let FieldValue::Variant { + constructor, + payload, + } = value + else { + panic!("`ascii_scan_order` elements must be constructor values"); + }; + assert!( + payload.is_empty(), + "`ascii_scan_order` class entries must be nullary constructors" + ); + let label = char_class_variants + .iter() + .find(|field| field.ty == *constructor) + .map(|field| field.label.clone()) + .unwrap_or_else(|| { + panic!( + "`ascii_scan_order` contains constructor {:?} not owned by `CharClass`", + constructor + ) + }); + out.push(label); + } + + let expected = ["Whitespace", "Digit", "IdentStart", "IdentContinue"]; + for class in &expected { + assert!( + out.contains(&class.to_string()), + "`ascii_scan_order` in `{TOKENIZE_AUTHORITY_FILE}` must include `{class}`" + ); + } + + out +} + fn assert_shared_syntax_raw_source_scaffold_still_required(shared_syntax_source: &str) { let lowered = match compile_to_dag(shared_syntax_source, SHARED_SYNTAX_FILE) { Ok(dag) => dag, @@ -671,7 +781,18 @@ fn emit_tokenize_fn( diag_int_pre: &str, diag_int_suf: &str, escapes: &[(u8, i64)], + ascii_scan_order: &[String], ) -> String { + let ensure_classes = |required: &[&str]| { + for name in required { + assert!( + ascii_scan_order.iter().any(|class| class == name), + "`ascii_scan_order` in tokenize.dag missing `{name}`" + ); + } + }; + ensure_classes(&["Whitespace", "Digit", "IdentStart", "IdentContinue"]); + let mut arms = String::new(); for (spelling, kind) in keywords { arms.push_str(&format!( @@ -705,10 +826,86 @@ fn emit_tokenize_fn( )); s.push_str(" while pos < bytes.len() {\n"); s.push_str(" let byte = bytes[pos];\n\n"); - s.push_str(" if byte_matches(byte, TokenizerCharClass::Whitespace) {\n"); - s.push_str(" pos += 1;\n"); - s.push_str(" continue;\n"); - s.push_str(" }\n\n"); + s.push_str(" let start = pos;\n\n"); + for class in ascii_scan_order { + if class == "Whitespace" { + s.push_str(" if byte_matches(byte, ScannerCharClass::Whitespace) {\n"); + s.push_str(" pos += 1;\n"); + s.push_str(" continue;\n"); + s.push_str(" }\n\n"); + } else if class == "Digit" { + s.push_str(" if byte_matches(byte, ScannerCharClass::Digit) {\n"); + s.push_str(" let mut end = pos;\n"); + s.push_str( + " while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::Digit) {\n", + ); + s.push_str(" end += 1;\n"); + s.push_str(" }\n"); + s.push_str(" let literal = &source[start..end];\n"); + s.push_str( + " let value: i64 = literal.parse().map_err(|_| Diagnostic::TokenizerError {\n", + ); + s.push_str(&format!( + " message: format!(\"{{}}{{}}{{}}\", {}, literal, {}),\n", + int_pre, int_suf + )); + s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); + s.push_str(" fixes: Vec::new(),\n"); + s.push_str(" })?;\n"); + s.push_str(" tokens.push(Token {\n"); + s.push_str(" kind: TokenKind::IntLit(value),\n"); + s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); + s.push_str(" });\n"); + s.push_str(" pos = end;\n"); + s.push_str(" continue;\n"); + s.push_str(" }\n\n"); + } else if class == "IdentStart" { + s.push_str(" if byte_matches(byte, ScannerCharClass::IdentStart) {\n"); + s.push_str(" let mut end = pos;\n"); + s.push_str( + " while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::IdentContinue) {\n", + ); + s.push_str(" end += 1;\n"); + s.push_str(" }\n"); + s.push_str(" let text = &source[start..end];\n"); + s.push_str(" let kind = match text {\n"); + s.push_str(&arms); + s.push_str(" _ => TokenKind::Ident(text.to_string()),\n"); + s.push_str(" };\n"); + s.push_str(" tokens.push(Token {\n"); + s.push_str(" kind,\n"); + s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); + s.push_str(" });\n"); + s.push_str(" pos = end;\n"); + s.push_str(" continue;\n"); + s.push_str(" }\n\n"); + } else if class == "IdentContinue" { + s.push_str(" if byte_matches(byte, ScannerCharClass::IdentContinue) {\n"); + s.push_str(" let mut end = pos;\n"); + s.push_str( + " while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::IdentContinue) {\n", + ); + s.push_str(" end += 1;\n"); + s.push_str(" }\n"); + s.push_str(" let text = &source[start..end];\n"); + s.push_str(" let kind = match text {\n"); + s.push_str(&arms); + s.push_str(" _ => TokenKind::Ident(text.to_string()),\n"); + s.push_str(" };\n"); + s.push_str(" tokens.push(Token {\n"); + s.push_str(" kind,\n"); + s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); + s.push_str(" });\n"); + s.push_str(" pos = end;\n"); + s.push_str(" continue;\n"); + s.push_str(" }\n\n"); + } else { + panic!( + "unsupported scanner class `{}` in `ascii_scan_order`", + class + ); + } + } s.push_str(" // Line comment prefix from `tokenize.dag` (`line_comment_prefix`).\n"); s.push_str(" if bytes.len() >= pos + LINE_COMMENT_PREFIX.len()\n"); s.push_str( @@ -721,7 +918,6 @@ fn emit_tokenize_fn( s.push_str(" }\n"); s.push_str(" continue;\n"); s.push_str(" }\n\n"); - s.push_str(" let start = pos;\n\n"); s.push_str(" if let Some((kind, width)) = punctuation_token(bytes, pos) {\n"); s.push_str(" tokens.push(Token {\n"); s.push_str(" kind,\n"); @@ -732,50 +928,6 @@ fn emit_tokenize_fn( s.push_str(" pos += width;\n"); s.push_str(" continue;\n"); s.push_str(" }\n\n"); - s.push_str(" if byte_matches(byte, TokenizerCharClass::Digit) {\n"); - s.push_str(" let mut end = pos;\n"); - s.push_str( - " while end < bytes.len() && byte_matches(bytes[end], TokenizerCharClass::Digit) {\n", - ); - s.push_str(" end += 1;\n"); - s.push_str(" }\n"); - s.push_str(" let literal = &source[start..end];\n"); - s.push_str( - " let value: i64 = literal.parse().map_err(|_| Diagnostic::TokenizerError {\n", - ); - s.push_str(&format!( - " message: format!(\"{{}}{{}}{{}}\", {}, literal, {}),\n", - int_pre, int_suf - )); - s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); - s.push_str(" fixes: Vec::new(),\n"); - s.push_str(" })?;\n"); - s.push_str(" tokens.push(Token {\n"); - s.push_str(" kind: TokenKind::IntLit(value),\n"); - s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); - s.push_str(" });\n"); - s.push_str(" pos = end;\n"); - s.push_str(" continue;\n"); - s.push_str(" }\n\n"); - s.push_str(" if byte_matches(byte, TokenizerCharClass::IdentStart) {\n"); - s.push_str(" let mut end = pos;\n"); - s.push_str( - " while end < bytes.len() && byte_matches(bytes[end], TokenizerCharClass::IdentContinue) {\n", - ); - s.push_str(" end += 1;\n"); - s.push_str(" }\n"); - s.push_str(" let text = &source[start..end];\n"); - s.push_str(" let kind = match text {\n"); - s.push_str(&arms); - s.push_str(" _ => TokenKind::Ident(text.to_string()),\n"); - s.push_str(" };\n"); - s.push_str(" tokens.push(Token {\n"); - s.push_str(" kind,\n"); - s.push_str(" span: SourceSpan::new(file, start as u32, end as u32),\n"); - s.push_str(" });\n"); - s.push_str(" pos = end;\n"); - s.push_str(" continue;\n"); - s.push_str(" }\n\n"); s.push_str( " // String literal (`string_literal_delimiter` + `StringEscapeSpec` rows).\n", ); diff --git a/src/v3/compiler/src/lib.rs b/src/v3/compiler/src/lib.rs index cb5e724a191..3508e898ffd 100644 --- a/src/v3/compiler/src/lib.rs +++ b/src/v3/compiler/src/lib.rs @@ -1162,7 +1162,6 @@ mod regen_parse_tables_emit; )] #[path = "tokenize_generated.rs"] mod tokenize; -mod tokenize_char_class; pub use regen_parse_emit::{render_parse_generated_rs, RenderParseGeneratedError}; pub use regen_parse_tables_emit::{ @@ -1642,3 +1641,37 @@ mod lower_helpers_type_alias_refinement_patch_tests { assert_eq!(out, s); } } + +#[cfg(test)] +mod tokenize_ascii_parity_tests { + use super::tokenize::{byte_matches, ScannerCharClass}; + + #[test] + fn ascii_byte_class_predicates_match_std_unicode_ascii_boundary() { + for byte in 0u8..=127 { + assert_eq!( + byte_matches(byte, ScannerCharClass::Whitespace), + byte.is_ascii_whitespace(), + "tokenizer whitespace predicate diverged at byte {byte:#04x}" + ); + + assert_eq!( + byte_matches(byte, ScannerCharClass::Digit), + byte.is_ascii_digit(), + "tokenizer digit predicate diverged at byte {byte:#04x}" + ); + + assert_eq!( + byte_matches(byte, ScannerCharClass::IdentStart), + (byte.is_ascii_alphabetic() || byte == b'_'), + "tokenizer ident-start predicate diverged at byte {byte:#04x}" + ); + + assert_eq!( + byte_matches(byte, ScannerCharClass::IdentContinue), + (byte.is_ascii_alphanumeric() || byte == b'_'), + "tokenizer ident-continue predicate diverged at byte {byte:#04x}" + ); + } + } +} diff --git a/src/v3/compiler/src/tokenize_char_class.rs b/src/v3/compiler/src/tokenize_char_class.rs deleted file mode 100644 index 5e6af5ee284..00000000000 --- a/src/v3/compiler/src/tokenize_char_class.rs +++ /dev/null @@ -1,149 +0,0 @@ -//! ASCII-oriented lexical classes for the SG-1 tokenizer. -//! -//! Authority: `dsl/std/unicode.dag` — `CharClass` + `char_in_class` define the -//! same ASCII scalar predicates. This module is the Rust projection the -//! `regen_tokenize` binary emits into `tokenize_generated.rs` because M1(2.8) -//! still treats list / sum-variant `data` bodies in `tokenize.dag` as opaque -//! (`DOWNSTREAM_REQUIREMENTS.md` class-5 gap #3), so the scanner cannot yet -//! read `CharClass` rows structurally from that authority file. -//! -//! `byte_matches` is **hand-synced** with `char_in_class` in `dsl/std/unicode.dag` -//! on code points U+0000–U+007F (same Int-range semantics). There is no runtime -//! bridge from lowered `.dag` yet: `#[cfg(test)]` checks lock the mirror against -//! Rust’s `u8::is_ascii_*` and use substring anchors on `unicode.dag` source — not -//! evaluated `char_in_class`. That is intentional **bounded debt** under -//! modeling-discipline P2 (single authority / drift risk): the executable truth -//! for predicates is only in `.dag` today. **ROADMAP** row *`char_in_class` -//! interpreter parity (tokenizer bridge finish, PR #693)* tracks replacing the -//! host-predicate ratchet with `0..=127` parity vs evaluated `char_in_class` once -//! the v3 test harness can run it (same gate as deleting this mirror). -//! -//! **Lane framing:** tokenizer-side interim only — not structural consumption of -//! `CharClass` from lowered `tokenize.dag` (see M1(2.8) class-5 gap #3). Remove -//! this module **and** [`TokenizerCharClass`] when `regen_tokenize` reads class -//! predicates from lowered `.dag` — do not leave a duplicate Rust sum of variant -//! names beside the `.dag` authority. - -/// Mirrors `std.unicode::CharClass` variant names for `tokenize_generated.rs` call -/// sites only. **Dissolution:** delete this enum with this module when structural -/// scanner rows land — not just the `byte_matches` body. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum TokenizerCharClass { - Whitespace, - Digit, - IdentStart, - IdentContinue, -} - -#[inline] -pub(crate) fn byte_matches(byte: u8, class: TokenizerCharClass) -> bool { - let cp = byte as i64; - match class { - // Match Rust `u8::is_ascii_whitespace` (excludes vertical tab U+000B). - TokenizerCharClass::Whitespace => { - matches!(byte, b'\t' | b'\n' | b'\x0C' | b'\r' | b' ') - } - TokenizerCharClass::Digit => (48..=57).contains(&cp), - TokenizerCharClass::IdentStart => { - (65..=90).contains(&cp) || (97..=122).contains(&cp) || cp == 95 - } - // Match `unicode.dag` `IdentContinue` arm shape: inline digit range (same - // as `Digit`), then letter/underscore — no `byte_matches` self-call so - // the mirror stays line-aligned with the `.dag` CX choice. - TokenizerCharClass::IdentContinue => { - (48..=57).contains(&cp) - || (65..=90).contains(&cp) - || (97..=122).contains(&cp) - || cp == 95 - } - } -} - -#[cfg(test)] -mod sub_charclass_in_std_unicode_gate { - //! **Layer:** unit (TESTING.md) — T-Sub `sub_charclass_in_std_unicode` tokenizer - //! half / bounded interim (see ROADMAP.md: T-Sub lane + `char_in_class` interpreter - //! parity row). Ratchets `std.unicode` + generated tokenizer wiring + `byte_matches` - //! vs `u8::is_ascii_*` on 0..=127 (behavioral until interpreter parity lands). - //! - //! **Sync boundary:** parity here is `byte_matches` vs `u8::is_ascii_*`, not - //! evaluated `char_in_class` from `.dag`; keep `tokenize_char_class.rs` and - //! `unicode.dag` predicates edited together until an interpreter-backed check exists. - //! Substring anchors below only catch gross drift (missing sum, restored host - //! `is_ascii_*`, accidental reintroduction of `char_in_class` self-call on the - //! same `c` in `IdentContinue`); they do not prove arithmetic matches `.dag` and - //! are fragile to harmless `unicode.dag` reformatting. The `0..=127` - //! `byte_matches` vs `is_ascii_*` loop is the primary behavioral guard until - //! interpreter-backed parity (ROADMAP) replaces these anchors. - - use super::{byte_matches, TokenizerCharClass}; - - const UNICODE_DAG: &str = include_str!("../../../../dsl/std/unicode.dag"); - const TOKENIZE_GENERATED: &str = include_str!("tokenize_generated.rs"); - - #[test] - fn unicode_dag_defines_char_class() { - assert!( - UNICODE_DAG.contains("type CharClass") - && UNICODE_DAG.contains("fn char_in_class") - && UNICODE_DAG.contains("Whitespace") - && UNICODE_DAG.contains("IdentContinue"), - "expected `CharClass` sum + `char_in_class` predicate in `dsl/std/unicode.dag` authority" - ); - assert!( - UNICODE_DAG.contains("IdentContinue => (cp >= 48 && cp <= 57)"), - "expected `IdentContinue` to inline the digit range (no `char_in_class` recursion on the same `c`)" - ); - assert!( - !UNICODE_DAG.contains("char_in_class(c: c, class: Digit)"), - "`IdentContinue` must not recurse into `char_in_class` with the same `c` (CX / bounded recursion)" - ); - } - - #[test] - fn generated_tokenizer_avoids_ascii_host_predicates() { - assert!( - !TOKENIZE_GENERATED.contains("is_ascii_whitespace") - && !TOKENIZE_GENERATED.contains("is_ascii_digit") - && !TOKENIZE_GENERATED.contains("is_ascii_alphabetic") - && !TOKENIZE_GENERATED.contains("is_ascii_alphanumeric"), - "tokenize_generated.rs should route ASCII classes through `byte_matches` / `TokenizerCharClass`, \ - not std-lib `is_ascii_*` helpers" - ); - assert!( - TOKENIZE_GENERATED.contains("byte_matches") - && TOKENIZE_GENERATED.contains("TokenizerCharClass::Whitespace") - && TOKENIZE_GENERATED.contains("TokenizerCharClass::Digit") - && TOKENIZE_GENERATED.contains("TokenizerCharClass::IdentStart") - && TOKENIZE_GENERATED.contains("TokenizerCharClass::IdentContinue"), - "expected structural CharClass projection wired into generated tokenizer" - ); - } - - #[test] - fn byte_matches_locks_ascii_scanner_semantics() { - for byte in 0u8..=127 { - let b = byte; - assert_eq!( - byte_matches(b, TokenizerCharClass::Whitespace), - b.is_ascii_whitespace(), - "Whitespace mismatch at {byte}" - ); - assert_eq!( - byte_matches(b, TokenizerCharClass::Digit), - b.is_ascii_digit(), - "Digit mismatch at {byte}" - ); - assert_eq!( - byte_matches(b, TokenizerCharClass::IdentStart), - b.is_ascii_alphabetic() || b == b'_', - "IdentStart mismatch at {byte}" - ); - assert_eq!( - byte_matches(b, TokenizerCharClass::IdentContinue), - b.is_ascii_alphanumeric() || b == b'_', - "IdentContinue mismatch at {byte}" - ); - } - } -} diff --git a/src/v3/compiler/src/tokenize_generated.rs b/src/v3/compiler/src/tokenize_generated.rs index ffc1a804312..d0ad7752589 100644 --- a/src/v3/compiler/src/tokenize_generated.rs +++ b/src/v3/compiler/src/tokenize_generated.rs @@ -2,7 +2,26 @@ // `regen_tokenize`. Regenerate instead of hand-editing. use crate::diagnostics::{Diagnostic, SourceSpan}; -use crate::tokenize_char_class::{byte_matches, TokenizerCharClass}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ScannerCharClass { + Whitespace, + Digit, + IdentStart, + IdentContinue, +} + +#[inline] +pub(crate) fn byte_matches(byte: u8, class: ScannerCharClass) -> bool { + match class { + ScannerCharClass::Whitespace => matches!(byte, b'\t' | b'\n' | b'\x0c' | b'\r' | b' '), + ScannerCharClass::Digit => byte.is_ascii_digit(), + ScannerCharClass::IdentStart => { + byte.is_ascii_lowercase() || byte.is_ascii_uppercase() || byte == 0x5f + } + ScannerCharClass::IdentContinue => byte.is_ascii_alphanumeric() || byte == 0x5f, + } +} #[derive(Debug, Clone, PartialEq, Eq)] pub enum TokenKind { @@ -67,36 +86,16 @@ pub fn tokenize(source: &str, file: &str) -> Result, Diagnostic> { while pos < bytes.len() { let byte = bytes[pos]; - if byte_matches(byte, TokenizerCharClass::Whitespace) { - pos += 1; - continue; - } - - // Line comment prefix from `tokenize.dag` (`line_comment_prefix`). - if bytes.len() >= pos + LINE_COMMENT_PREFIX.len() - && bytes[pos..pos + LINE_COMMENT_PREFIX.len()].eq(LINE_COMMENT_PREFIX) - { - pos += LINE_COMMENT_PREFIX.len(); - while pos < bytes.len() && bytes[pos] != b'\n' { - pos += 1; - } - continue; - } - let start = pos; - if let Some((kind, width)) = punctuation_token(bytes, pos) { - tokens.push(Token { - kind, - span: SourceSpan::new(file, start as u32, (start + width) as u32), - }); - pos += width; + if byte_matches(byte, ScannerCharClass::Whitespace) { + pos += 1; continue; } - if byte_matches(byte, TokenizerCharClass::Digit) { + if byte_matches(byte, ScannerCharClass::Digit) { let mut end = pos; - while end < bytes.len() && byte_matches(bytes[end], TokenizerCharClass::Digit) { + while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::Digit) { end += 1; } let literal = &source[start..end]; @@ -113,9 +112,39 @@ pub fn tokenize(source: &str, file: &str) -> Result, Diagnostic> { continue; } - if byte_matches(byte, TokenizerCharClass::IdentStart) { + if byte_matches(byte, ScannerCharClass::IdentStart) { + let mut end = pos; + while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::IdentContinue) { + end += 1; + } + let text = &source[start..end]; + let kind = match text { + "data" => TokenKind::KwData, + "else" => TokenKind::KwElse, + "false" => TokenKind::KwFalse, + "fn" => TokenKind::KwFn, + "if" => TokenKind::KwIf, + "import" => TokenKind::KwImport, + "let" => TokenKind::KwLet, + "match" => TokenKind::KwMatch, + "module" => TokenKind::KwModule, + "then" => TokenKind::KwThen, + "true" => TokenKind::KwTrue, + "type" => TokenKind::KwType, + "where" => TokenKind::KwWhere, + _ => TokenKind::Ident(text.to_string()), + }; + tokens.push(Token { + kind, + span: SourceSpan::new(file, start as u32, end as u32), + }); + pos = end; + continue; + } + + if byte_matches(byte, ScannerCharClass::IdentContinue) { let mut end = pos; - while end < bytes.len() && byte_matches(bytes[end], TokenizerCharClass::IdentContinue) { + while end < bytes.len() && byte_matches(bytes[end], ScannerCharClass::IdentContinue) { end += 1; } let text = &source[start..end]; @@ -143,6 +172,26 @@ pub fn tokenize(source: &str, file: &str) -> Result, Diagnostic> { continue; } + // Line comment prefix from `tokenize.dag` (`line_comment_prefix`). + if bytes.len() >= pos + LINE_COMMENT_PREFIX.len() + && bytes[pos..pos + LINE_COMMENT_PREFIX.len()].eq(LINE_COMMENT_PREFIX) + { + pos += LINE_COMMENT_PREFIX.len(); + while pos < bytes.len() && bytes[pos] != b'\n' { + pos += 1; + } + continue; + } + + if let Some((kind, width)) = punctuation_token(bytes, pos) { + tokens.push(Token { + kind, + span: SourceSpan::new(file, start as u32, (start + width) as u32), + }); + pos += width; + continue; + } + // String literal (`string_literal_delimiter` + `StringEscapeSpec` rows). if byte == b'"' { let mut end = pos + 1; diff --git a/src/v3/compiler/tests/integration/parse_corpus_manifest.txt b/src/v3/compiler/tests/integration/parse_corpus_manifest.txt index 222f29da0c2..99737756917 100644 --- a/src/v3/compiler/tests/integration/parse_corpus_manifest.txt +++ b/src/v3/compiler/tests/integration/parse_corpus_manifest.txt @@ -16,7 +16,7 @@ src/v3/compiler/tests/four_fixture_pressure/drop.v3 2 2517 00953bc6b6d8c811 src/v3/compiler/tests/four_fixture_pressure/id.v3 2 2452 ec8b40edc3d3c017 src/v3/compiler/tests/four_fixture_pressure/is_empty.v3 3 6755 00117f14eb43dd20 src/v3/compiler/tests/four_fixture_pressure/wrap.v3 4 5189 ac633b042c944480 -src/v3/compiler/tokenize.dag 27 56809 6a375f3c832b2af8 +src/v3/compiler/tokenize.dag 29 59946 330ebc6371e9a03c src/v3/spec/go.dag 122 1126317 82d4ec71acfdd80e src/v3/spec/python.dag 88 579466 4d7f9bb14b85da04 src/v3/spec/rust.dag 140 1199314 5bd6bf1bf8a928b5 diff --git a/src/v3/compiler/tests/integration/sg0_census_test.rs b/src/v3/compiler/tests/integration/sg0_census_test.rs index 271151b57e6..6feb8e5fada 100644 --- a/src/v3/compiler/tests/integration/sg0_census_test.rs +++ b/src/v3/compiler/tests/integration/sg0_census_test.rs @@ -203,9 +203,6 @@ const EXPECTED_HAND_AUTHORED_NON_TEST: &[&str] = &[ "src/v3/compiler/src/regen_parse_emit.rs", "src/v3/compiler/src/regen_parse_tables_emit.rs", "src/v3/compiler/src/test_runner.rs", - // R1 T-Sub `sub_charclass_in_std_unicode`: Rust mirror of `std.unicode::CharClass` - // until M1(2.8) allows `CharClass` / `List` in `tokenize.dag` `data` bodies. - "src/v3/compiler/src/tokenize_char_class.rs", "src/v3/compiler/src/workflow_idempotency.rs", "src/v3/compiler/src/workflow_parallelism.rs", ]; diff --git a/src/v3/compiler/tokenize.dag b/src/v3/compiler/tokenize.dag index 3e19d193cd0..da0d385313e 100644 --- a/src/v3/compiler/tokenize.dag +++ b/src/v3/compiler/tokenize.dag @@ -52,14 +52,10 @@ // // **CharClass — tokenizer half (bounded interim, not full lane closure).** // `std.unicode` declares `CharClass` + `char_in_class` (canonical `.dag` -// authority). The ROADMAP lane also names `syntax.dag` retagging and -// structural class rows in *this* file — those are **not** claimed here: -// `dsl/std/syntax.dag` / `dag_keyword_set` / `OperatorSpec.symbol` are unchanged -// in this tranche. Once this compiler authority authors scanner rows such as -// `data ascii_scan_order: List = [...]`, `regen_tokenize` can consume -// them structurally instead of routing through `tokenize_char_class.rs`, the -// current handwritten Rust mirror of the `.dag` ASCII predicates (ratcheted by -// unit tests in `tokenize_char_class.rs`). +// authority). `ascii_scan_order` makes tokenizer scanner precedence a +// structural `List` consumed directly by `regen_tokenize`; ASCII +// predicate bodies remain a bounded generator bridge until `char_in_class` +// semantics are structurally consumed. // // **Framing.** Phase 1 was a consumption gap: the character-level modeling was // already present, but tokenize.dag and syntax.dag encoded the same activity @@ -95,6 +91,15 @@ import v3.std.tokenize { TokenKind } +import v3.std.unicode { + CharClass +} + +// Scanner order is now structurally owned by tokenizer authority. Predicate +// bodies still live in `regen_tokenize` until `char_in_class` is structurally +// executable for this path. +data ascii_scan_order: List = [Whitespace, Digit, IdentStart, IdentContinue] + // Fixed diagnostic message bodies for tokenizer failures (C-8: errors only). data diagnostic_unterminated_string_escape: String = "unterminated string escape" data diagnostic_unterminated_string_literal: String = "unterminated string literal"