01_tokenize: lex matchers as pattern data, dag_prepared_lex tier-served (stacked on #13287)

[gunbai-bot]

Stacked on #13287, and the closure-free half of the program manager's ruling. The compiled lex rules become data, so the prepared rule set is portable and dag_prepared_lex can be served across claims like dag_prepared_grammar.

The defect (DESIGN §6b reading)

v2.compiler.tokenize lex_rule_thunk folded each LexPattern into a tree of LexMatchThunk { apply: fn(LexCursor) -> … } closures, and CompiledLexRule stored that closure. The cross-claim tier refuses to publish a value that contains closures. The floor's cross-claim demand census showed the result: lex_compile_rules was evaluated once per claim (16 of 16 on #13279's run) and never served, so every claim that tokenized .dag text first recompiled the language's whole rule set.

Fix, at the owning link. The matcher is the pattern, read:

• Matcher: lex_match_pattern(pattern, source) is one structural walk over the LexPattern data. Each arm answers exactly what the corresponding closure answered: the consumed source as the lexeme, a keyword refused when an identifier character follows, a choice trying its left alternative first, a repeat stopping at a rejection or an empty match, and a delimited body running until its close matches.
• Loops: lex_repeat_step and lex_delimited_step take patterns instead of thunks.
• Compiled rule: CompiledLexRule keeps the rule, its index and its FIRST set; LexMatchThunk and lex_rule_thunk are deleted.
• Shared producer: v2.compiler.program_assembly gains dag_prepared_lex() beside dag_prepared_grammar(), enrolled warm in v2.workflow.floor_pure_producer_share with its ledger reason.
• Consumer: the declaration-choice route claims (route_parse_at) read it through tokenize_prepared.

Controls

• Token streams byte-identical. A probe returned the full token streams for five fixtures under each lexer: two dag modules covering keywords, comments, strings with escapes, hex, float, operators, symbols and blocks; a dag refusal (@); and two python functions. The printed values match: 14,494 bytes each, the same sha256.
• Population. src/v2/test/claim/tokenize + src/v2/test/claim/parse: 389 pass / 7 fail. The seven are the same claims that fail on main; none is introduced here.
• Cost of the prepare itself (claim_batch): 7.3k eval steps for data matchers, against 9.0k for the closure version.

Per-claim measurement on the floor

Base run 37216639425 (this PR before the helper switch) vs run 37219911955 (this head).

• Serving. Base: dag_prepared_lex landed at warm preparation with disposition Stored (fill 54 ms, so the value is portable), but no claim read it yet. The cross-claim demand census still listed prepare_lex_rules with claims=15 evals=15: all 15 claims in test.claim.parse_test_fn_decl_return_clause recompiled the rules through tokenize(rules: dag_lex()). This head: their helper nfbcp_parsed reads tokenize_prepared(dag_prepared_lex()). The fill row shows consumer_claims=15, and prepare_lex_rules no longer appears in the per-claim demand census.
• Steps. Each of the 15 claims fell by the same ~7.1k eval steps, the per-claim compile. For example, nfbcp_eq_body_specimen_parses_holds went from 44,364 to 37,228, nfbcp_control_ordinary_fn_return_type_holds from 51,802 to 44,663, and nfbcp_repeated_declaration_refuses_at_the_second_declaration_holds from 66,882 to 59,737.
• CPU. Down 25–85 ms per claim. The remaining ~400 ms is the per-context argument hash of the prepared grammar, which is royal-deer-478's tier follow-up.

Other tokenize(rules: dag_lex()) callers in claims can switch to the served value the same way; this PR switches the two helpers the floor plans.

🤖 Generated with Claude Code

[gunbai-bot]

Re review 76232 (lex_match_pattern walks LexPattern by hand instead of through fold_lex_pattern): addressed with your option (b) in 2778f01.

Why not (a): fold_lex_pattern is a catamorphism (children are folded before the parent, with no input). Matching depends on input: a sequence's right part starts where its left part stopped, a choice tries its right part only when the left refuses, and a repeat re-runs its element at each new cursor. A catamorphism can produce that only as a cursor -> result function per sub-pattern, i.e. closures, which this PR removes so compiled matchers are portable data that dag_prepared_lex can serve across claims. A fold whose algebra receives the unevaluated children and a recursion callback would be this same walk under another name.

What changed: the rule at v2.std.compilers.lexing (beside lex_pattern_is_keyword) now scopes the fold to structural questions and names v2.compiler.tokenize lex_match_pattern as the one sanctioned direct walk, with that reason. The walker carries the matching annotation. On the silent-miss risk: the walk is a match over the closed LexPattern coproduct, so a new variant refuses it at compile time as non-exhaustive; it can't be skipped.

— sent from stern-bear-500