interpreted-parse cost forensics: discriminate interpretation-tax vs algorithmic residue in v2 02_parse under the v1 interpreter - same-file compiled-vs-interpreted timing, packrat memo effectiveness + table-carry cost interpreted, verdict = engine fix / interpreter fix / honest no-op-until-S2 (upda

[gunbai-bot]

Summary

• Fix 4/4 (this cycle): lex_walk_step (01_tokenize.dag) accumulated the token list via list_snoc_item every token -- an O(n²) tax paid before parsing starts, in a different pipeline stage from the three already-landed 02_parse.dag fixes. Rewritten as reverse-cons + a single .reverse() at tokenize()'s exit.
• Verified byte-equal via a new order-sensitive token-stream content hash (class+lexeme, in stream order) run against independently-built pre-fix and post-fix binaries on tiny/small corpus samples -- Present{1ca5eb5ff95f753f} / Present{2ebd5cca313cc457}, identical both sides. New file: src/v2/test/manual/token_stream_content_hash_witness.dag.
• Full read-based census of 02_parse.dag's 9 list_snoc_item call sites, each dispositioned (dead/test-only, bounded fixed grammar-analysis cost, or already-fixed) -- see table below.
• grammar_coverage_p8_wall_clock_blocker marker updated to the honest current state: tokenize is now confirmed non-bottleneck, but datetime.dag still DNFs deeper in live parsing (past grammar_validate_for_parse); the driver is not yet pinned to a specific call site. Not claiming a false completion.
• Interpreter call-frequency watchlist extended with parse_table_record_hit/_miss/_lookup_call and parse_choice_residue_backtrack for the in-session continuation of the hunt (memo hit-rate discriminator).

02_parse.dag census (9 list_snoc_item sites)

Site	Disposition
parse_match_arm_stmt_step	fixed this cycle (prior commit)
parse_expr_repeat_step	fixed this cycle (prior commit)
token_position_indices (line 437)	dead -- only reference is a test file
parse_sync_step/parse_skip_to_sync (1575/1578)	dead -- only reference is a test file
first_list_union (1007), upsert_production_first_row (1097), append_choice_ambiguity_row (1157), compute_left_corner_closure (1418)	bounded receiver -- all inside grammar_validate_for_parse's one-time, grammar-size-bound (P=36 productions) analysis; confirmed file-size-independent (call counts identical between small-file and full-file runs)
grammar_choice_ambiguity_row_diagnostic_list (1305)	dead on the accept path -- only fires on grammar rejection

None of the 9 sites explain the file-size-proportional growth still observed deep in live parsing on datetime.dag (30M+ items at the native fold sites). That residual is tracked as an open, in-session hunt (work item node://adhoc-c328b166-bca), not claimed as resolved by this PR.

Test plan

• token_stream_content_hash_witness.dag run on tiny/small samples against both pre-fix and post-fix independently-built binaries -- byte-equal
• parse_tree_content_hash_witness.dag (existing) confirms fix-2/fix-3 (parse_expr_repeat_step, parse_match_arm_stmt_step) byte-equal
• Direct call-frequency comparison (list_snoc_item flat=6 vs climbing 284→1502 at matched checkpoints) confirms the tokenizer fix's effect
• CI (ci / ci_regen / rust_tests) -- pending on latest push

🤖 Generated with Claude Code

Update: cut-b bundled in (not a separate PR)

upsert_production_first_row's O(P²) grammar-analysis carrier (GrammarFirstAnalysis.first_rows / ParseTableRealization.first_rows) has been converted from a List<ProductionFirstRow> linear-scan-then-append to a native-HAMT-backed Map<Symbol, List<Symbol>> (matching the existing ParseTableRealization.entries: Map<...> carrier pattern in this same file). All consumers (production_first_row_lookup and everything downstream through expr_first_fold) were confirmed name-keyed-only (no order dependency) before the conversion.

This was intended to land as its own follow-up PR per plan, but an auto-commit captured a mid-edit snapshot on this branch before the split could happen; unwinding it cleanly would have left the branch in a worse state under time pressure, so it's bundled here instead. Verified byte-equal via parse_tree_content_hash_witness.dag on tiny/small samples before and after (9afb740ee43f9e61 / ed071cb2baeaeb61, identical both sides) -- pure algorithmic rewrite, confirmed the same way as the other three fixes in this PR.

Before/after cycle receipt (small file, execution_mode.dag)

Measured time gunbc run --claim-run ... forensics_scaling_witness ... witness_forensics_scale_small_report (full cold-load + tokenize + grammar-validate + parse cycle, this PR's branch vs the pre-cut-b f8f6f80f44 baseline 02_parse.dag, same binary, 3 runs each):

• Pre-cut-b: 21.99s
• Post-cut-b: 19.17s / 20.41s / 19.28s (~19.6s avg)

Honest number: cut-b shaves ~2.4s (~11%) off this cycle, not the whole ~20s floor -- the fixed-cost grammar-analysis pass this rewrite targets is real but smaller in wall-clock terms than the estimate. The dominant remaining cost is still the unidentified file-size-scaling residual deeper in live parsing (this file: memo stats hits=0, misses=200, lookup_calls=200 -- 0% hit rate, feeding directly into the in-session memo-effectiveness hunt below).

Update: 2 more diag-accumulator sites + memo distinct-key discriminator (decisive negative result)

parse_match_arm_stmt_step and parse_expr_repeat_step both accumulated per-iteration diagnostics via list_append(left: diags.reverse(), right: acc.acc_diags) -- missed by the earlier census because it scoped only to list_snoc_item, a different primitive. Converted both to List<List<Diagnostic>> batches (Cons-prepended per step, O(1)) + a single flatten pass at the end. Verified byte-equal (9afb740ee43f9e61 / ed071cb2baeaeb61, unchanged).

Honest caveat: list_append(left: small, right: large) walks left (the native fold_list_right implementation), so it's O(len(left)) per call, not O(len(right)) -- this may NOT have been an O(n²) site despite the growing call-count signature that flagged it. Landed as a correctness-neutral cleanup (parity with the already-safe acc_nodes pattern), not claimed as a proven complexity fix.

Added a cross-thread parse-memo discriminator (PARSE_MEMO_LOOKUPS/HITS/DISTINCT_KEYS, global so it survives a DNF under GUNBC_FLATTEN_SITE_DUMP_SECS) per the manager's request for a decisive test. Result on the datetime.dag half-truncation: lookups=0, hits=0, distinct_keys=0 -- the native fast-path parse-table memo cache never engages at all on this run. Real, separate finding; not proven to be the DNF driver (the .dag-level table's own stats are never reached either, since the run still doesn't finish in 90s).

New lead, not yet attributed: v1_interpreter.rs:2556 (eval_fold_list_native's free_monoid_to_vec) is called with a rapidly growing total item count -- 6.4M items/1748 calls at t=15s, 14M items/9482 calls by t=75s, on a file that still hasn't finished parsing by 90s -- and this predates all of today's changes (confirmed present in the very first run of this session, before any .dag edits). Average ~1400 items per fold_list call is far too large to be my new parse_flatten_diag_batches (only called once per repeat-construct, on small per-step batches). Caller isn't yet attributed. This is the strongest open lead for the actual residual driver.

Correction: native memo cache is NOT dead; DNF is upstream of the memo table entirely

Verified via a fast-completing control file (witness.dag, 43 productions): the native fast-path parse-table memo cache works correctly end-to-end (43 lookups, 43 distinct keys, 0 hits -- the expected benign all-distinct signature). The earlier "lookups=0 hits=0 distinct_keys=0" report for the datetime.dag half-truncation was correct data, not a broken instrument: parse_table_lookup is only reachable from inside parse_production_memo_stats, which requires grammar_validate_for_parse to have already returned -- and on the truncated file, grammar_validate_for_parse calls=1 only appears in the periodic dump at ~t=75s of a 90s budget. The process spends essentially the entire window before the first memo lookup ever happens. The residual driver is upstream of parse-proper -- inside tokenize() or, more likely, grammar_validate_for_parse() -- not in packrat re-parsing or the memo/Map layer. This reverses the list_append/fold_list framing from the update above; that growth is grammar-analysis-phase residue, not parse-proper residue. Next step (not started, next session): bisect/instrument grammar_validate_for_parse's internals (compute_left_corner_closure, compute_nullable_set, first-row analysis) directly.

Update: forensics recorders env-gated, ADHOC-DEBUG removed (review fix)

cursor/composer-2.5's REQUEST_CHANGES flagged that this session's forensic instrumentation (record_call_frequency, record_parse_memo_lookup, record_flatten_site, record_list_cons_tail_split) recorded unconditionally on hot paths -- only the periodic stderr dump was gated on GUNBC_FLATTEN_SITE_DUMP_SECS, not the recording itself -- and that parse_table_memo_scope_and_key shipped unconditional ADHOC-DEBUG eprintln! arms (a local diagnostic captured by an auto-commit before I could revert it). Both valid. Fixed: added residual_hunt_forensics_enabled() (a OnceLock<bool> on the same env var) with an early-return guard at the top of each recorder -- default/production path now pays one relaxed atomic load per call site, not a mutex lock + HashMap/HashSet write -- and added the SCAFFOLD/dissolve-on marker DESIGN Sec6 requires (dissolve when adhoc-c328b166-bca closes). Removed the ADHOC-DEBUG eprintln arms entirely; they'd already served their purpose (proved the memo-key extraction logic is sound) and are not needed going forward. Re-verified byte-equal content hash and forensics on/off behavior after the change.

Update: pipeline-wide list_snoc_item census (33 sites, 03_resolve/04_infer/06_translate/00_compile/03_body_producer/03_normalize)

Per the manager's instruction to widen the class-audit beyond 02_parse.dag/01_tokenize.dag once the same disease showed up in two stages, read and dispositioned every remaining list_snoc_item call site across the rest of the pipeline (34 sites total; one, 04_infer.dag's, gets its own entry below).

33 sites: bounded and safe. Every one folds over a single node's own children/arity -- n.children, an arrow's input-type list, a lens roster, a fixed enum-variant list -- never over the whole tree or whole file, so none can blow up with file size the way tokenize/02_parse.dag's whole-file accumulators did. 06_translate.dag (translate_fold_step) and 03_normalize.dag (normalize_fold_step) both use the fold_node tree catamorphism correctly: each step appends exactly one edge into parent.children, which stays bounded by that node's own arity as the walk goes up -- this is the safe reference shape.

One new genuine finding, 04_infer.dag (flagged, not fixed, not the live driver): infer_gather_fold_step (line 998) does NOT follow the safe fold_node shape above -- at every node in the tree it merges the entire accumulated subtree entry list of each child into the parent's via concat_inferred_facts_entries (line 270), which itself folds list_snoc_item one item at a time over the child's list. Cost per node is proportional to its already-accumulated subtree size, not O(1) -- the same disease as tokenize/parse, structured as a tree-wide concat instead of a flat accumulator. facts_map_from_entries (line 352) re-walks the fully-flattened list the same way. Confirmed NOT the live datetime.dag driver: parse itself never returns on this file, so infer never runs. 04_infer.dag is a DESIGN-named pipeline stage, so a fix here needs its own witness discipline rather than riding this PR -- tracked as a separate future work item, not fixed in-session.

[gunbai-bot]

Addressed in e9fb696.

• 2412/6403 record_call_frequency / record_parse_memo_lookup: valid finding. Added residual_hunt_forensics_enabled() (a OnceLock<bool> reading the same GUNBC_FLATTEN_SITE_DUMP_SECS env var the dump already gates on) and an early-return guard at the top of each recorder. Default/production path now pays one relaxed atomic load per call site instead of a mutex lock + HashMap/HashSet write. Added a SCAFFOLD/dissolve-on marker (dissolve when adhoc-c328b166-bca closes) since there was none.
• 6350 record_flatten_site / 2144 record_list_cons_tail_split: same fix, same gate.
• 2606-2641 ADHOC-DEBUG eprintln: these were a local one-off diagnostic I'd already reverted in my working tree before the auto-committer captured a mid-investigation snapshot onto the branch (a known recurring hazard this session, unrelated to this review). They'd already served their purpose (confirmed the memo-key extraction logic is sound) and are removed — the match arms are back to plain _ => return None.

record_flatten (thread-local, pre-existing on main, not part of this PR's diff) was left as-is since it's outside the scope of this finding. Re-verified byte-equal content hash and forensics on/off behavior after the change.

— sent from neat-stag-780