02_parse interpreted-cost forensics: refute memo-table-carry, fix 3 O(n)/O(n^2) drivers, name honest residual

[gunbai-bot]

Summary

Work item node://adhoc-c328b166-bca: discriminate interpretation-tax vs algorithmic residue in v2 02_parse.dag run under the v1 seed interpreter.

Prime-suspect hypothesis refuted. The packrat memo table (ParseTable) was suspected of O(table-size) carry cost under the interpreter's persistent-value semantics. Direct read of v1_interpreter.rs's Value enum shows ParseTableRealization's record fields are Rc-shared (cheap rebuild) and its entries: Map is backed by a genuine im_rc::HashMap HAMT (O(log n) insert/lookup), not a full clone. The memo mechanism itself is fine.

Same-file compiled-vs-interpreted timing (v1 seed's own generated parser, native, vs v1_interpreter.rs tree-walking the same 02_parse.dag): the compiled realization scales linearly across 43w-6173w files (~350-450 words/ms, no complexity-class break). The interpreted realization showed a genuine complexity-class break, not just the constant-factor tax a tree-walker should honestly pay.

Three real O(n)/O(n^2) drivers found and fixed:

1. length/count/size fell back to free_monoid_to_vec's O(n) clone even for Value::List/Map/Set, which already track their length natively. parse_current_position (02_parse.dag) calls length() on the full token stream on every parse attempt -- an O(n) tax per attempt the compiled realization never pays. Fix: native_len O(1) fast path in v1_interpreter.rs, checked before the free_monoid_to_vec fallback.
2. parse_token_first used list_at_optional(xs, 0) (itself fold_list-based, no short-circuit) to peek the first token on every lookahead/terminal-check. Swapped to the already-existing list_head (direct Empty/Cons pattern match, genuine O(1)) -- verified empirically first that native Value::List correctly matches Empty/Cons before editing this load-bearing file.
3. parse_match_arm_stmt_body folded over the entire remaining token list per match-arm-body parse via fold_list -- because fold_list has no short-circuit, even after the accumulator's done flag flipped true it kept no-op-iterating every remaining token to EOF: an O(remaining-tokens-to-EOF) cost per match-arm body, i.e. O(n^2) across a file full of match expressions (which real .dag source is). Rewritten as parse_match_arm_stmt_step, a genuine short-circuiting recursion that stops as soon as done is true.

Discriminating agreement test (native_list_length_equals_freemonoid_cons_chain_length, src/v1/tests/src/list_free_monoid_chokepoint_test.rs): pins the new native_len fast path and the free_monoid_to_vec fallback equal on an equivalent native-list vs Cons/Empty-chain -- this repo's cross-representation-equality precedent (numeric tower, #5428).

regen_stage0 --verify: divergence 0 -- the hand-maintained v1_interpreter.rs edit is consistent with the committed stage0.

Per-fix verification receipts (merge-bar demand):

• Fix 1 (native_len fast path, v1_interpreter.rs): native_list_length_equals_freemonoid_cons_chain_length, a plain non-ignored #[test] in src/v1/tests/src/list_free_monoid_chokepoint_test.rs -- pins the fast path and the free_monoid_to_vec fallback equal. cargo test -p v1-compiler-tests list_free_monoid_chokepoint -- 9/9 passed.

• Fix 2 (parse_token_first: list_at_optional(xs, 0) -> list_head): src/v2/test/claim/parse/parse_token_first_empty_semantics_test.dag, two test fn claims pinning the two implementations equal on the empty-token-stream boundary case that motivated the rewrite (Absent == Absent) and the ordinary present case (same lexeme). Both PASS via claim_batch.

• Fix 3 (parse_match_arm_stmt_body: fold_list -> parse_match_arm_stmt_step, a rewrite inside DESIGN-named load-bearing 02_parse.dag): src/v2/test/manual/parse_tree_content_hash_witness.dag computes content_hash(ParseTree) (v2/std/node.dag's structural node hash) over the tiny/small corpus sample, run once against the true PR base (0f98a264dc, built as its own isolated release binary in a separate worktree -- the local main ref was stale at 21fb5d1b2d; using it produced a false grammar_invalid rejection from an unrelated grammar-ambiguity fix, parse: nullable-only ambiguity rejection; non-nullable FIRST overlap becomes counted residue (fixes main parse refusing all input) #6337, that landed between the two) and once against this branch's HEAD. Hashes are byte-identical both ways:

  • witness.dag: 9afb740ee43f9e61 == 9afb740ee43f9e61
  • execution_mode.dag: ed071cb2baeaeb61 == ed071cb2baeaeb61

  This proves fix 2 and fix 3 together produce the exact same parse tree pre/post -- not just "no test regressions" on a load-bearing rewrite.

Honest residual, named with numbers, not absorbed into "fixed": datetime.dag (1956 words) still does not finish after all three fixes -- DNF at 1200s+, RSS ~4.3GB at ~15min elapsed, essentially unchanged from the pre-fix baseline (DNF at 900s, RSS 4.35GB). The compiled-vs-interpreted gap is not yet flat across sizes; a further, unidentified superlinear driver remains in 02_parse.dag and/or v1_interpreter.rs beyond the three closed here. tiny/small witness files (43w/75w) complete correctly and unchanged both before and after (~20-25s each -- the interpreter's real combinator-heavy base tax, a separate, expected constant-factor cost, not this bug).

Verdict (layered, per the brief): engine fix (#2, #3 in 02_parse.dag) + interpreter fix (#1 in v1_interpreter.rs) landed and verified individually correct; full resolution for realistic file sizes (~2000+ words) remains an open residual -- grammar_coverage_p8_wall_clock_blocker (src/v2/lens/enforcement/grammar_coverage.dag:68) updated with full receipts. P8 full-corpus census stays blocked, narrowed but not closed.

Emit-on-demand feasibility (deliverable 4): attempted gunbc compile --source-root src/v2 --source-root dag --target rust as a same-realization double-check. There is no scoped/closure-only compile entry point today -- it always compiles the whole reachable transitive-import closure (indexed 1867 modules, resolved 1201 sources) and was OOM-killed during reconcile after ~67s / RSS 667MB+. Emit-on-demand is not viable via the existing CLI without a scoped compile mode; that is itself the answer, not something fixed in this PR.

Test plan

• cargo test -p v1-compiler-tests list_free_monoid_chokepoint -- 9 passed, including the new agreement test.
• cargo fmt --all --check -- clean.
• cargo clippy --all-targets -- -D warnings -- clean.
• regen_stage0 --verify -- regen_divergence_count=0.
• claim_batch --entry src/v2/test/claim/parse/parse_token_first_empty_semantics_test.dag --function <both> -- PASS/PASS (fix 2 empty-list semantics pin).
• gunbc run --entry src/v2/test/manual/parse_tree_content_hash_witness.dag against the true PR base (0f98a264dc, isolated worktree + binary) and HEAD -- byte-identical content_hash(ParseTree) on both corpus files (fix 2+3 byte-equal parse-output witness).
• Before/after interpreted timing on the 5-file scaling witness (src/v2/test/manual/parse_forensics_scaling_witness.dag): tiny/small unchanged and correct; medium (datetime.dag) still DNF, receipts and numbers above and in the updated blocker marker.
• Compiled-realization calibration harness: src/v1/stage0/examples/parse_compiled_scale.rs (committed receipt harness, not wired into any build/CI gate).

Closes node://adhoc-c328b166-bca

[gunbai-bot]

Re: dashboard "CI FAILING" notification for the `ci` check — this is the known era-4 10-minute job-kill (`##[error]The action 'gunbc ci (.dag witnesses + gates)' has timed out after 10 minutes.`), not a regression from this PR's changes. Per the current CI regime ruling (#6323), a cancelled/timed-out `ci` job is expected; merge basis is dashboard approvals + `rust_tests` + `mergeable`, all of which are green (2/2 fresh approvals, rust_tests pass, ci_regen pass, mergeable=CLEAN). No fix pushed — nothing to fix here.

— sent from neat-stag-780