parse cost forensics

[briansrls]

✅ #6365 has merged; origin/main is merged in — this PR's diff is now just two files: src/v1/stage0/src/v1_interpreter.rs (caller-attribution probe + measured-zero note) and src/v1/stage0/src/cli_run.rs (probe dump section).

Parse-cost forensics: pin the interpreted-parse DNF root cause + a measured land-or-zero on the interpreter fold-site cut

Continuation of the adhoc-c328b166-bca residual-driver hunt (neat-stag-780's #6365 named it "next-session work"). This lane took the interpreter-side free_monoid_to_vec fold chokepoint; the .dag-side accumulator fixes stay in #6365. Verdict up front: the datetime DNF root cause is pinned to one function, and the interpreter cut is a measured clean zero (dropped), leaving a reusable caller-attribution probe and a permanent no-op note.

Root cause — PINNED: lex_repeat_loop (src/v2/compiler/01_tokenize.dag:158)

A new caller-attribution probe (below) attributes each large fold_list to the .dag function that contains it. On datetime.dag (the DNF file), one caller dominates:

caller = lex_repeat_loop [left fold]   max_len=19021   elem=Int
  calls   330 -> 710 -> 1204   (climbing, timeboxed dumps)
  items   6.0M -> 12.3M -> 19.2M   (climbing => DNF)

19021 is datetime.dag's exact character count. lex_repeat_loop does fold_list(xs: init.remaining) — it folds the entire remaining source char-list purely to bound an iteration, and after done=true it still visits every remaining char (the true => state pass-through). So each call is O(remaining source length) regardless of the actual token length; once per repeat-token ⇒ O(n²) in source length. This is a second, deeper tokenizer O(n²), distinct from neat-stag's list_snoc reverse-cons fix — it confirms the "dies upstream of parse-proper" reframe and names the culprit. The root-cause dataflow fix (bounded recursion/Loop, bound = lexeme length not source length) is a .dag tokenizer change routed to the tokenize lane (neat-stag right-of-first-refusal); this probe receipt is the acceptance baseline.

The interpreter fold-site cut — MEASURED CLEAN ZERO (dropped)

Candidate: stream eval_fold_list_native (the left fold) over the Cons-chain instead of materializing an intermediate Vec (DESIGN §2 redundancy removal). It was built, proven byte-identical, and measured against two independently-built release binaries (base = neat-stag head; fix = base + streaming):

Check	base	fix (streaming)
parse-tree content hash, tiny	9afb740ee43f9e61	9afb740ee43f9e61 ✓
parse-tree content hash, small	ed071cb2baeaeb61	ed071cb2baeaeb61 ✓
wall-clock, small (end-to-end)	20.5 / 19.5 s	19.2 / 20.9 s
peak RSS, small (end-to-end)	~168.3 MiB	~168.2 MiB

Byte-identical, but neither wall-clock nor peak RSS moves (both within run-to-run noise). Why it must be zero: the datetime driver folds elem=Int codepoint lists (a trivial copy, not a deep clone — so the removed materialization is pure alloc churn of the cheapest element), and the intermediate Vec is transient (freed each fold, so it never contributes to peak RSS). The real O(n²) is the caller re-folding the whole source (lex_repeat_loop), not this per-call materialization — so no interpreter change to fold_list can reprice the DNF.

Per DESIGN §6 (denominate in displaced cost; a no-op displaces nothing) and the operator's "root-cause dataflow only" steer, the streaming rewrite is dropped. A permanent in-place note records that it was tried and measured, so the next hunter does not re-derive it.

Two-axis site data (why tiny/small looked different from datetime)

free_monoid_to_vec per-site (by items), tiny → small → datetime:

• tiny/small are fixed-cost-dominated by the witness harness's 43-module import-closure resolution (identical in both runs): value_hash (284) 427k→455k (1.06×, ~flat), right-fold (2580) call-heavy on ~2-element lists, contains (5434), left-fold (2556) small. The parsed file grew 4.65× (43→200 memo positions) while these grew ≤1.6× — they are not the file-size driver.
• datetime is left-fold-dominated: site 2556 (left fold) alone reaches 30M+ items and climbing, everything else negligible — and the caller probe attributes 2556 entirely to lex_repeat_loop.

This de-lists all four free_monoid_to_vec heavy sites (value_hash, left/right fold, contains) for future hunts: none is the datetime driver; the driver is a .dag caller.

What lands

• Caller-attribution probe (v1_interpreter.rs): DagFnGuard pushes the innermost .dag fn name on each call_function; record_fold_caller attributes large folds (≥100 items) to that caller with element type_label. Gated behind GUNBC_FLATTEN_SITE_DUMP_SECS (a no-op otherwise — the disabled overhead is within the ~20s noise, confirmed since the fix binary carrying it matched base). Extends neat-stag's flatten-site instrument with the .dag-caller axis that pinned lex_repeat_loop. dissolve-on: the residual-hunt work item closes.
• Measured-zero note in eval_fold_list_native — records the streaming attempt + result so it is not re-investigated.

Test plan

• Byte parity: parse_tree_content_hash_witness tiny+small, base vs fix binaries — identical.
• Verdict: wall-clock + peak RSS, base vs fix, small end-to-end (both complete identical work — no progress confound) — zero within noise.
• cargo fmt --all --check, cargo clippy --all-targets -- -D warnings (local).
• CI (ci / ci_regen / rust_tests) — on ready.

🤖 Generated with Claude Code

[gunbai-bot]

Addressed — the conflict markers were transient, never in a pushed commit. Timeline: #6365 squash-merged into main at 02:28:04, which put my then-head a33dfcc into merge conflict with main; this review ran at 02:30:28 inside that window, so its merge-preview injected the <<<<<<< / ======= / >>>>>>> markers at cli_run.rs:2847 and v1_interpreter.rs:6674. I resolved the conflict with a merge commit (d24205d6, current head): the two probe blocks are purely additive next to neat-stag's now-merged flatten-site instrument, so I kept both sides and dropped the markers.

Verified on the current head d24205d6:

• git grep '<<<<<<<' / >>>>>>> over both files → empty (no markers)
• cargo build --release --bin gunbc → clean
• cargo fmt --all --check and cargo clippy --bin gunbc -- -D warnings → clean
• diff vs origin/main is now just the two files (v1_interpreter.rs +99, cli_run.rs +9)

DagFnGuard::enter (call_function) and record_fold_caller (both fold sites) are the intended additive wiring, not a broken intermediate state. No code change needed beyond the already-pushed merge resolution; a re-review of d24205d6 should come back clean.

— sent from nimble-otter-476