Perf C: per-file, per-phase attribution of the native route's context fold (measurement, not a fix)

[gunbai-bot]

Perf item C, deliverable 1. Based on #10940's head (stacked on session/deep-swift-530; retargets to main when #10940 lands).

Result first: the expected defect is ruled out by measurement

The brief said to EXPECT a per-file recompute of something invariant, and to find it or rule it out before any parallelism. It is ruled out, and so is a size-quadratic. What is there instead is a flat per-token constant.

Full closure, 2236 files, from the run's own [native-context-partition] line (srv1, load ~52, emitted driver in adjudicate mode):

phase	time	share	calls
front_end_prepare	0.14 s	0.006%	1
tokenize	277.22 s	12.3%	2236
parse	1803.09 s	80.3%	2236
normalize	138.40 s	6.2%	2236
absorb (qualified-name + index insert)	0.40 s	0.018%	2236
residual (loop + the per-file eprintln)	27.07 s	1.2%	—
context span	2246.32 s		2,285,079 tokens / 25.6 MB

Mean 1005 ms/file, p50 422 ms, p95 2943 ms, max 94.4 s (src/v2/std/compilers/target_model.dag, 55832 tokens). That mean independently reproduces the ~1.1 s/file arithmetic on a different subject — so the per-file cost is subject-independent.

The discriminator

Median ns/token by token decile (1 = smallest files):

2748k 3245k 948k 840k 840k 835k 833k 840k 814k 864k

Deciles 3–10 are dead flat at ~840k ns/token. A per-file invariant recompute makes this curve fall sharply as files grow; a quadratic in file size makes it rise. Neither happens. Three hypotheses die here:

• per-file invariant recompute — ruled out by the flat curve, and directly: front_end_prepare_calls=1, 0.14 s total. The language model and prepared grammar are paid once and carried, and the relay proves it rather than asserting it.
• quadratic accumulator / index insert — absorb is 0.40 s of 2246 s (0.018%). list_snoc_item on roots and source_root_index_insert are measured out, not argued out.
• per-parse grammar analysis (the Carry the grammar digest and production index on the prepared grammar; share the parse-witness fixture trees warm #11032 shape) — parse_table_for_production_analyzed reuses prepared's GrammarFirstAnalysis; no per-file recompute.

Deciles 1–2 are elevated, and that is fold position, not size: at fixed small size (≤90 tokens), files in the first quarter of the fold cost 247 ms median against 57.6 ms in the last quarter — a 4.3× position effect. I have not fully attributed it (it is consistent with first-touch page-fault billing) and I am not claiming it as a per-file constant.

What is actually there

~840 µs to consume one token, uniformly, 80% of it in parse. A recursive-descent parser is normally well under 1 µs/token. This is a different defect class from the one the brief expected — not an invariant recomputed per file, but a per-token constant inside parse — and this PR does not fix it, because deliverable 1 is measurement.

What I checked and did NOT find: the memo is threaded correctly at the nonterminal site (parse_table_record_lookup_call → parse_table_lookup → hit/miss → parse_table_insert → table_after_memo), terminal arms correctly return the input table unchanged, the memo key is (character position, production), and Map is primitive-backed since #11121. "The memo is simply dropped" is not the answer.

The next instrument is small and named: ParseTable already carries memo_hits / memo_misses / memo_lookup_calls and parse_table_memo_stats already exists, but ParseArtifact does not carry them out, so nothing prints them. Threading those three counters into the split line distinguishes "a memo that never hits" from "genuine per-token work in the combinators".

Second finding, not asked for

file_refusals=212 of 2236. Counting the parse/tokenize subset by signature (normalize binds the Rejected and returns in <10 µs while parse has spent seconds): 193 files, 217.4 s = 9.8% of the fold. A tenth of the context fold front-ends files that contribute no root — the v2 front-end corpus frontier (v2.workflow.compile_door_ledger) paid for in full at ~840 µs/token before it refuses. Flagging, not taking.

The seam

v2.compiler.program_assembly — the per-file front end becomes one declaration per phase:

• program_assembly_phase_tokenize(read, lm) -> Outcome<TokenStream>
• program_assembly_phase_parse(tokens, prepared, residue) -> Outcome<ParseArtifact>
• program_assembly_phase_normalize(parsed) -> Outcome<NormalizedTree>
• program_assembly_phase_token_count(tokens) -> Int

Each phase takes the PREVIOUS PHASE'S OUTCOME and binds it, so diagnostic merging and refusal propagation stay inside .dag. program_assembly_read_to_normalized_root_prepared is retained and is now defined by those three calls — entering the phases separately is the same computation, not a second front end beside it.

v2.compiler.00_compile — the matching seam: native_test_front_end_prepare() carries the once-per-fold invariant as NativeTestFrontEnd; native_test_context_absorb(state, read, root_outcome) is the non-front-end half of the step; native_test_context_state_empty() / native_test_context_finish(state, ingest). native_test_context_from_ingest is retained as the composition every clock-free caller uses — including the driver's own run_census, which wants per-file refusals and no attribution.

std.compiler_entry.SourceRootEvalDriver — a phase span is a REALIZATION fact (.dag is pure and cannot read a clock), so the driver that owns the clock enters each phase and times the call. It prints [native-context-split] per file (path, bytes, tokens, four phase spans) and [native-context-partition] once.

Counts sit beside times because a span alone cannot separate "this file is large" from "this file pays a per-file constant" — the whole question here. The token count is read OUTSIDE every phase span so it prices none of them.

No double counting

Per eager-raven-113's binding accounting ruling: these rows are their own relay on their own basis (basis: context_span) and partition the CONTEXT span. [native-cost-partition]'s exclusive rows are untouched — context remains one exclusive row of the driver wall, so the parent span still reconciles. The four phases partition the per-file span with 0.00 s unattributed; the 1.2% residual is the loop and the per-file eprintln and is reported, not absorbed.

Deliverable 3

1256 is a real closure, not the tree by another name. Independent oracle over the same two roots — seeds = every declared v2.test.* module (the native_route_universe_prefix) plus native_route_live_control_module, edges = the import headers each source declares, transitive to fixpoint — gives on the full tree: scanned 5578, closure_modules 2091 in 14 rounds, closure FILES 2089. Genuinely closure-scoped at ~37% of the tree, and the driver's own fold independently reports 2236 files over the same roots.

That also explains the previously unexplained gap of 4 between closure_modules and closure_reads: closure_modules counts module NAMES reached, closure_reads counts FILES ingested, and the difference is closure members that no scanned source declares. The full tree has exactly 2 (gunbc.recurring_failure_mode.roster, v1.compiler.ownership); a parked tree has more because parking removes declaring files. native_lane_ingest_matches_closure is consistent: its "closure module was never ingested" arm is quantified over modules some source DECLARES, so an undeclared closure member is not a refusal.

The oracle is stated here as a method rather than committed as an artifact, deliberately: a .dag function under gunbc run cannot read the corpus, so the only available forms would be a second closure implementation in the corpus — a §3 fork of native_lane_closure_modules, which already answers this — or a hand-authored script, which §6 presumes scaffold.

Deliverable 2

Not in this PR, by instruction: parallelizing before the per-file cost is honest hides the defect. The seam landed here IS the fold-parallel seam — program_assembly_phase_* is pure in (read, lm, prepared, residue) — so D2 is a realization change over these same functions. The measurement changes its priority in both directions: there is no per-file defect left for parallelism to hide, but it would multiply throughput without touching an ~840 µs/token constant that §6 prices as a cost-shape defect in its own right, and ~10% of the parallel work would still be spent parsing files that refuse.

Verification

• both changed .dag modules typecheck under the whole dag + src/v2 closure
• the emitted crate builds clean, zero warnings, under RUSTFLAGS=-D warnings, and the malformed control refuses correctly (tokenize_lex_e1_unrecognized_char)
• --required-regen named exactly v1_compiler_emit_rust.rs as drifted — also the positive control that the emitter edit is live — and the regenerated file is committed
• merged session/deep-swift-530 rather than rebasing; both .dag authorities auto-merged and only the generated projection conflicted, which the merge driver correctly refuses. Regenerated through the declared repair route and confirmed both relays survive in the merged projection (native-context-split and deep-cat's native-prepare-split), since a textual merge previously dropped that line with no conflict marker.

One caveat on the subject: these numbers are over the FULL closure at this branch head, not the bounded subject, so they are NOT directly comparable to the 1400.79 s bounded figure — same host, different subject. The full closure turns out to be the better subject for D1 anyway: the context fold completes and reports before any per-module prepare, so it needs no reduced claim population to finish.

🤖 Generated with Claude Code

https://claude.ai/code/session_019gufjy7FmMzjTRwAnUgZcZ