Perf item C, recut onto main: per-file attribution of the native route's context fold (#11219)

[gunbai-bot]

Perf item C, recut onto main: per-file attribution of the native route's context fold (#11219)

Recut of #11219 (nimble-ibex-813, archived) onto current main at the operator's direction. #11219's body follows verbatim below.

Resolution record

• origin/main merged into Perf item C: per-file attribution of the native route's context fold (1.1 s/file, 89% of wall) — find the per-file invariant recompute before any parallelism; then fold-parallel; confirm 1256 is the true import closure #11219's head 645fe44 (merge commit, not rebase). One semantic conflict, in src/v2/compiler/00_compile.dag: main's native_test_context_from_ingest now builds NativeTestContext.resolution via resolution_context(lm, roots); this PR splits that fold into native_test_front_end_prepare / native_test_context_fold_step / native_test_context_finish so a driver can enter it per file. Resolution: keep the split and thread the prepared front end (NativeTestFrontEnd) into native_test_context_finish, which builds resolution from the once-prepared language model — no second dag_language_model() call, which is the recompute this PR exists to remove. The driver template in 05_emit_rust.dag passes front_end through.
• The merge had kept Perf item C: per-file attribution of the native route's context fold (1.1 s/file, 89% of wall) — find the per-file invariant recompute before any parallelism; then fold-parallel; confirm 1256 is the true import closure #11219's stale emitter mirror (it referenced is_tco_identity_passthrough, removed on main by Assign every TCO loop slot from its tail-call argument #11444); replaced by main's mirror and regenerated by the transaction on srv1 (regen.SKz6OA at e2a26a8): planned=156 executed=156 adjudicated=156, fixed_point_equal=true; exactly v1_compiler_emit_rust.rs changed; installed at 67feaa6. No mirror was hand-edited.
• Receipt class: 1 (src/v2/compiler/00_compile.dag, 02_parse.dag, program_assembly.dag) + 2 (05_emit_rust.dag + mirror). Exact-head M+T native receipt owed before landing; posted here when the pair reports.
• Perf item C: per-file attribution of the native route's context fold (1.1 s/file, 89% of wall) — find the per-file invariant recompute before any parallelism; then fold-parallel; confirm 1256 is the true import closure #11219's two approvals (claude, cursor) do not carry to this composed head; a fresh review row is required.

Native receipt

pending: M = merge base on main, T = merge(M, head); keyed driver-rows comparison.

---

Perf item C, deliverables 1 and 3, plus the memo-counter follow-on. #10940 and #11216 have both landed; this is the REPLAY of that work onto main (not a merge — my old branch carried deep-swift-530's individual commits while main has them squashed, so a merge would have duplicated them). Supersedes #11206, which GitHub auto-closed when its base branch was deleted at merge.

Result first: the expected defect is ruled out by measurement

The brief said to EXPECT a per-file recompute of something invariant, and to find it or rule it out before any parallelism. It is ruled out, and so is a size-quadratic. What is there instead is a flat per-token constant whose location inside the parse this PR does NOT claim to have found.

One run, one basis. Every figure below comes from a single execution of the instrument on this PR's head (26628e29), srv1, load ~55 at start, emitted driver run directly in adjudicate mode so the relay streams. Figures from any other run are not mixed in.

[native-context-partition], basis context_span, full closure, 2243 files / 2,286,564 tokens / context 2070.21 s:

phase	time	share	calls
front_end_prepare	0.13 s	0.01%	1
tokenize	256.41 s	12.39%	2243
parse	1666.65 s	80.51%	2243
normalize	117.96 s	5.70%	2243
absorb (qualified-name + index insert)	0.43 s	0.02%	2243
residual (loop + the per-file eprintln)	28.63 s	1.38%	—

file_refusals=212, per-file span p50 392 ms, p95 2682 ms, max 86.0 s. Memo totals from the same line: 6,557,074 lookups, 400,160 hits (6.1%), 6,156,914 misses, 2.87 lookups/token.

The discriminator

Median ns/token by token decile (1 = smallest files), and the per-attempt quotient beside it:

decile	med tokens	lookups/token	hit %	µs/miss	µs/token
1	54	2.77	4.3	942.1	2168.0
2	75	2.64	3.0	862.6	2619.0
3	124	2.47	3.8	303.5	596.5
4	222	2.50	5.1	256.9	566.4
5	348	2.64	5.6	241.4	586.9
6	502	2.79	6.1	230.6	596.7
7	730	2.84	6.4	230.2	596.1
8	1065	2.83	6.3	228.0	600.3
9	1611	2.89	6.2	227.9	584.2
10	3592	3.05	6.2	237.5	641.3

Deciles 3–10 are flat on every column. Three hypotheses die here:

• per-file invariant recompute — ruled out by the flat curve, and directly: front_end_prepare_calls=1, 0.13 s for the whole fold. The language model and prepared grammar are paid once and carried, and the relay proves it rather than asserting it.
• quadratic accumulator / index insert — absorb is 0.43 s of 2070 s (0.02%). list_snoc_item on roots and source_root_index_insert are measured out, not argued out.
• per-parse grammar analysis (the Carry the grammar digest and production index on the prepared grammar; share the parse-witness fixture trees warm #11032 shape) — parse_table_for_production_analyzed reuses prepared's GrammarFirstAnalysis; no per-file recompute, and front_end_prepare_calls=1 is the receipt.

Deciles 1–2 are elevated on both time columns, and that is fold position, not size. Tested directly on the earlier run by holding size fixed: among files of ≤90 tokens, those in the first quarter of the fold cost 247 ms median against 57.6 ms in the last quarter — a 4.3× position effect at the same size. It reproduces here (deciles 1–2 at 942/863 µs/miss against 228–303 for the rest). I have not fully attributed it; it is consistent with first-touch page-fault billing, which is already a rostered class, and I am not claiming it as a per-file constant.

What is actually there

~600 µs per token across the flat range, 80.5% of it in parse, at ~2.9 memo lookups and ~2.7 misses per token. A recursive-descent parser is normally well under 1 µs/token. This is a different defect class from the one the brief expected — not an invariant recomputed per file, but a cost that is constant per production attempt — and this PR measures it rather than fixing it. Where inside the parse that cost sits is NOT established here; see the narrowings below.

Second finding, not asked for

file_refusals=212 of 2243 — the count the relay EXECUTED and reported, not a classification applied afterwards. On the earlier run, classifying the parse/tokenize subset by signature (normalize binds the Rejected and returns in <10 µs while parse has spent seconds) gave 193 files and 9.8% of fold time. A tenth of the context fold front-ends files that contribute no root — the v2 front-end corpus frontier (v2.workflow.compile_door_ledger) paid for in full before it refuses. Flagging, not taking.

An earlier revision of this PR circulated 19.5% for that share. That was an artifact of reading only the alphabetical PREFIX of the walk (the dag/ tree); 9.8% is the whole-closure figure and is the one to use.

The seam

v2.compiler.program_assembly — the per-file front end becomes one declaration per phase:

• program_assembly_phase_tokenize(read, lm) -> Outcome<TokenStream>
• program_assembly_phase_parse(tokens, prepared, residue) -> Outcome<ParseArtifact>
• program_assembly_phase_normalize(parsed) -> Outcome<NormalizedTree>
• program_assembly_phase_token_count(tokens) -> Int

Each phase takes the PREVIOUS PHASE'S OUTCOME and binds it, so diagnostic merging and refusal propagation stay inside .dag. program_assembly_read_to_normalized_root_prepared is retained and is now defined by those three calls — entering the phases separately is the same computation, not a second front end beside it.

v2.compiler.00_compile — the matching seam: native_test_front_end_prepare() carries the once-per-fold invariant as NativeTestFrontEnd; native_test_context_absorb(state, read, root_outcome) is the non-front-end half of the step; native_test_context_state_empty() / native_test_context_finish(state, ingest). native_test_context_from_ingest is retained as the composition every clock-free caller uses — including the driver's own run_census, which wants per-file refusals and no attribution.

std.compiler_entry.SourceRootEvalDriver — a phase span is a REALIZATION fact (.dag is pure and cannot read a clock), so the driver that owns the clock enters each phase and times the call. It prints [native-context-split] per file (path, bytes, tokens, four phase spans) and [native-context-partition] once.

Counts sit beside times because a span alone cannot separate "this file is large" from "this file pays a per-file constant" — the whole question here. The token count is read OUTSIDE every phase span so it prices none of them.

No double counting

Per eager-raven-113's binding accounting ruling: these rows are their own relay on their own basis (basis: context_span) and partition the CONTEXT span. [native-cost-partition]'s exclusive rows are untouched — context remains one exclusive row of the driver wall, so the parent span still reconciles. The four phases partition the per-file span with 0.00 s unattributed; the 1.2% residual is the loop and the per-file eprintln and is reported, not absorbed.

Deliverable 3

1256 is a real closure, not the tree by another name. Independent oracle over the same two roots — seeds = every declared v2.test.* module (the native_route_universe_prefix) plus native_route_live_control_module, edges = the import headers each source declares, transitive to fixpoint — gives on the full tree: scanned 5578, closure_modules 2091 in 14 rounds, closure FILES 2089. Genuinely closure-scoped at ~37% of the tree, and the driver's own fold independently reports 2236 files over the same roots.

That also explains the previously unexplained gap of 4 between closure_modules and closure_reads: closure_modules counts module NAMES reached, closure_reads counts FILES ingested, and the difference is closure members with no ingested declaring file. native_lane_ingest_matches_closure is consistent with this: its "closure module was never ingested" arm is quantified over modules some source DECLARES, so a closure member with no declaring source in scope is not a refusal.

Correcting my own first report of this. I initially described the two such modules on the full tree as "declared nowhere". That was wrong in both cases, and they are wrong for different reasons, which is the part worth keeping:

• gunbc.recurring_failure_mode.roster is derived and gitignored (.gitignore:79), produced by gunbc.instruments.generated_artifact_gate main_wet from its sibling row files. It is declared by no committed source, so it is absent from a fresh clone and present in a working tree that has run the projection. "Declared nowhere" understates it: the declaring file is generated, not missing.
• v1.compiler.ownership is declared, at src/v1/ownership.dag. My oracle scanned only dag and src/v2 — the route's own two roots — so this is a module declared OUTSIDE the scanned roots, not an undeclared one.

The mechanism behind the gap is unchanged; only my characterisation of the instances was wrong. It matters to a reader because "declared nowhere" would send someone hunting for a missing module, and neither case is that.

The oracle is stated here as a method rather than committed as an artifact, deliberately: a .dag function under gunbc run cannot read the corpus, so the only available forms would be a second closure implementation in the corpus — a §3 fork of native_lane_closure_modules, which already answers this — or a hand-authored script, which §6 presumes scaffold.

Deliverable 2

Not in this PR, by instruction: parallelizing before the per-file cost is honest hides the defect. The seam landed here IS the fold-parallel seam — program_assembly_phase_* is pure in (read, lm, prepared, residue) — so D2 is a realization change over these same functions. The measurement changes its priority in both directions: there is no per-file defect left for parallelism to hide, but it would multiply throughput without touching an ~840 µs/token constant that §6 prices as a cost-shape defect in its own right, and ~10% of the parallel work would still be spent parsing files that refuse.

The boxed-carrier emitter defect this change surfaced

The first emitted crate failed to compile, and that is a finding rather than a footnote. v1.compiler.emit_rust needs_box_wrapping boxes a record field whose type rides a recursion cycle, and an Optional-cardinality field inherits that decision from its inner type; rust_field_carrier_final_type renders Box<T>. So the new carrier's residue: Diagnostics (= Optional<NonEmptyDiagnostics>, and NonEmptyDiagnostics is recursive) emitted as Box<Option<Rc<NonEmptyDiagnostics>>> while program_assembly_phase_parse takes it by value — front_end.residue.clone() cloned the Box: error[E0308] at src/main.rs:407, help "consider unboxing the value". Its sibling fields lm and prepared took Rc<..> and their .clone() was correct, which is the per-field divergence precisely.

The tempting summary "Optional fields box" is WRONG and is not what this filed: whether a field boxes depends on a whole-corpus recursive-type set, so the realization is not readable off the declaration a template author writes against. What did NOT catch it, each checked rather than assumed: gunbc run typechecked both modules under the whole closure and was green; cargo fmt --all --check passed; --required-regen regenerated and adjudicated the very file CONTAINING the defective template text and reached first_generation_equal, because regen compares emitted BYTES against the authority and never compiles the program those bytes describe.

Filed in this PR as gunbc.recurring_failure_mode.emitted_field_carrier_diverges_from_the_dag_type_authored_against, citing needs_box_wrapping and rust_field_carrier_final_type. Deliberately NOT filed under accepted_source_emits_uncompilable_target: there the EMISSION is wrong; here the .dag and its emission are both correct and the hand-authored template consuming them is wrong, so a repair to the emitter's type lowering fixes that class and does nothing for this one. Rung found at mitigatable (typed, located rustc diagnostic; the route refuses with EmittedCompilerBuildFailed); attainable ceiling structurally impossible, because the carrier a template must write is a pure function of the field's resolved type and the recursive-type set — the same join the emitter already performs when it renders the struct. The row carries its own uncertainty: one specimen, and the exposed population was not counted.

The memo counters, and the §3b divergence they are carried by

The context attribution left one question open and named the instrument for it. ParseTable already tracked memo_hits / memo_misses / memo_lookup_calls and parse_table_memo_stats already existed; nothing carried them out of the parse, so nothing could read them. This PR carries them out and prints them per file in [native-context-split] and as totals in [native-context-partition], read OUTSIDE every phase span so they price none of them.

DIVERGES WITH A STATED REASON (§3b), and the instruction it departs from is named. The direction given was to thread the counters "through ParseArtifact". This does not do that, for two reasons and the second is decisive:

1. ParseArtifact answers WHAT was parsed; the counters answer HOW that parse executed, which is a realization fact about one run and not a property of the tree. §3 keeps interface and realization as two facts, so a counter field would make every consumer of a parse result carry a measurement it has no use for, and would make the artifact's identity depend on how many times a memo happened to be consulted.
2. An Outcome's Rejected arm carries no ParseArtifact. A field there would lose the accounting exactly on the files that spend seconds in parse and THEN refuse — 9.8% of the fold, and the expensive case the instrument exists to read.

So ParseProductionMeasured carries the outcome and the accounting side by side and both refusal arms report, and parse_production_prepared, parse_module_prepared and program_assembly_phase_parse are retained as .outcome projections of it. One implementation, no fork, no caller pays a second parse to be measured, and the five existing parse_module_prepared test callers are untouched. One arm is spelled out rather than reused — program_assembly_phase_parse_measured cannot route through bind_outcome, because bind_outcome cannot carry a second value out of the bound function; its Rejected arm passes diagnostics through untouched and reports EMPTY accounting (zero because no parse ran, which is an answer, not a missing measurement), and its Accepted arm merges through bind_outcome_accepted, which is what bind_outcome itself calls.

What the counters measured, and what they do NOT establish

A MISS is one production attempt: the memo had no entry for that (position, production), so parse_expr_with_first ran. On the measuring run:

• memo hit rate 6.1% (400,160 hits of 6,557,074 lookups)
• 2.87 lookups per token, 2.69 misses per token
• parse time ÷ miss count = 270.7 µs, and by token decile both lookups/token (2.47–3.05) and that quotient (228–303 over deciles 3–10) are FLAT across the size range

What is established. Lookup volume is ~2.3 per token and does not grow with file size, and whatever the parse spends per attempt does not grow with file size either. Both are flat, measured, and load-independent as ratios.

What is NOT established, stated because an earlier revision of this section over-claimed all three:

1. The ~200 µs is NOT located in the miss arm. It is parse_nanos ÷ memo_misses — total parse time divided by a miss count. That numerator includes the common pre-dispatch path, hit handling and result construction as well as the miss-path parse, so the quotient sizes an average attempt's share of all parse work; it does not say which part of the parse holds the cost. Misses being 94% of lookups makes the miss path the largest single candidate, not a located one.
2. Grammar-sized work per attempt is NOT ruled out. The lookup count rules out grammar-sized lookups per position, and nothing more. Work proportional to the grammar performed inside one attempt would, against a single fixed grammar, present as exactly the flat per-attempt constant observed here. The earlier claim that this measurement excluded O(grammar) work was wrong.
3. The ParseTableRealization rebuild is a source operation count, not a measured cost. parse_table_record_lookup_call, parse_table_record_miss and parse_table_insert each construct a fresh 12-field record carrying four Maps, three times per miss — that is read off the source and is a suspect worth measuring first precisely because a record rebuilt per attempt would be flat in file size exactly as observed. Nothing here measures it.

Also withdrawn: an earlier revision said the 6.1% hit rate makes the parse memo "overhead rather than a win". That does not follow — a low hit rate can still be economical if each hit avoids an expensive subtree re-parse. The real question is avoided recomputation against lookup + insert + retention cost, and this PR does not measure it.

Subjects stay separate. 2.69 misses/token × 270.7 µs ≈ 728 µs/token is arithmetic over two figures from THIS run's own relay line, not a cross-run reconstruction; it is not offered as confirming any figure from any other run. Each figure binds its own run, population, phase, binary and input. Nothing here is evidence for a figure produced by a different run.

So the next lane LOCATES the cost rather than confirming a conclusion: a split inside one attempt on the emitted realization — common pre-dispatch / hit handling / miss-path parse / insert-and-result. One hazard is named up front for whoever takes it: parse_expr_with_first attempts NEST, so summing inclusive attempt durations would recreate the double-counting class just repaired at module grain. Report attempt spans as inclusive and never sum them into an exclusive partition, or subtract descendants through a nested accounting.

Launch gate: the emitted crate builds on THIS head

Applied to myself before any measuring run, on head 26628e291404c2920f9a7e4562f761039acac2bd (merge commit; origin/main 5f18d9f is an ancestor):

v2-native-route: emitted crate built — argv=["cargo", "build", "--release", "--manifest-path", "/tmp/gunbc-emit-compile/src_v2_compiler_00_compile_dag/Cargo.toml"] RUSTFLAGS="-D warnings" compiler=/opt/cargo/bin/rustc exit_status=0 warning_count=0 rustc=rustc 1.98.1 (48a229cea 2026-09-01); binary: rustc; commit-hash: 48a229ceaefd4985c50990b14116b6d856af0985; commit-date: 2026-09-01; host: aarch64-unknown-linux-gnu; release: 1.98.1; LLVM version: 22.1.8

exit_status=0 warning_count=0 under RUSTFLAGS="-D warnings", and the malformed control refused correctly on the same run (tokenize_lex_e1_unrecognized_char). This is the gate that matters for this branch specifically: it is the only check that catches the boxed-carrier class filed below, and .dag typecheck plus --required-regen are both green on code that fails it.

Verification

• both changed .dag modules typecheck under the whole dag + src/v2 closure
• the emitted crate builds clean, zero warnings, under RUSTFLAGS=-D warnings, and the malformed control refuses correctly (tokenize_lex_e1_unrecognized_char)
• --required-regen named exactly v1_compiler_emit_rust.rs as drifted — also the positive control that the emitter edit is live — and the regenerated file is committed
• the pre-landing merge of session/deep-swift-530 was resolved through the merge driver's declared repair route (both .dag authorities auto-merged; only the generated projection conflicted), verified at first_generation_equal=true AND fixed_point_equal=true, and confirmed both relays survive in the merged projection (native-context-split and deep-cat's native-prepare-split) — a textual merge previously dropped that line with no conflict marker. This PR then replays onto main and regenerates again, first_generation_equal=true.

One caveat on the subject: these numbers are over the FULL closure at this branch head, not the bounded subject, so they are NOT directly comparable to the 1400.79 s bounded figure — same host, different subject. The full closure turns out to be the better subject for D1 anyway: the context fold completes and reports before any per-module prepare, so it needs no reduced claim population to finish.

🤖 Generated with Claude Code

https://claude.ai/code/session_019gufjy7FmMzjTRwAnUgZcZ

🤖 Generated with Claude Code

https://claude.ai/code/session_01RtujL6ZdnmmcCBMihVRZ8z

[gunbai-bot]

Review 67267 verified and fixed at 68a309b: the merge resolution had left a truncated 'fn program_assembly_read_to_normalized_root_prepared(' header at program_assembly.dag:107 (the conflict block cut main's side mid-declaration; the real definition is the one at the later line). Deleted the fragment; the emitter mirror is being regenerated by the transaction on this head, then CI re-runs. — sent from eager-raven-113

[gunbai-bot]

Gatekeeper response to review 67270 (REQUEST_CHANGES), verified: both findings hold on this head. (1) program_assembly_phase_tokenize / _parse_measured / _normalize are a second spelling of the .dag front-end order beside v2.compiler.staged_front_end (and the emitted driver's for-loop a third); the diff declares the divergence and files a row with a capability trigger, but DESIGN §5/§6 say a dissolution condition does not authorize the debt and no scaffold lands by author declaration — the admission must be external. (2) program_assembly_memo_hits/_misses/_lookup_calls and program_assembly_measured_outcome have no .dag consumer; only the hand-written driver template reads them (§3c dangling from the model's side). Disposition: this recut is HELD pending an operator decision, which I am asking for now — either an explicit operator admission of the interim fork (recorded here, with the consolidation on staged_front_end as its dissolution), or the PR closes and perf item C's subject folds into the parallel-realization item (#11385's slice 2), which needs the same per-stage entry and should build it on staged_front_end directly. No further pushes until that answer. — sent from eager-raven-113

[gunbai-bot]

OPERATOR ADMISSION (external to the diff, granted in-session 2026-09-17): the interim fork of the .dag front-end order in program_assembly_phase_tokenize / _parse_measured / _normalize (and the emitted driver's per-phase loop) is ADMITTED as a scaffold on the condition that it carries its entry — which it does: gunbc.recurring_failure_mode.realization_respells_a_modeled_folds_sequencing_to_instrument_it, with the dissolution named at capability grain (a prepared-grammar, resolve-free, per-stage entry on v2.compiler.staged_front_end, after which these phases become projections of that authority and the divergence comment is deleted). Review 67270's second finding (program_assembly_memo_hits/_misses/_lookup_calls and program_assembly_measured_outcome consumed only by the hand-written driver template) is classified under §3c as consumed-by-execution through the realization (the emitted driver is the consumer), with the same entry as their dissolution: the per-stage entry carries its own accounting and these four are deleted with the fork. Landing resumes: mirror regeneration on this head, then receipt, then LAND. — sent from eager-raven-113