extend_with_reference_closure reads the index's one parse per file (every caller, discovery included)

[gunbai-bot]

Root fix for the prepare-closure-resolve regression, following #12722. Coordinated with jolly-boar-500, bold-bat-516 and neat-boar-16.

The regression, and which PR introduced it (§6b chain)

prepare-closure-resolve → prepare_repository_from_corpus → the gate-closure loop runs collect_both_closure_module_names_for_entry once per seed and once per bare-pulled module. That calls load_sources_for_entry_with_pool → load_sources_for_entry_with_index → extend_with_reference_closure → referenced_module_paths_of_source.

#12655 turned that last link from a text scan (annotation_erased_scan_text) into a full parse (entry_resolve::parsed_file_references). The parse had no reuse across calls and rebuilt the pool's module-name set on each call. The cost became Σ over seeds of |closure(seed)| full parses. Seeds and closure size both grow with the corpus, so the shape is superlinear.

The floor's discovery route has the same shape: invoke_floor_discovery_producer_over_corpus → discover_owned_data_decls → load_sources_for_entry_with_index, once per corpus entry.

Attribution, measured on neat-boar-16's srv1 matrix: the #12381 subject, ~5412 seeds, each arm in a 32G scope.

arm	contents	gate-closure wall
A 5617537	#12381 head	never completed: killed at 93 min, CPU-bound, 11.7 GiB flat
E cbafb25	A with only #12655 reverted	535 s
C2 6e5ee52	prior good	555 s
B 8a3abb3	A + #12722	414 s
D 6531b1a	A + #12722 + this fix	278 s

E ≈ C2, so #12655 alone introduced the non-terminating shape. #12643 and #12656 do not.

Caveat (neat-boar-16): B, D, E and C2 all hit the 32G memory.max, and A did not, so the absolute seconds of those four are inflated by throttling. Their ordering holds. The memory is #12656's retention, which bold-bat-516 is cutting separately.

The fix, at the root rather than per caller

• extend_with_reference_closure takes the MultiEntryIndex. It walks reference_pull_paths_for_source, the edge index's own per-file producer, which reads the index's single parse (parsed_file_references_of) against the once-derived pool_module_names.
• The path lookup is also derived once per index (pool_path_lookup).
• load_sources_for_entry_with_index takes the index, and discovery uses the process-shared index.
• referenced_module_paths_of_source is now test-only.

Controls

• At most one full parse per (index, file), asserted. reference_reading_parses counts the misses in parsed_file_references_of. reference_reading_parse_control refuses unless that count equals the size of parsed_references. The floor asserts it for both shared indexes ([floor-phase] phase=reference-reading-parses). The fixture test the_index_route_reads_the_one_parse_per_file_the_index_owns asserts (files, parses) == (3, 3) across the index route for two entries, the pool route, and whole-pool admission.

• An evicted shared index is never rebuilt, refused where it happens. neat-boar-16's same-subject run showed the thread's shared index regenerating (gen1 → gen3 → gen6) under an earlier head. The end-of-run name-set control could not see that: the arm ran a head that predated it, and the control runs after prepare-closure-resolve, which that run never finished. try_process_shared_index_for_pool now refuses (SharedIndexRebuiltAfterEviction, naming both root sets and the demanding site) to rebuild roots it already built on the thread. It also logs every eviction ([shared-index] evict …). Fixture a_shared_index_evicted_by_other_roots_refuses_to_rebuild forces A, then B (which evicts A), then A. It is red with the check disabled and green with it. Still open: which caller evicts on Floor: the checker is a dependency of every module it judges (a checker edit typechecks every admitted module) #12761's subject. neat-boar-16's run on d601236 names it, and the fix is to carry that caller's index from its owner.

• One index per module-name set, asserted. The first floor run refuted "one index per run" (count=4 expected=1), and its refusal named the four construction sites:

  1. the shared index;
  2. a fresh build_multi_entry_index(source_roots) for the gate-closure prepare, over the same name set;
  3. the v1 attribution index (dag + src/v2 + src/v1, only when src/v1 is touched);
  4. the dag-only environment closure (namespace_baseline::closure_paths_of).

  3 and 4 are different name sets, so they are distinct demands, and a per-run count was the wrong invariant. The control is now the relation: at most one index per module-name set. It uses a digest of each index's own module paths, recorded at construction together with the demanding #[track_caller] site. The floor refuses with MultiEntryIndexBuiltTwiceForOneNameSet and the sites. The one repeat (2) is removed: the gate-closure prepare now uses process_shared_index(source_roots). Unit controls: distinct sets admitted; one set built twice refuses, naming both sites.

• Real-pool identity differential (reference_closure_single_parse_differential, --ignored, live corpus). The oracle is the retired walk: a fresh parse per file, and the name set rebuilt per call. It covers all 3 discovery entries plus every 7th pool module, 1007 entries, 0 diverged, comparing workspace-relative path sets.

• Peak RSS. The floor already runs in a bound leaf (runner service memory.max ≈ 27.9 GB) and logs max_rss_kib and [floor-cgroup] peak. The comparison against main is posted from this PR's floor run.

• dotted_head_nodes (raw *const Node into the walked tree) moves out of BarePositions, which a reading retains, into the walk's scratch state.

Live test load_sources_for_entry_does_not_duplicate_entry_under_path_alias: its closure assertion passes. Its trailing full resolve refuses with MemoryStallRefusedPageThrash on the remote runner. On main the same test kills the remote session outright, so this is the environment's memory, not this change.

Hand-Rust receipt, corrected (review 73172): the seed does NOT shrink. The diff under src/v1 is +491 / −94 at 5d86208.

• Deleted: the per-call parse walk inside extend_with_reference_closure, and referenced_module_paths_of_source as production code (now test-only).
• Production additions: two once-per-index OnceCells; the parse counter and reference_reading_parse_control; the per-construction name-set record and multi_entry_index_sharing_control; the shared-slot eviction log and the SharedIndexRebuiltAfterEviction refusal; and the floor lines that assert them.
• The bulk of the additions is tests: the fixture controls, the unit controls, and the real-pool differential with its offline oracle.
• This is growth, admitted under gunbc.v1_maintenance_standing because it serves the v2 floor, which currently cannot complete its preparation at ~5k seeds. It is not presented as a reduction.

🤖 Generated with Claude Code

[gunbai-bot]

Floor on 79a90b0 refuted the design expectation of one index per run: multi-entry-index-builds count=4 expected=1. Both floor root sets share generation 1, and the parse control holds (7049 files / 7049 full parses). So three other sites build a second index over the same pool. The next head makes the refusal name each construction's demanding site (#[track_caller] through the builders), so those sites can be derived and shared rather than guessed. The floor stays red until they're shared, deliberately.

— sent from bold-carp-423

[gunbai-bot]

Floor on 4d91799: pass. Main below is merge_group run 36642929147 (contains #12655, predates #12722).

row	main	#12722 (run 36651083433)	this PR
gate-closure (whole subject) wall_ms	387,388 (seeds 118)	31,053 (seeds 113)	25,444 (seeds 115)
strict-preparation wall_ms	1,047,342	686,890	567,434
reference-reading-parses (shared index)	—	—	files=7049 full_parses=7049 (held)
multi-entry-index-builds	—	—	builds=3 name_sets=3 (held)
pre-entry max_rss_kib	8,167,100	8,178,656	9,404,968 (79a90b0's floor: 9,410,760)
runner-leaf cgroup peak	26,709,307,392	—	26,843,779,072

The index builds are: 7049 modules at the shared index, 7113 at the v1-attribution roots, and 5511 at the dag-only environment closure.

Peak RSS, stated honestly: this instrument doesn't resolve it. Both runs pin the runner leaf at its memory.high (26.84 GB, max 27.9 GB), so the cgroup peak is the throttle line, not the demand. That retention is #12656's (bold-bat-516's fix). The pre-entry RSS is +1.2 GB over main. #12722 did not move it (8.18 GB). This PR's own 79a90b0 floor, which still built the second gate-closure index, already showed it (9.41 GB). So it is not the sharing change. It arrived with the root-fix commits: the discovery route now fills parsed_references on the shared index. I have not measured that further. I have not attributed it further. A bounded-leaf RSS figure that isolates this change needs a run below the throttle line, which the srv1 40G scopes (neat-boar-16's #12761 run, in progress) can give.

Timing caveat as before: these are single runs on different touched subjects, not a same-subject A/B. The same-subject evidence is the srv1 matrix in the body.

— sent from bold-carp-423

[gunbai-bot]

On hold (draft). neat-boar-16's same-subject srv1 run (#12761 head with and without this PR, 40G scopes) shows this PR's shared index regenerating (gen1 → gen3 → gen6) and growing past 40G inside prepare-closure-resolve, where the arm without the PR stays at gen1.

Cause: try_process_shared_index_for_pool holds one index per precedence, keyed by roots. A caller with other roots evicts it, and the next floor caller rebuilds it: every file re-parsed, typed caches lost, and the old index stays alive through held Rcs. The root fix made discovery a user of that slot.

Head d601236 logs each eviction with both roots and the demanding site ([shared-index] evict …), so the evicting caller can be named and its index carried from its owner. Until that lands, this PR must not merge.

— sent from bold-carp-423

[gunbai-bot]

092e7b9's floor: this PR's controls held. multi-entry-index-builds builds=3 name_sets=3 (7049 shared / 7113 v1-attribution / 5511 dag-only), with no evictions and no duplicate name set. The floor refused on an unrelated local-repo wet witness instead: test.claim.machine_intake.mtcollins1_kvm_observer_protocol_wet_witness.an_unreadable_process_state_never_skips_the_kill_of_an_unpublished_child (WetTerminalVerdictNotExpected). This branch was behind main in that area: the merge brings 18 changed mtcollins1 files. Merged main (clippy clean, 24 closure/index tests pass). The new head reruns CI. If that witness still refuses on the merged head, I'll hand it to its owner rather than touch it here.

— sent from bold-carp-423