Live-pool unit claims share one whole-pool index per test process

[gunbai-bot]

Work item adhoc-d23b2a7d-d80 (§3 witness rule on the Rust unit suite). Follows neat-ant-823's #12450 measurement. The fix shape is neat-boar-16's ruling (option (b)).

What was slow

Each slow required_regen_host / required_floor_runner claim evaluates a small .dag model. It resolves that model through process_shared_index([dag, src/v2]), a whole-pool index over about 6,950 .dag files. That index is thread-local, because it is Rc-based. libtest runs every test on a fresh thread, even under RUST_TEST_THREADS=1, so every claim rebuilt the index and re-typechecked the shared prefix. One fact was paid about 16 times, and its least common ancestor is the test process (DESIGN §2, demand minimization).

Why not supply a narrower pool

• Entry-scoped loader (option (c)). It can't be used: load_sources_for_entry_with_pool resolves bare names through a census of the whole pool (admit_pool_bare_references). No entry resolves without the whole pool.
• Checked-in per-entry module list (option (a)). Refused in the ruling: it would be a second authority for the import graph (§3).

Change

• cli_run::entry_resolve on_live_pool_thread (test-only). One thread lives for the whole test process and holds the index in its thread-local slot. A live-pool claim runs its body on that thread. Resolution itself is unchanged: the same process_shared_index still builds the same pool on first demand. Only the number of builds changes. A panic in the body is caught on the worker and re-raised on the caller's thread with its original payload.
• 16 claims now run their bodies there:
  • the 7 regen_convergence_host_instrument_tests claims from the Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450 table
  • 2 regen_emission_scope_tests model claims
  • 3 regen_affected_set_tests claims
  • 2 regen_round_cost_tests model claims
  • 2 pure_producer_share_tests claims
• Real live path. Every one of those claims still executes the real [dag, src/v2] pool, so no boundary loses its real execution. The route claims that already existed stay as they were, e.g. live_tree_controls_land_on_the_measured_arms and a_retirement_against_the_real_base_read_admits_and_a_growth_refuses.
• New controls in entry_resolve live_pool_thread_tests:
  • two_claims_on_the_live_pool_thread_share_one_index asserts the route. Two calls see one index generation, and a fresh thread sees a different one, which is the red the old shape produces.
  • a_panic_on_the_live_pool_thread_reds_the_caller_and_the_thread_survives checks that a failing claim still fails its own test with its own message.

Left alone

• the_emitted_tree_package_graph_decodes_... resolves over the regen pool (src/v1 + dag), which no other wrapped claim uses. Moving it would evict the [dag, src/v2] slot and save nothing.
• filter_in_branch_... uses the witness-roots module path index, not this one.
• heartbeat and corpus_scope are outside the regen/floor scope.

Evidence

• Wall times: pending. BuildBuddy amd64 runners SIGKILL the live resolve (probe: index build 48–51s, then killed), so they cannot give honest walls. The before/after suite wall comes from this PR's rust_unit_tests step on the hosted arm runner, compared with Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's 30m39s. The srv1 per-test timings are on the operator's list.
• Mutations. The converted claims keep their assertions unchanged, and a failure inside them re-raises on the caller: the panic-propagation control checks that directly. Per-test mutation receipts are not yet run.
• Memory risk (to watch on CI). The index now stays resident for the whole test process under the 12 GiB cgroup, instead of being freed after each test.

Hand-Rust receipt (v1 seed)

• Admission. Serves the v2 program (gunbc.v1_maintenance_standing v1_seed_standing): it cuts the cost of the unit lane that the drop rust_unit_tests_off_the_merge_path is waiting on.
• Growth. Test-only (#[cfg(test)]): on_live_pool_thread, release_per_entry_graphs_on_live_pool_thread and live_pool_thread_tests in cli_run::entry_resolve. No production seed code changes and nothing is emitted. No scaffold is deleted, so there is no path to cite.
• Seed census. No census run: the production seed is unchanged.

Do not merge; landing goes through neat-boar-16.

🤖 Generated with Claude Code

[gunbai-bot]

Head 3905160. The first CI run of this PR stalled in the unit step (50-min timeout) just after the live-pool block began. Two causes, both fixed:

• Route-control fixture. two_claims_on_the_live_pool_thread_share_one_index placed its fixture under the system temp dir, and the module-graph facts refuse paths outside the workspace. The fixture now sits under target/.
• Resident set. The live-pool thread kept every claim's resolved entry graphs.

Measured with the same 39 claims, --test-threads=1, 24 GB BuildBuddy runner, peak VmRSS sampled at 1s:

	wall	peak RSS
main (index per test)	1789s	7.1 GiB
this PR, graphs held	333s	10.6 GiB
this PR, per-entry graphs dropped after each claim (head)	387s*	8.8 GiB

*This run also included the 2 new live_pool_thread_tests (41 tests total).

The pool facts, censuses and capped typed cache stay shared. Whether the resident pool fits the full suite under the 12 GiB cap is what this CI run measures.

— sent from warm-wolf-829

[gunbai-bot]

Memory bound and margin, for neat-boar-16's pre-landing check.

• Bound. The rust_unit_tests step binds a cgroup leaf at memory.max = gunbc.ci_spec hosted_arm_runner_cgroup_memory_max = gibibyte(count: 12) = 12,582,912 KiB (gunbc.compiler_gate_workflow compiler_gate_unit_test_command). There is no swap, so exceeding it is an OOM kill of the test process, and under pressure it can thrash before that, which is what the first run's stall looked like.

• Measured peak at head. 9,264,744 KiB (8.84 GiB) of VmRSS, sampled at 1s, for the 41 live-pool claims run alone (--test-threads=1, 24 GB BuildBuddy runner). Margin: 3,318,168 KiB = 3.16 GiB (26% of the bound). main's per-test shape peaked at 7,488,532 KiB (7.14 GiB) on the same claims.

• What this does not cover. That figure is the live-pool block in isolation. In the full suite, the worker's index stays resident after the block while later tests build their own indexes. The full-suite peak under the real cgroup is what this CI run (7c2d8ce) measures; I'll post it when the step finishes.

• What holds the extra ~1.7 GiB. I haven't measured it by component. The candidates, all per-process on the worker thread, are:

  • the retained [dag, src/v2] MultiEntryIndex: pool_parse heads nodes for ~6,950 modules, the per-root tree_bare_census, pool_qualified_fill, module-graph facts, and typed_module_cache (capped by the host-budget-derived typed_module_cache_cap);
  • on current main, gunbc run: name the pre-entry phases; produce the selection tier's reference edges per closure file #12471's reference-name index and per-file reference rows.

  Per-entry resolved graphs are already dropped after each claim (release_per_entry_graphs_on_live_pool_thread); holding them was the 10.6 GiB shape.

• If the full-suite margin is thin, the next lever is to release the worker's pool slots once the contiguous live-pool block ends, so the index doesn't outlive its consumers. It costs nothing in the block itself. I'd only add it on evidence, not speculatively.

— sent from warm-wolf-829

[gunbai-bot]

Re review 72425's unverified item: the required_regen_host.rs churn is only the wrappers plus rustfmt re-wrapping, checked mechanically against the merge base.

Method.

1. In the head file, drop each inserted wrapper: the crate::cli_run::on_live_pool_thread(|| { line and the }); directly before that test fn's closing }. The script asserts that shape for every wrapper.
2. Strip all whitespace from both sides and compare.

Result.

• required_regen_host.rs: 14 wrappers. Identical to the merge base once trailing commas are ignored; the only non-whitespace differences are trailing commas rustfmt added or removed when it re-wrapped lines at the deeper indent.
• required_floor_runner.rs: 2 wrappers. Byte-identical ignoring whitespace.

No test body, assertion or call changed.

— sent from warm-wolf-829

[gunbai-bot]

Head b29cf8d. Mutation evidence, and the fix for the unit-step stall.

The stall, reproduced under the exact bind

A 24 GB BuildBuddy runner, the unit binary run whole, as root inside a cgroup leaf with memory.max = 12884901888 (the gunbc.ci_spec hosted_arm_runner_cgroup_memory_max value), sampling memory.current every 5s.

1. Before (7c2d8ce). oom_kill 1 on the first live-pool claim. The process already held 7.6 GiB when the claim started: freed-but-retained heap from heartbeat_tests::render_heartbeat_line_mirror_matches_seed_oracle on its own thread. glibc keeps a finished thread's arena, and the long-lived worker allocates from a different one, so the pool was built on top of that residue.
2. Trim before and after each claim (the existing trim_retained_heap instrument, malloc_trim(0)). 1.4 GiB under the first claim, and the whole block ran in ~5 min. It was then oom_kill 1 on required_regen_host::tests::the_emitted_tree_package_graph_decodes_...: the ~5.2 GiB pool outlived its block, and that test built its own regen-pool index on top of it.
3. Release when the block ends (head). The worker exits after LIVE_POOL_IDLE_RELEASE (2s) with no claim, dropping the pool; the next claim rebuilds it. Claims inside the block arrive milliseconds apart. The threshold only sets how often the pool is built, never what a claim answers. Send and exit share one lock, so no claim can be lost to the exit. New control: the_live_pool_is_released_when_its_block_of_claims_ends.
   • Full suite under the bound: 1006 passed, 7 failed, 156 ignored, 529.6s on that runner, memory.events: max 0, oom 0.
   • The 7 failures:
     • six are on Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's list of reds on main;
     • generated_artifact_boundary_host::tests::the_authority_entry_resolves_from_a_non_root_cwd isn't on the live-pool path and passed on hosted CI at 7c2d8ce, so I read it as a runner difference.
   • The hosted CI run on this head gives the comparable wall against Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's 30m39s.

Mutation per converted test

One production mutation per boundary, each run on the live-pool thread (24 GB runner, --test-threads=1). Every converted test reds; its unconverted neighbours stay green.

mutation	reds
M1: convergence model entry path deleted (workflow/regen_convergence_transaction.dag → a nonexistent file)	all 7 regen_convergence_host_instrument_tests claims
M2: WholePopulationScope ↔ ScopeUnlocatable swapped in the host-built scope value	host_selection_and_model_selection_agree, the_model_selects_nothing_on_the_unlocatable_arm
M3: the reverse closure seeded from nothing, unlocatable list dropped	live_tree_controls_land_on_the_measured_arms, host_walk_and_model_fold_agree_on_the_fixture_graph, an_unlocatable_edited_path_refuses_and_selects_nothing
M4: host receipt field value→valu; .dag owner refusal count(owners) == 0 → == 99	host_built_receipt_renders_through_the_model, live_module_mirrors_have_owners_and_removed_shell_owner_refuses
M5: roster judge_edit judges the head against itself	unimported_bare_provider_roster_edit_is_judged_across_frames, a_retirement_against_the_real_base_read_admits_and_a_growth_refuses

— sent from warm-wolf-829

[gunbai-bot]

Suite wall on the hosted runner, head b29cf8d (rust-unit-tests job 109245032999, hosted arm, memory.max read back 12884901888).

• Test run finished in 532.27s (8m52s), against Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's finished in 1838.58s (30m39s) on the same runner type and bound: −71%. Release build 6m15s; whole job 15m29s.
• 1007 passed; 6 failed; 156 ignored, no stall, no OOM. All 6 failures are already on Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's list of reds on main: construction_authority_graph_tests::wall_now_authority_graph_is_total, the two floor_skip_frontier_tests::rename_*, nfr_tests::nfr_roster_receipt, compiler_tests::function_value_named_application_controls_witness, memory_governor::tests::whole_corpus_compile_another_repositorys_projection_refuses.
• generated_artifact_boundary_host::tests::the_authority_entry_resolves_from_a_non_root_cwd passes here. The earlier BuildBuddy failure was a runner difference.
• The live-pool controls pass: route, release on block end, and panic propagation.

— sent from warm-wolf-829

[gunbai-bot]

Head 101130a: the pool is released by ownership, not a timer, per neat-boar-16's design point.

• The end signal. libtest has no end-of-block or teardown hook: each #[test] is its own job, and merging the claims into one test is the conjunction §3 rules out. So the block's end never reaches the pool, but the demand that conflicts with the pool does. Every whole-pool index build (try_process_shared_index_for_pool, build_multi_entry_index, the primary-precedence builder) first calls entry_resolve::yield_live_pool_before_building_another. That ends the live-pool thread, which drops its index and every thread-local cache built over it, joins it, and trims.
• Nested builds are exempt. A claim's own nested builds are exempt: builds on the live-pool thread, and builds by threads a running claim spawned (LIVE_POOL_BUSY). The test process holds one live whole-pool index at a time.
• The timer is only a backstop. LIVE_POOL_IDLE_BACKSTOP (10s) only bounds how long the pool outlives its block when no later test builds a pool of its own. The memory bound does not depend on it for any test that builds one, which is the OOM case measured before. What would replace the backstop entirely is an end-of-block signal from the harness, e.g. a custom test harness (harness = false) with suite-level fixtures. That's out of scope here.
• Control. another_threads_pool_build_releases_the_live_pool: back-to-back claims share one index generation, and after another thread builds a pool, the next claim's generation differs.
• Cost of trim_retained_heap per claim. It runs before and after each of the 16 claims. It's not isolated in a timer, but the full-suite wall moved 529.6s → 521.3s across the change, which is within run-to-run noise.
• Full suite under a 12 GiB memory.max leaf (24 GB BuildBuddy runner): 1006 passed, the same 7 failures as before (6 of them on Rust unit tests run in the clippy job, non-required until measured (step 1 of retiring rust_unit_tests_off_the_merge_path) #12450's list of reds on main; the_authority_entry_resolves_from_a_non_root_cwd is BuildBuddy-only), 521.27s, memory.events oom=0, sampled memory.current peak 10.1 GiB (1.9 GiB, 16%, under the bound). The hosted run on this head follows.

— sent from warm-wolf-829

[gunbai-bot]

Re review 72467 (of b29cf8d):

• Hand-Rust receipt. It's in the PR body under "Hand-Rust receipt (v1 seed)". The change is test-only (#[cfg(test)]), the production seed is unchanged, nothing is deleted, and no census was run because no production code moved.
• Idle release. The review describes the pool as released after an idle gap. That was the reviewed head. Head 101130a replaces it: the pool is released by the conflicting whole-pool build (yield_live_pool_before_building_another), and the idle timer is only a 10s leak backstop. The control is another_threads_pool_build_releases_the_live_pool; details are in the comment above.

— sent from warm-wolf-829

[gunbai-bot]

Hosted rust-unit-tests at head 101130a (job 109254341931): finished in 521.73s against #12450's 1838.58s (−72%). 1007 passed; 6 failed, and the 6 failures are the same reds on main listed above. No stall and no OOM under the 12 GiB bind.

— sent from warm-wolf-829