Root-cause disk-tier repeat-resolve memory growth (>8GiB OOM on second resolve through v1 disk cache seam)

[briansrls]

Root cause

The resolved-graph disk seam suppressed materialization-provider routing on the probe (read) direction only. The store (write) direction never consulted the same flag.

So when the provider's own authority closure (dag/std/materialization_provider.dag) was being resolved during bootstrap, that nested resolve reached the store branch, which called resolve_closure_request_key_from_digests → materialization_provider_ctx(). The ctx memo slot is written only after build_materialization_provider_ctx_cold() returns, so the re-entrant demand read an empty slot and rebuilt the whole provider closure — a fresh nested resolve, which re-entered the store again, unbounded.

The observed shape was an OOM kill, not a diagnostic.

Evidence by execution

Reproduced on unmodified origin/main (7024637) via the already-merged sibling test same_subject_resolves_share_one_graph_store_hits_v2_disk — SIGKILL, zero diff to the seam files. Phase-RSS probes (temporary, removed before this PR's final state) located it exactly:

test:after-index1-build:                                     9 MiB
seam:enter depth=1 subject=085f8ffb30a15f7e nsources=3:     11 MiB
seam:before-probe:                                          11 MiB
seam:after-probe:                                           11 MiB
store:enter depth=1 suppressed=false:                       13 MiB
seam:enter depth=2 subject=fd663d39a4ac42de nsources=34:  1064 MiB
store:enter depth=2 suppressed=true:                      1251 MiB
   <SIGKILL>

Window/denominator for those numbers: single process, one cargo test --release run of that one test on a ctrl-build remote host whose available-memory signal read ~1.9–2.3 GiB ([floor-drain] degraded_budget_source: cap=1895..2285 source=/proc/meminfo MemAvailable). RSS is field 2 of /proc/self/statm × 4096. The absolute kill threshold is therefore host-dependent — this host dies at ~1.3 GiB where the reporting session saw >8 GiB — but the shape (unbounded recursion, ~1 GiB of resolved graph per level) is the same defect.

The decisive readings:

• depth=2 with nsources=34 — the nested resolve is a different, larger closure (the provider's own), not a repeat of the 3-module fixture.
• depth=2 suppressed=true — the suppression flag was correctly set; the store branch simply ignored it.
• The backtrace at depth 2 confirmed the cycle: resolved_graph_from_sources_with_index → resolve_closure_request_key_from_digests → build_materialization_provider_ctx_cold → resolve_entry_graph_shared → resolved_graph_from_sources_with_index.

Refuted hypotheses (recorded so nobody re-derives them)

• Not the decode/rewire path. My first hypothesis was that serde drops Rc sharing and the rewire_*_parent_links functions rebuild it badly. Refuted by execution: probes placed inside decode_v3_payload_from_file never printed at all — decode is not reached before the process dies.
• Not install_cross_process_materialization_hit (the candidate named in the dispatch). It is a memo insert plus three Rc clones; it is never reached.
• Not the second resolve. The dispatch described growth "on the second resolve against a fresh MultiEntryIndex". Measured, the growth is entirely inside the first resolve — test:after-g1 never printed. The second resolve is where a reader notices, not where the memory goes.

The fix

Two changes, both in the seed:

1. The store direction honours the same bootstrap suppression as the probe (cli_run.rs). The flag names a window in which provider routing may not be re-entered at all; honouring it in one direction only left the recursion one seam away. The skip is counted (provider_bootstrap_store_skip_count), not silent — the window is bounded (only the provider's own closure resolves inside it) and its frequency stays observable, per DESIGN §5 (a failure arm must refuse, never widen; a degradation must stay countable).

2. Re-entrant provider-ctx construction is a typed, located refusal (materialization_provider_consumer.rs). Suppressing routing at each known seam is validation — it concedes the re-entrant state is writable and depends on remembering every call site. The wall makes the recursion unreachable regardless of which caller re-enters, and fails loud and located rather than as a dead runner (DESIGN §5, construction over validation).

Change 1 alone fixes the observed defect. Change 2 is what stops the class coming back as an OOM if a future caller re-enters from a seam nobody guarded.

Status

DONE, green by execution:

• same_subject_resolves_share_one_graph_store_hits_v2_disk — SIGKILL on origin/main, passes with this change (41.73s).
• repeat_resolve_through_disk_seam_does_not_reenter_provider (new) — asserts the bootstrap store skip is counted and the provider ctx is built exactly once across a repeat resolve through the disk seam.
• reentrant_provider_ctx_construction_refuses (new) — drives the wall directly; fails if the wall is removed.

BLOCKED — origin/main does not compile. The dashboard asked me to merge main into this branch. I did; the merge conflict was genuine and is resolved (main landed a placeholder comment where my tests go — I kept both, and rewrote the placeholder to say the finding is now root-caused rather than pending). The resulting tree then stopped compiling, inherited, not caused by me:

error[E0061]: this function takes 9 arguments but 6 arguments were supplied
  src/v1/stage0/src/v1_compiler_emit_rust.rs:14088   v1_generic_params_needing_clone_bound(...)  // 6 args
  src/v1/stage0/src/v1_compiler_trait_derive_emit.rs:321  pub fn v1_generic_params_needing_clone_bound(  // 9 params

How that attribution was derived, because it matters: I read both sides out of origin/main blobs with git cat-file, not out of my working tree. The caller is ordinary code, not text inside an emitter string literal. My branch differs from origin/main in exactly 3 files and none of them touch either file. CI on this PR reproduces the identical E0061 independently (run 30784776982, build job). Caveat I will not paper over: I never executed a standalone build of a clean main checkout — my observation is on the merge commit, whose copies of both files are byte-identical to main.

Suspected but not bisected: #7708 added bounds/type_decl_items/source_indices to the callee; #7709 is the most recent commit touching the caller file and landed after it. That shape matches a stale copy overwriting an updated call site. Someone should confirm before acting. Reported to still-bat-561 at high priority.

Module test status, measured BEFORE the merge (so these numbers stand): serial run, --test-threads=1, 20 passed; 3 failed; 664.22s. The 3 are one real failure plus two collateral:

• provider_disposition_four_arm_matrix — panics at resolve_cross_process_cache_test.rs:867, "v3 disk hit must install into share". This is the only real failure.
• repeat_resolve_through_disk_seam_does_not_reenter_provider (mine) and same_subject_resolves_share_one_graph_store_hits_v2_disk — both die on GUNBC_RESOLVED_GRAPH_CACHE_DIR env mutex poisoned. The first test panicked while holding CACHE_ENV_MUTEX, poisoning it; everything downstream that takes that lock dies. Pure collateral, not independent failures.

Running serially refuted my earlier parallel-interference hypothesis, and it also refuted my other one: this is not memory pressure.

Is provider_disposition_four_arm_matrix my fault? NO — settled by execution. Baselined on 7024637b05 (my pre-merge base) in a clean clone, running that test in isolation:

thread '...provider_disposition_four_arm_matrix' panicked at
  src/v1/tests/src/resolve_cross_process_cache_test.rs:865:9:
v3 disk hit must install into share

test result: FAILED. 0 passed; 1 failed; 730 filtered out; finished in 32.28s

Identical assertion, unmodified tree, no involvement of this change (line 865 vs 867 on my branch is the two-line import shift). This failure is pre-existing on main.

So the accounting for this PR is: zero test regressions. The module's 3 failures are one pre-existing real failure plus two collateral deaths on the mutex it poisons.

That pre-existing failure is itself a finding worth routing. provider_disposition_four_arm_matrix asserts Rc::ptr_eq across graphs from two different MultiEntryIndex instances plus exactly one decode — i.e. a process-level share that does not appear to exist (install_cross_process_materialization_hit installs into the per-index resolved_graph_memo). Whether the test or the implementation is wrong is an open question I did not chase; it is not this defect and not this PR.

Why it went unnoticed — and what is and is not new here. The non-execution itself is not a discovery: it is a deliberate operator ruling, recorded at gunbc.commit_workflow commit_gate_rust_suite_removed_disposition (2026-07-11), which removed the nextest run half (~37 GiB, red on main, non-required) as a permanent tolerated red under DESIGN §5 — and notably refused an escape hatch, deleting the heavy arm from the model rather than hiding it behind a skip toggle. What I observed is its deliberately narrower successor, ci_v1_compiler_tests_compile_gate_note (re-enrolled 2026-07-31 on #7490, compile-only by design, ~41s). The step name "compile gate; no test execution" is that gate honestly saying what it is. I initially reported this as an undiscovered coverage gap; that framing was wrong and is corrected here.

The new part is the blast radius, and it is what this PR is evidence for. The ruling priced the removal in nextest's own costs — size, runtime, a tolerated red. What was never priced is which tests fell into the non-executing set. Two receipts from this work:

1. the test that reproduces the OOM defect I was dispatched to root-cause lives in that module, so the defect was structurally unobservable rather than merely unnoticed;
2. provider_disposition_four_arm_matrix is a genuine failure that has been sitting on main unobserved, in the same module.

So the removal did not only drop a noisy suite — it made a specific class of defect invisible by construction. That is an argument about the ruling's blast radius, not a rediscovery of the ruling, and it is the part worth putting in front of the operator.

NOT ready to merge, and it cannot be evaluated at all until main compiles again.

Note on the red control. The honest limitation, stated plainly: the discriminating RED for change 1 is the OOM — revert it and the test does not fail cleanly, it takes the runner down. That is exactly why the dispatch declined to enroll cross_process_hit_skips_semantic_recompute. Change 2 is the part with a clean loud control, and repeat_resolve_through_disk_seam_does_not_reenter_provider asserts on the counted skip rather than on memory, which is the closest well-behaved proxy available. I did not find a way to make the store-suppression half fail loudly-but-safely without bounding resolve depth, which would be a larger change than this defect warrants.

Side finding, not fixed here

A 34-module closure resolve costs ~1.05 GiB of RSS in a single level (11 MiB → 1064 MiB across one nested resolve_entry_graph_shared). That is the known per-module env-duplication cost (DESIGN open thread: v1 whole-corpus run stability / build_type_env closure materialization), not something this PR touches. It is what made this recursion fatal within two levels instead of merely slow. Same window and derivation as the trace above.

What downstream needs

The warm-hit-skips-semantic-recompute clause in gunbc.materialization_kernel_closing_frontier_audit (currently PartiallyDelivered, blocked on this) can be re-attempted once this merges: a second resolve through the disk seam no longer recurses, so a witness for it no longer OOMs the runner.

Next if continued

1. Route the pre-existing provider_disposition_four_arm_matrix failure and the "CI never executes this module" coverage gap to owners — both are recorded above, neither is this PR's scope. If my new test is the flaky one, either serialize it behind the existing CACHE_ENV_MUTEX (the module already uses that to serialize env-dependent tests) or assert on a delta captured inside that lock rather than on absolute global counts.
2. Run the broader v1-compiler-tests suite — the suppression change alters when artifacts are written during bootstrap, and I have not swept for a test asserting on that write.
3. Consider whether the counted skip deserves a modeled disposition row rather than a bare atomic counter.

Doc obligation this PR creates (not done here)

The finding's single home is the DESIGN Open Threads entry "disk-tier repeat-resolve memory growth", whose carrier is gunbc.design_document (dag/gunbc/design_document.dag) — DESIGN.md is a projection of it, so it must never be hand-edited.

I deliberately did not update that row. It is still accurate for main: the defect exists there until this merges, and it correctly records the state at the time it was written. It becomes stale the moment this lands, because it names as "next trigger: root-cause" two candidates (install_cross_process_materialization_hit, and the decode path failing to share across MultiEntryIndex instances) that are now both refuted by execution.

Whoever merges this should update that row in the same motion, or the next person will re-derive two dead ends. gunbc.materialization_kernel_closing_frontier_audit cites the same entry and will need its PartiallyDelivered blocking-evidence clause revisited.

Tooling note (cost me two attempts; will cost the next person the same two)

To baseline an old commit under ctrl-build:

• git worktree add + ctrl-build from that path fails — ctrl-build does not resolve the worktree's .git file; it emits a branch listing and runs nothing, exit 0 with no usable output.
• git clone --local <worktree> also fails — ctrl-build re-fetches the checkout on the remote build host (git fetch --depth=1 origin <sha>), and a local-path origin does not exist there: fatal: '<path>' does not appear to be a git repository, exit 128.
• What works: clone, then git remote set-url origin https://github.com/gunb-ai/gunbc, then run ctrl-build from the clone. Add --no-local at clone time if /tmp is a different filesystem (hardlinking fails with Invalid cross-device link).

[gunbai-bot]

Flipped to ready. The stated blocker — origin/main does not compile — cleared when #7733 landed (19cc776d4e, restoring the #7708 emitter fixes that #7709's stale seed regen dropped). This branch is now green on all jobs: build, ci, regen, heal. Nothing about the PR's content changed; its blocker was external and is gone.

Note for whoever reads the body next: the paragraph explaining the E0061 breakage is now historical, and the materialization_kernel_closing_frontier_audit row it declines to update becomes stale the moment this merges — as the body itself says.

— sent from still-bat-561