Skip to content

Floor OOM fix: sample typed-cache cap once per run, not per insert - #7002

Merged
briansrls merged 3 commits into
mainfrom
session/royal-dove-562
Jul 21, 2026
Merged

briansrls merged 3 commits into
mainfrom
session/royal-dove-562

Conversation

@briansrls

@briansrls briansrls commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fleet CI floor OOM-kill incident (2026-07-21, redded main + every PR floor from ~12:17Z on srv3-06/07): enforce_typed_cache_entry_cap (landed #6972) re-derived its typed-cache entry cap from read_host_budget_bytes() on every cache insert. On CI runners no private cgroup memory.max/memory.high is discoverable, so the derivation fell through to live /proc/meminfo MemAvailable — a host-wide signal shared across co-resident sessions. Re-reading it per insert let the cap chase host noise within a single run, and each eviction's recompute-on-miss added memory pressure exactly when pressure was already high — a thrashing feedback loop ending in claim_executor SIGKILL (exit 137).

Fix

Per signed fix direction from the incident owner:

  • typed_module_cache_cap_derivation() factors the existing env-override/budget-read logic out of typed_module_cache_max_entries (kept, unchanged signature, for the existing direct-derivation tests) and adds a degraded flag (true when the source isn't a private cgroup limit).
  • MultiEntryIndex gains typed_module_cache_cap: OnceCell<usize>. The new typed_module_cache_cap(index) accessor samples the derivation exactly once per index lifetime — a run-start fact, never re-read per insert.
  • On that first sample, if the source is degraded, logs a typed, counted [floor-drain] degraded_budget_source line — an honesty arm (§5), not a widened failure: the cap still derives from whatever source was found, it's simply named so the degraded case is observable.
  • enforce_typed_cache_entry_cap and emit_floor_drain_receipt now go through this sampled-once accessor instead of re-deriving per call.
  • No floor-pin added. No env-var escape hatch.

Diff is contained to the #6972 mechanism + its two call sites in cli_run.rs, per manager sign-off — no PR-beta/SpacePacked dissolve-on work included.

Evidence

  • srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run 29829512709 both show the eviction-storm-then-Killed(137) signature (cap collapsing under live-signal noise, thousands of evictions).
  • Wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence (build job succeeded; ci job's floor-run step never emitted a single [floor-drain] line before being starved by host contention) — noting the cancellation here so it isn't mistaken for a hidden failure.

Test plan

  • typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift — by-execution witness that the cap holds fixed across a live signal move within one index's lifetime.
  • uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not — RED control reproducing the pre-fix oscillation shape (cap values drawn from the incident's own log: 300/227/294/204/325) via the still-live uncached derivation; catches a regression that reverts runtime call sites back to per-insert re-derivation.
  • All 6 pre-existing floor_drain_retention_test.rs tests pass unmodified.
  • cargo fmt --all --check clean; cargo check -p v1-compiler clean.

🤖 Generated with Claude Code

Brian Searls and others added 2 commits July 21, 2026 15:25
2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on
srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap
from read_host_budget_bytes() on every cache insert. On CI runners no
private cgroup memory.max/memory.high is discoverable, so the derivation
fell through to live /proc/meminfo MemAvailable — a host-wide signal shared
across co-resident sessions. Re-reading it per insert let the cap chase host
noise within a single run (e.g. 701->682->...->204->325), and each eviction's
recompute-on-miss added pressure exactly when pressure was already high — a
thrashing feedback loop ending in claim_executor SIGKILL (exit 137).

Fix, per signed direction: typed_module_cache_cap_derivation() factors the
existing env-override/budget-read logic out of typed_module_cache_max_entries
(kept as the pure per-call form for existing tests) and adds a `degraded`
flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field;
the new typed_module_cache_cap(index) accessor samples the derivation exactly
once per index lifetime and, on that first call, logs a typed, counted
`[floor-drain] degraded_budget_source` line when the source isn't a private
cgroup limit (an honesty arm, not a widened failure -- the cap still derives
from whatever source was found). enforce_typed_cache_entry_cap and
emit_floor_drain_receipt now go through this accessor instead of re-deriving
per call. No floor-pin added; no env-var escape hatch.

Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift
proves the cap holds fixed across a live signal move within one index's
lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not
is the RED control, reproducing the pre-fix oscillation shape (cap values
drawn from the incident's own log: 300/227/294/204/325) via the still-live
uncached derivation, so a regression that reverts the runtime call sites back
to the uncached form is caught.

Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run
29829512709 both show the eviction-storm-then-Killed(137) signature; the
wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence
(build job succeeded, ci job's floor-run step never emitted a single
[floor-drain] line before being starved by host contention) -- noted here so
the cancellation isn't mistaken for a hidden failure.

Contained to the #6972 mechanism + its two call sites, per manager sign-off;
no PR-beta/SpacePacked dissolve-on work included.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI floor OOM-kill class — claim_executor exit 137 after typed_cache_eviction storm (cap 294->227, 2300+ evictions), redding main + every PR floor since 12:17Z on srv3-06 AND srv3-07. Diagnose the differential (12:14Z merge corpus g Floor OOM fix: sample typed-cache cap once per run, not per insert Jul 21, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review July 21, 2026 15:34
Non-blocking review observation from quick-carp-521 on #7002: the
degraded-source check string-scans read_host_budget_bytes' display label.
Fine for now (it returns (Option<u64>, String)), but if that fn ever grows
a typed source enum, this check should ground on it instead of re-parsing
the label — a one-line note so it doesn't cement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@briansrls
briansrls merged commit 7ed7e09 into main Jul 21, 2026
2 checks passed
@briansrls
briansrls deleted the session/royal-dove-562 branch July 21, 2026 16:28
gunbai-bot Bot pushed a commit that referenced this pull request Jul 21, 2026
Refresh branch to current main so compile-clean scope narrows to the
3-file .dag strip; drops stale src/v1 .rs two-dot noise.
briansrls added a commit that referenced this pull request Jul 21, 2026
gunbai-bot Bot pushed a commit that referenced this pull request Jul 21, 2026
Post-merge regen_stage0 --verify reported regen_divergence_count=1
(std_measure.rs; main's measure.dag delta, not #7002 cli_run). Re-emitted;
verify now regen_divergence_count=0.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jul 21, 2026
…erialization_carriers, program_assembly, source_authority, 02_parse, 03_ingest) refuse with UNRESOLVED_CompilerError in curated probes — census + receipts in #6872 TSVs (calm-boar-697). Read the actual probe error output per module (#6986)

* Gate-A flip prep + Measure nested-generic emitter fix (rebased on main)

Rebased session/neat-swift-795-flip-prep onto origin/main: drop superseded
module-identity/integration history already landed via #6866; retain flip
infrastructure (frontier 6→12 SelfEmitted, wet enrollment, behavioral
transports/shims), Measure emitter generic-env threading (EmitGraphInfo
fn_generic_param_names/fn_type_env, fold lambda param_nodes, container
child recursion gated to generic-fn context), stage0 regen, and nested
List<List<Measure>> emit witness.

PR #6881 remains draft until sign-off receipts complete.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix PR #6881 CI: exclude wet behavioral receipts and repair census gates.

Hermetic discovery was running the three new self-host behavioral witnesses
(shell.Mktemp.Dir has no mock); add them to witness_exclusion_substrings like
the other wet receipts. Discharge Band-A OtherGate rows when the frontier
module is already SelfEmitted. Rename nested-fold witness so its stem covers
the new v1 emit test module for the migration-debt ratchet.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* Revert unrelated effect-namespace-grants.md collateral from Gate-A PR.

Restore origin/main section 6 (FLAG A interim): workspace Read grant,
ownership-implies-read note, and PR-1/PR-2 LANDED paragraphs. The hunk
was accidental doc drift with no tie to the flip or emitter work.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* Add Measure sign-off POST-fix 6-module probe receipt TSV.

Documents hash-verified gunbc rebuild probe results for the flip-wave
baseline: emit 0-diag on all six, cargo at named next layers (std_dup on
5/6; materialization_carriers E0433 deferred per operator routing).

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* Fix PR #6881 CI: regen verify gate + stage0 infer sync.

Remove duplicate empty_type_env fallout from stage0 infer emit drift,
and normalize rustfmt-only diffs in regen_stage0 verify so workspace
cargo fmt and fresh self-compile compare cleanly.

Co-authored-by: Cursor <cursoragent@cursor.com>

* ci: retrigger after regen verify + cargo fmt fix (44b23d6).

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* frontier: sync stage0 crate layout after honest SelfEmitted baseline revert (12→6).

Regenerated stage0_crate_layout_generated artifacts to drop the six
prematurely-flipped modules from hand-maintained pub mod / filename rosters
now that frontier.dag marks them SeedRetained again.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(frontier): restore sweep-order monotonicity after honest revert.

Re-sort compiler_frontier_sweep_order after SeedRetained revert changed
tractability ranks (materialization_carriers/program_assembly →
NameResolutionGap band; 00_compile → EmitSurfaceGap). Fixes
compiler_frontier_census_composite_test CI failure on #6881.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* fix(frontier): align reverted behavioral transport docs with honest refresh.

Six reverted-module transport docs now cite SeedRetained / honest_frontier_refresh_probe_note instead of stale "flip landed" prose. Remove three flip-wave seed stubs (parse/ingest/materialization_carriers) that were never wired into the frontier-derived crate layout.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Gate-1 emit regression (post-#6848): the Rust emit target renders namesp

* WIP: Gate-1 emit regression (post-#6848): the Rust emit target renders namesp

* emit_imports: field-struct import synth asks "provides", not "physically defines"

An anonymous record literal takes its type from the FIELD it initializes, so the
constructed type name appears nowhere in the source and can never sit in an
authored import list. emit_imports derives use-lines only from import lists, so
the emitted Rust refused: E0422 cannot find struct ... in this scope.

module_data_field_struct_import_names gated candidates on
has_physical_type_def_in_module_filename — field type PHYSICALLY DEFINED in the
module being imported from. That is narrower than the question being asked.
compile_stage_memo imports CacheInterfaceCatalogFacts from extdeps.cache.types
but its io/placement field types are defined in the SIBLING modules
extdeps.cache.catalog_io / catalog_placement and merely re-exported, so they were
rejected and no use-line was synthesized.

Now uses name_in_transitive_export_surface, the existing authority for "does this
module provide this name" (get_exported_names counts specific-import names as
exports, which is exactly re-export) — reused, not re-minted. Routing needed no
change: graph_type_import_module_filename already groups emitted use-lines by the
type's DEFINING module. export_sets is threaded as a parameter rather than rebuilt
inside the filter, which would have been a per-field quadratic cost-shape defect.

Verified by execution, both halves:
  regen_divergence_count=0 (two-stage bootstrap; stage 2 with the new emitter LIVE
    over the whole import-BEARING seed, which is the run that counts)
  materialization_carriers first error advanced
    CacheInterfaceCatalogIoSemantics -> CacheEvidence

Known residue, documented in-code not hidden: a generic field type
(evidence: List<CacheEvidence>) still misses, because build_field_type_map stores
only the type node's HEAD name, so CacheEvidence is absent from field_type_map
entirely and no string-level test can recover it. Distinct axis, separate increment.

Honest frontier refresh (receipt must match measurement, independent of any flip):
01_tokenize / 04_infer / program_partition were recorded as import_closure but all
three now MEASURE as the dotted namespace-qualified render class; rows corrected.
Flip wave produced ZERO flips — 4 of 5 probed modules refuse on the dotted class,
which is emitted into std.collection itself and so sits in every deep closure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* emit_imports increment-2: recurse field-type node tree for import surface names.

build_field_type_map now walks resolved generic/container argument nodes so
List<CacheEvidence> contributes CacheEvidence to field_import_surface_names,
not just the List head in field_type_map. emit_imports field-struct synth
consumes the expanded surface list (rides on #6981 sibling-axis machinery).

Oracle: materialization_carriers first cargo error advances past CacheEvidence
E0422 (verified via cssl probe + emitted use of std_cache_interface::CacheEvidence).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(emit_imports): refresh provider note after increment-2 generic axis.

Replace stale KNOWN RESIDUE paragraph claiming CacheEvidence still misses —
increment-2 now walks the resolved field-type node tree into
field_import_surface_names. Update frontier_probe_types receipt to record
materialization_carriers advancing past CacheEvidence E0422.

Co-authored-by: Cursor <cursoragent@cursor.com>

* ci: cargo fmt for increment-2 stage0 + test module ordering.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi

* Regen stage0 infer_emit_info seed after main merge (#6983).

Post-merge regen_stage0 --verify reported regen_divergence_count=1
(v1_compiler_infer_emit_info.rs stale vs live emitter). Re-emitted seed;
verify now regen_divergence_count=0.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Regen std_measure seed after main merge (#6991 measure.dag).

Post-merge regen_stage0 --verify reported regen_divergence_count=1
(std_measure.rs; main's measure.dag delta, not #7002 cli_run). Re-emitted;
verify now regen_divergence_count=0.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Jul 22, 2026
…srv1 to srv3 (multi-unit) (#7009)

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* Floor OOM fix: sample typed-cache cap once per run, not per insert

2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on
srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap
from read_host_budget_bytes() on every cache insert. On CI runners no
private cgroup memory.max/memory.high is discoverable, so the derivation
fell through to live /proc/meminfo MemAvailable — a host-wide signal shared
across co-resident sessions. Re-reading it per insert let the cap chase host
noise within a single run (e.g. 701->682->...->204->325), and each eviction's
recompute-on-miss added pressure exactly when pressure was already high — a
thrashing feedback loop ending in claim_executor SIGKILL (exit 137).

Fix, per signed direction: typed_module_cache_cap_derivation() factors the
existing env-override/budget-read logic out of typed_module_cache_max_entries
(kept as the pure per-call form for existing tests) and adds a `degraded`
flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field;
the new typed_module_cache_cap(index) accessor samples the derivation exactly
once per index lifetime and, on that first call, logs a typed, counted
`[floor-drain] degraded_budget_source` line when the source isn't a private
cgroup limit (an honesty arm, not a widened failure -- the cap still derives
from whatever source was found). enforce_typed_cache_entry_cap and
emit_floor_drain_receipt now go through this accessor instead of re-deriving
per call. No floor-pin added; no env-var escape hatch.

Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift
proves the cap holds fixed across a live signal move within one index's
lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not
is the RED control, reproducing the pre-fix oscillation shape (cap values
drawn from the incident's own log: 300/227/294/204/325) via the still-live
uncached derivation, so a regression that reverts the runtime call sites back
to the uncached form is caught.

Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run
29829512709 both show the eviction-storm-then-Killed(137) signature; the
wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence
(build job succeeded, ci job's floor-run step never emitted a single
[floor-drain] line before being starved by host contention) -- noted here so
the cancellation isn't mistaken for a hidden failure.

Contained to the #6972 mechanism + its two call sites, per manager sign-off;
no PR-beta/SpacePacked dissolve-on work included.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Note the degraded-detection string-scan as a future grounding point

Non-blocking review observation from quick-carp-521 on #7002: the
degraded-source check string-scans read_host_budget_bytes' display label.
Fine for now (it returns (Option<u64>, String)), but if that fn ever grows
a typed source enum, this check should ground on it instead of re-parsing
the label — a one-line note so it doesn't cement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* host_converge_slice1: fix .dag lexer trap in awk literal, document first-execution safety

awk '{print $1}' triggered a string-interpolation lexer fast-path on the
bare $ inside an unspaced brace, producing a parser panic never caught
because the file had never been compiled. Space-padded the braces.

Also records why the apply arm already satisfies first-execution safety
by construction: enumeration + cross-check + per-unit live read + usage
guard must all succeed before set-property is reachable, so the first
live invocation against srv3 is necessarily read-only up to several
fail-closed gates.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* PR #7009 review fixes: wire srv3 CLI routing, ByteSize typing, empty-enumeration handling

Addresses review 40879 (cursor/composer-2.5) on PR #7009:
- Finding 1: gunbc converge --host srv3 now routes into
  host_converge_slice1_converge_srv3()/converge_host() via a new
  converge_cli_slice1_srv3_receipt() aggregator, instead of silently
  falling through to the generic converge_apply_for_host frontier.
  The srv3 multi-unit enumeration/cross-check/usage-guard machinery
  was previously unreachable from any CLI entrypoint.
- Finding 2: HostConvergeSlice1UsageGuardOutcome and
  HostConvergeSlice1UnitUsageGuardRefused now carry ByteSize instead
  of bare Int, and host_converge_slice1_usage_guard reuses
  runner_unit_usage_exceeds_bytes instead of re-deriving the
  current-vs-target comparison.
- Finding 3: host_converge_slice1_enumerate_units now returns
  Present{value: []} on a successful read with zero active units,
  distinct from none (transport/read failure), so a real zero-unit
  state reaches the typed HostEnumerationMismatch refusal instead of
  being conflated with a generic HostRefused.

Adds witness coverage for the srv3 routing predicate and the
ByteSize-typed usage-guard-refused legacy-result mapping.

* WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI

* Fix orphan-doc CI failure on PR #7009: bind docs/probes/emitter_residual_site_map_2026-07-21.md

The doc landed on main via #7023 with no doc-graph link, which is a
pre-existing main-red (main failed at 50bb7e9 and d36515b too) that
bled into this PR's CI once it merged main. Mirrors the existing
bind: pattern in this file for curated_cargo_frontier_probe_report.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* PR #7009 review 41089: document fleet-wide-vs-per-host declared_runner_count gap with named dissolution trigger

host_converge_slice1_cross_check_enumeration's host param labels the
mismatch record only; declared_runner_count() is fleet-wide by design
(one global budget, no per-host divergence exists yet). Modeling a
per-host budget now would be speculative per DESIGN §6. Recorded as a
named, triggered gap on the scope-disposition Terminal note instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant