Repository navigation
Floor OOM fix: sample typed-cache cap once per run, not per insert - #7002
Merged
Merged
Conversation
2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap from read_host_budget_bytes() on every cache insert. On CI runners no private cgroup memory.max/memory.high is discoverable, so the derivation fell through to live /proc/meminfo MemAvailable — a host-wide signal shared across co-resident sessions. Re-reading it per insert let the cap chase host noise within a single run (e.g. 701->682->...->204->325), and each eviction's recompute-on-miss added pressure exactly when pressure was already high — a thrashing feedback loop ending in claim_executor SIGKILL (exit 137). Fix, per signed direction: typed_module_cache_cap_derivation() factors the existing env-override/budget-read logic out of typed_module_cache_max_entries (kept as the pure per-call form for existing tests) and adds a `degraded` flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field; the new typed_module_cache_cap(index) accessor samples the derivation exactly once per index lifetime and, on that first call, logs a typed, counted `[floor-drain] degraded_budget_source` line when the source isn't a private cgroup limit (an honesty arm, not a widened failure -- the cap still derives from whatever source was found). enforce_typed_cache_entry_cap and emit_floor_drain_receipt now go through this accessor instead of re-deriving per call. No floor-pin added; no env-var escape hatch. Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift proves the cap holds fixed across a live signal move within one index's lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not is the RED control, reproducing the pre-fix oscillation shape (cap values drawn from the incident's own log: 300/227/294/204/325) via the still-live uncached derivation, so a regression that reverts the runtime call sites back to the uncached form is caught. Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run 29829512709 both show the eviction-storm-then-Killed(137) signature; the wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence (build job succeeded, ci job's floor-run step never emitted a single [floor-drain] line before being starved by host contention) -- noted here so the cancellation isn't mistaken for a hidden failure. Contained to the #6972 mechanism + its two call sites, per manager sign-off; no PR-beta/SpacePacked dissolve-on work included. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-blocking review observation from quick-carp-521 on #7002: the degraded-source check string-scans read_host_budget_bytes' display label. Fine for now (it returns (Option<u64>, String)), but if that fn ever grows a typed source enum, this check should ground on it instead of re-parsing the label — a one-line note so it doesn't cement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
briansrls
pushed a commit
that referenced
this pull request
Jul 21, 2026
briansrls
pushed a commit
that referenced
this pull request
Jul 21, 2026
gunbai-bot Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
Refresh branch to current main so compile-clean scope narrows to the 3-file .dag strip; drops stale src/v1 .rs two-dot noise.
briansrls
added a commit
that referenced
this pull request
Jul 21, 2026
gunbai-bot Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
Post-merge regen_stage0 --verify reported regen_divergence_count=1 (std_measure.rs; main's measure.dag delta, not #7002 cli_run). Re-emitted; verify now regen_divergence_count=0. Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls
added a commit
that referenced
this pull request
Jul 21, 2026
2 of 3 tasks
This was referenced Jul 21, 2026
briansrls
added a commit
that referenced
this pull request
Jul 21, 2026
…erialization_carriers, program_assembly, source_authority, 02_parse, 03_ingest) refuse with UNRESOLVED_CompilerError in curated probes — census + receipts in #6872 TSVs (calm-boar-697). Read the actual probe error output per module (#6986) * Gate-A flip prep + Measure nested-generic emitter fix (rebased on main) Rebased session/neat-swift-795-flip-prep onto origin/main: drop superseded module-identity/integration history already landed via #6866; retain flip infrastructure (frontier 6→12 SelfEmitted, wet enrollment, behavioral transports/shims), Measure emitter generic-env threading (EmitGraphInfo fn_generic_param_names/fn_type_env, fold lambda param_nodes, container child recursion gated to generic-fn context), stage0 regen, and nested List<List<Measure>> emit witness. PR #6881 remains draft until sign-off receipts complete. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix PR #6881 CI: exclude wet behavioral receipts and repair census gates. Hermetic discovery was running the three new self-host behavioral witnesses (shell.Mktemp.Dir has no mock); add them to witness_exclusion_substrings like the other wet receipts. Discharge Band-A OtherGate rows when the frontier module is already SelfEmitted. Rename nested-fold witness so its stem covers the new v1 emit test module for the migration-debt ratchet. Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * Revert unrelated effect-namespace-grants.md collateral from Gate-A PR. Restore origin/main section 6 (FLAG A interim): workspace Read grant, ownership-implies-read note, and PR-1/PR-2 LANDED paragraphs. The hunk was accidental doc drift with no tie to the flip or emitter work. Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * Add Measure sign-off POST-fix 6-module probe receipt TSV. Documents hash-verified gunbc rebuild probe results for the flip-wave baseline: emit 0-diag on all six, cargo at named next layers (std_dup on 5/6; materialization_carriers E0433 deferred per operator routing). Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * Fix PR #6881 CI: regen verify gate + stage0 infer sync. Remove duplicate empty_type_env fallout from stage0 infer emit drift, and normalize rustfmt-only diffs in regen_stage0 verify so workspace cargo fmt and fresh self-compile compare cleanly. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: retrigger after regen verify + cargo fmt fix (44b23d6). Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * frontier: sync stage0 crate layout after honest SelfEmitted baseline revert (12→6). Regenerated stage0_crate_layout_generated artifacts to drop the six prematurely-flipped modules from hand-maintained pub mod / filename rosters now that frontier.dag marks them SeedRetained again. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(frontier): restore sweep-order monotonicity after honest revert. Re-sort compiler_frontier_sweep_order after SeedRetained revert changed tractability ranks (materialization_carriers/program_assembly → NameResolutionGap band; 00_compile → EmitSurfaceGap). Fixes compiler_frontier_census_composite_test CI failure on #6881. Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * fix(frontier): align reverted behavioral transport docs with honest refresh. Six reverted-module transport docs now cite SeedRetained / honest_frontier_refresh_probe_note instead of stale "flip landed" prose. Remove three flip-wave seed stubs (parse/ingest/materialization_carriers) that were never wired into the frontier-derived crate layout. Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: Gate-1 emit regression (post-#6848): the Rust emit target renders namesp * WIP: Gate-1 emit regression (post-#6848): the Rust emit target renders namesp * emit_imports: field-struct import synth asks "provides", not "physically defines" An anonymous record literal takes its type from the FIELD it initializes, so the constructed type name appears nowhere in the source and can never sit in an authored import list. emit_imports derives use-lines only from import lists, so the emitted Rust refused: E0422 cannot find struct ... in this scope. module_data_field_struct_import_names gated candidates on has_physical_type_def_in_module_filename — field type PHYSICALLY DEFINED in the module being imported from. That is narrower than the question being asked. compile_stage_memo imports CacheInterfaceCatalogFacts from extdeps.cache.types but its io/placement field types are defined in the SIBLING modules extdeps.cache.catalog_io / catalog_placement and merely re-exported, so they were rejected and no use-line was synthesized. Now uses name_in_transitive_export_surface, the existing authority for "does this module provide this name" (get_exported_names counts specific-import names as exports, which is exactly re-export) — reused, not re-minted. Routing needed no change: graph_type_import_module_filename already groups emitted use-lines by the type's DEFINING module. export_sets is threaded as a parameter rather than rebuilt inside the filter, which would have been a per-field quadratic cost-shape defect. Verified by execution, both halves: regen_divergence_count=0 (two-stage bootstrap; stage 2 with the new emitter LIVE over the whole import-BEARING seed, which is the run that counts) materialization_carriers first error advanced CacheInterfaceCatalogIoSemantics -> CacheEvidence Known residue, documented in-code not hidden: a generic field type (evidence: List<CacheEvidence>) still misses, because build_field_type_map stores only the type node's HEAD name, so CacheEvidence is absent from field_type_map entirely and no string-level test can recover it. Distinct axis, separate increment. Honest frontier refresh (receipt must match measurement, independent of any flip): 01_tokenize / 04_infer / program_partition were recorded as import_closure but all three now MEASURE as the dotted namespace-qualified render class; rows corrected. Flip wave produced ZERO flips — 4 of 5 probed modules refuse on the dotted class, which is emitted into std.collection itself and so sits in every deep closure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * emit_imports increment-2: recurse field-type node tree for import surface names. build_field_type_map now walks resolved generic/container argument nodes so List<CacheEvidence> contributes CacheEvidence to field_import_surface_names, not just the List head in field_type_map. emit_imports field-struct synth consumes the expanded surface list (rides on #6981 sibling-axis machinery). Oracle: materialization_carriers first cargo error advances past CacheEvidence E0422 (verified via cssl probe + emitted use of std_cache_interface::CacheEvidence). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(emit_imports): refresh provider note after increment-2 generic axis. Replace stale KNOWN RESIDUE paragraph claiming CacheEvidence still misses — increment-2 now walks the resolved field-type node tree into field_import_surface_names. Update frontier_probe_types receipt to record materialization_carriers advancing past CacheEvidence E0422. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: cargo fmt for increment-2 stage0 + test module ordering. Co-authored-by: Cursor <cursoragent@cursor.com> * WIP: unresolved_compiler_error diagnosis + fix: 6 modules (00_compile, materi * Regen stage0 infer_emit_info seed after main merge (#6983). Post-merge regen_stage0 --verify reported regen_divergence_count=1 (v1_compiler_infer_emit_info.rs stale vs live emitter). Re-emitted seed; verify now regen_divergence_count=0. Co-authored-by: Cursor <cursoragent@cursor.com> * Regen std_measure seed after main merge (#6991 measure.dag). Post-merge regen_stage0 --verify reported regen_divergence_count=1 (std_measure.rs; main's measure.dag delta, not #7002 cli_run). Re-emitted; verify now regen_divergence_count=0. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3 of 4 tasks
briansrls
added a commit
that referenced
this pull request
Jul 22, 2026
…srv1 to srv3 (multi-unit) (#7009) * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * Floor OOM fix: sample typed-cache cap once per run, not per insert 2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap from read_host_budget_bytes() on every cache insert. On CI runners no private cgroup memory.max/memory.high is discoverable, so the derivation fell through to live /proc/meminfo MemAvailable — a host-wide signal shared across co-resident sessions. Re-reading it per insert let the cap chase host noise within a single run (e.g. 701->682->...->204->325), and each eviction's recompute-on-miss added pressure exactly when pressure was already high — a thrashing feedback loop ending in claim_executor SIGKILL (exit 137). Fix, per signed direction: typed_module_cache_cap_derivation() factors the existing env-override/budget-read logic out of typed_module_cache_max_entries (kept as the pure per-call form for existing tests) and adds a `degraded` flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field; the new typed_module_cache_cap(index) accessor samples the derivation exactly once per index lifetime and, on that first call, logs a typed, counted `[floor-drain] degraded_budget_source` line when the source isn't a private cgroup limit (an honesty arm, not a widened failure -- the cap still derives from whatever source was found). enforce_typed_cache_entry_cap and emit_floor_drain_receipt now go through this accessor instead of re-deriving per call. No floor-pin added; no env-var escape hatch. Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift proves the cap holds fixed across a live signal move within one index's lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not is the RED control, reproducing the pre-fix oscillation shape (cap values drawn from the incident's own log: 300/227/294/204/325) via the still-live uncached derivation, so a regression that reverts the runtime call sites back to the uncached form is caught. Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run 29829512709 both show the eviction-storm-then-Killed(137) signature; the wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence (build job succeeded, ci job's floor-run step never emitted a single [floor-drain] line before being starved by host contention) -- noted here so the cancellation isn't mistaken for a hidden failure. Contained to the #6972 mechanism + its two call sites, per manager sign-off; no PR-beta/SpacePacked dissolve-on work included. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Note the degraded-detection string-scan as a future grounding point Non-blocking review observation from quick-carp-521 on #7002: the degraded-source check string-scans read_host_budget_bytes' display label. Fine for now (it returns (Option<u64>, String)), but if that fn ever grows a typed source enum, this check should ground on it instead of re-parsing the label — a one-line note so it doesn't cement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * host_converge_slice1: fix .dag lexer trap in awk literal, document first-execution safety awk '{print $1}' triggered a string-interpolation lexer fast-path on the bare $ inside an unspaced brace, producing a parser panic never caught because the file had never been compiled. Space-padded the braces. Also records why the apply arm already satisfies first-execution safety by construction: enumeration + cross-check + per-unit live read + usage guard must all succeed before set-property is reachable, so the first live invocation against srv3 is necessarily read-only up to several fail-closed gates. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * PR #7009 review fixes: wire srv3 CLI routing, ByteSize typing, empty-enumeration handling Addresses review 40879 (cursor/composer-2.5) on PR #7009: - Finding 1: gunbc converge --host srv3 now routes into host_converge_slice1_converge_srv3()/converge_host() via a new converge_cli_slice1_srv3_receipt() aggregator, instead of silently falling through to the generic converge_apply_for_host frontier. The srv3 multi-unit enumeration/cross-check/usage-guard machinery was previously unreachable from any CLI entrypoint. - Finding 2: HostConvergeSlice1UsageGuardOutcome and HostConvergeSlice1UnitUsageGuardRefused now carry ByteSize instead of bare Int, and host_converge_slice1_usage_guard reuses runner_unit_usage_exceeds_bytes instead of re-deriving the current-vs-target comparison. - Finding 3: host_converge_slice1_enumerate_units now returns Present{value: []} on a successful read with zero active units, distinct from none (transport/read failure), so a real zero-unit state reaches the typed HostEnumerationMismatch refusal instead of being conflated with a generic HostRefused. Adds witness coverage for the srv3 routing predicate and the ByteSize-typed usage-guard-refused legacy-result mapping. * WIP: WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI * Fix orphan-doc CI failure on PR #7009: bind docs/probes/emitter_residual_site_map_2026-07-21.md The doc landed on main via #7023 with no doc-graph link, which is a pre-existing main-red (main failed at 50bb7e9 and d36515b too) that bled into this PR's CI once it merged main. Mirrors the existing bind: pattern in this file for curated_cargo_frontier_probe_report.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * PR #7009 review 41089: document fleet-wide-vs-per-host declared_runner_count gap with named dissolution trigger host_converge_slice1_cross_check_enumeration's host param labels the mismatch record only; declared_runner_count() is fleet-wide by design (one global budget, no per-host divergence exists yet). Modeling a per-host budget now would be speculative per DESIGN §6. Recorded as a named, triggered gap on the scope-disposition Terminal note instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fleet CI floor OOM-kill incident (2026-07-21, redded main + every PR floor from ~12:17Z on srv3-06/07):
enforce_typed_cache_entry_cap(landed #6972) re-derived its typed-cache entry cap fromread_host_budget_bytes()on every cache insert. On CI runners no private cgroupmemory.max/memory.highis discoverable, so the derivation fell through to live/proc/meminfo MemAvailable— a host-wide signal shared across co-resident sessions. Re-reading it per insert let the cap chase host noise within a single run, and each eviction's recompute-on-miss added memory pressure exactly when pressure was already high — a thrashing feedback loop ending inclaim_executorSIGKILL (exit 137).Fix
Per signed fix direction from the incident owner:
typed_module_cache_cap_derivation()factors the existing env-override/budget-read logic out oftyped_module_cache_max_entries(kept, unchanged signature, for the existing direct-derivation tests) and adds adegradedflag (true when the source isn't a private cgroup limit).MultiEntryIndexgainstyped_module_cache_cap: OnceCell<usize>. The newtyped_module_cache_cap(index)accessor samples the derivation exactly once per index lifetime — a run-start fact, never re-read per insert.[floor-drain] degraded_budget_sourceline — an honesty arm (§5), not a widened failure: the cap still derives from whatever source was found, it's simply named so the degraded case is observable.enforce_typed_cache_entry_capandemit_floor_drain_receiptnow go through this sampled-once accessor instead of re-deriving per call.Diff is contained to the #6972 mechanism + its two call sites in
cli_run.rs, per manager sign-off — no PR-beta/SpacePacked dissolve-on work included.Evidence
Killed(137) signature (cap collapsing under live-signal noise, thousands of evictions).cijob's floor-run step never emitted a single[floor-drain]line before being starved by host contention) — noting the cancellation here so it isn't mistaken for a hidden failure.Test plan
typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift— by-execution witness that the cap holds fixed across a live signal move within one index's lifetime.uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not— RED control reproducing the pre-fix oscillation shape (cap values drawn from the incident's own log: 300/227/294/204/325) via the still-live uncached derivation; catches a regression that reverts runtime call sites back to per-insert re-derivation.floor_drain_retention_test.rstests pass unmodified.cargo fmt --all --checkclean;cargo check -p v1-compilerclean.🤖 Generated with Claude Code