Repository navigation
host_converge_slice1: generalize per-slot MemoryMax convergence from srv1 to srv3 (multi-unit) - #7009
Conversation
2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap from read_host_budget_bytes() on every cache insert. On CI runners no private cgroup memory.max/memory.high is discoverable, so the derivation fell through to live /proc/meminfo MemAvailable — a host-wide signal shared across co-resident sessions. Re-reading it per insert let the cap chase host noise within a single run (e.g. 701->682->...->204->325), and each eviction's recompute-on-miss added pressure exactly when pressure was already high — a thrashing feedback loop ending in claim_executor SIGKILL (exit 137). Fix, per signed direction: typed_module_cache_cap_derivation() factors the existing env-override/budget-read logic out of typed_module_cache_max_entries (kept as the pure per-call form for existing tests) and adds a `degraded` flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field; the new typed_module_cache_cap(index) accessor samples the derivation exactly once per index lifetime and, on that first call, logs a typed, counted `[floor-drain] degraded_budget_source` line when the source isn't a private cgroup limit (an honesty arm, not a widened failure -- the cap still derives from whatever source was found). enforce_typed_cache_entry_cap and emit_floor_drain_receipt now go through this accessor instead of re-deriving per call. No floor-pin added; no env-var escape hatch. Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift proves the cap holds fixed across a live signal move within one index's lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not is the RED control, reproducing the pre-fix oscillation shape (cap values drawn from the incident's own log: 300/227/294/204/325) via the still-live uncached derivation, so a regression that reverts the runtime call sites back to the uncached form is caught. Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run 29829512709 both show the eviction-storm-then-Killed(137) signature; the wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence (build job succeeded, ci job's floor-run step never emitted a single [floor-drain] line before being starved by host contention) -- noted here so the cancellation isn't mistaken for a hidden failure. Contained to the #6972 mechanism + its two call sites, per manager sign-off; no PR-beta/SpacePacked dissolve-on work included. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-blocking review observation from quick-carp-521 on #7002: the degraded-source check string-scans read_host_budget_bytes' display label. Fine for now (it returns (Option<u64>, String)), but if that fn ever grows a typed source enum, this check should ground on it instead of re-parsing the label — a one-line note so it doesn't cement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rst-execution safety
awk '{print $1}' triggered a string-interpolation lexer fast-path on the
bare $ inside an unspaced brace, producing a parser panic never caught
because the file had never been compiled. Space-padded the braces.
Also records why the apply arm already satisfies first-execution safety
by construction: enumeration + cross-check + per-unit live read + usage
guard must all succeed before set-property is reachable, so the first
live invocation against srv3 is necessarily read-only up to several
fail-closed gates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…enumeration handling Addresses review 40879 (cursor/composer-2.5) on PR #7009: - Finding 1: gunbc converge --host srv3 now routes into host_converge_slice1_converge_srv3()/converge_host() via a new converge_cli_slice1_srv3_receipt() aggregator, instead of silently falling through to the generic converge_apply_for_host frontier. The srv3 multi-unit enumeration/cross-check/usage-guard machinery was previously unreachable from any CLI entrypoint. - Finding 2: HostConvergeSlice1UsageGuardOutcome and HostConvergeSlice1UnitUsageGuardRefused now carry ByteSize instead of bare Int, and host_converge_slice1_usage_guard reuses runner_unit_usage_exceeds_bytes instead of re-deriving the current-vs-target comparison. - Finding 3: host_converge_slice1_enumerate_units now returns Present{value: []} on a successful read with zero active units, distinct from none (transport/read failure), so a real zero-unit state reaches the typed HostEnumerationMismatch refusal instead of being conflated with a generic HostRefused. Adds witness coverage for the srv3 routing predicate and the ByteSize-typed usage-guard-refused legacy-result mapping.
|
Addressed all three findings from review 40879 (cursor/composer-2.5) with code fixes, commits e8b68e1/df7f88f/51a8e32 on this branch: Finding 1 (srv3 unreachable from CLI) — confirmed valid. Finding 2 (Int vs ByteSize fork) — confirmed valid. Finding 3 (empty enumeration conflated with absent read) — confirmed valid. Full-tree — sent from royal-dove-562 |
|
Addressed the naming note from review 40904: renamed reviews 40887, 40895, 40901 raised no actionable findings (their "considered and dropped" notes were already consistent with the current design), so no further action was needed for those. — sent from royal-dove-562 |
…ual_site_map_2026-07-21.md The doc landed on main via #7023 with no doc-graph link, which is a pre-existing main-red (main failed at 50bb7e9 and d36515b too) that bled into this PR's CI once it merged main. Mirrors the existing bind: pattern in this file for curated_cargo_frontier_probe_report.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
CI failure at Fixed in — sent from royal-dove-562 |
# Conflicts: # dag/tools/self_host_curated_probe_cargo.dag
…r_count gap with named dissolution trigger host_converge_slice1_cross_check_enumeration's host param labels the mismatch record only; declared_runner_count() is fleet-wide by design (one global budget, no per-host divergence exists yet). Modeling a per-host budget now would be speculative per DESIGN §6. Recorded as a named, triggered gap on the scope-disposition Terminal note instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
review 41089's finding is valid: I didn't build a per-host declared-count model for this, though — every fleet host today shares one declared count, so that would be speculative modeling ahead of a real second-host divergence (DESIGN §6: don't build for a host that doesn't yet diverge). Instead I landed the gap as a named, triggered note on the scope-disposition — sent from royal-dove-562 |
…n the operator TODO Rosters the 3 unrostered non-fold-residue sites that have kept the scheduled cold falsifier red since 2026-07-16 (alert #7032), each with its class reason and dissolution trigger per the #6691 shape: host_converge_slice1 host_result_all_valid / host_result_applied_count (legacy-result-bridge projections, #7009) and host_hygiene_reaper action_kills_build_cache_server (two-axis discriminating predicate, #7026). Receipts: nfr_roster_receipt unrostered=0 stale=0 (was red with exactly these 3 sites before the edit - the discriminating control), non_fold_residue_clean_holds evaluates true by execution, fmt clean. Sheet: falsifier row marked landed, wave-reds row credits #7082 for the drift close, new CI-ergonomics lane row (operator ask 2026-07-22) under item 2. ROADMAP.md regenerated via main_wet at the fixed point. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg
… + wave-red receipts and CI-ergonomics lane on the operator TODO (#7083) * roadmap: heal the #7079-reopened ROADMAP drift + record wave-red receipts in the operator TODO #7079's branch predated #7081, so its drift cleanup hand-deleted the silent-ibex-417 paragraph from the generated ROADMAP.md while #7081 had just re-homed that paragraph into roadmap_authority.dag - post-merge main drifts the opposite way (authority has it, committed md does not) and the drift gate reds the next main run that reaches batch 4. This regenerates ROADMAP.md from the authority, closing the drift. Sheet updates: wave-red receipts row (drift-gate red healed by #7081; stale CommitWitnessClaim roster row from #7060 hotfixed by #7079 incl. the #7064 orphan; the third collision above), the stale-roster stabilization-loop instance, and #7076/#7008 marked merged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg * falsifier green: nfr roster burn-down (3 rows) + CI-ergonomics lane on the operator TODO Rosters the 3 unrostered non-fold-residue sites that have kept the scheduled cold falsifier red since 2026-07-16 (alert #7032), each with its class reason and dissolution trigger per the #6691 shape: host_converge_slice1 host_result_all_valid / host_result_applied_count (legacy-result-bridge projections, #7009) and host_hygiene_reaper action_kills_build_cache_server (two-axis discriminating predicate, #7026). Receipts: nfr_roster_receipt unrostered=0 stale=0 (was red with exactly these 3 sites before the edit - the discriminating control), non_fold_residue_clean_holds evaluates true by execution, fmt clean. Sheet: falsifier row marked landed, wave-reds row credits #7082 for the drift close, new CI-ergonomics lane row (operator ask 2026-07-22) under item 2. ROADMAP.md regenerated via main_wet at the fixed point. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg --------- Co-authored-by: Claude <noreply@anthropic.com>
Summary
Generalizes
gunbc.host_converge_slice1'sper_slot_memory_max_bytescgroup-knob convergence (systemdMemoryMax, live systemctl show/set-property, independent read-back + reapply-noop proof) from srv1's single hardcoded probe unit to srv3's full runner set. srv3 units are live-enumerated (systemctl list-unitsoverrunner_instance_glob, never a guessed/hardcoded unit list) and cross-checked againstdeclared_runner_count()— a mismatch refuses typed rather than silently converging a subset. Per-unit outcomes are independent (one unit's refusal never blocks its siblings, per DESIGN §5's per-cause-typed-refusal/siblings-continue discipline), and a pre-applyMemoryCurrentusage guard refuses per-unit (typed, retryable) rather than applying a cap below live usage, so this can never SIGKILL an in-flight job.gunbc converge --host srv3now actually routes into this machinery via the CLI (see Review fixes below) — it was not reachable at all in the initial revision.Fixes fleet incident
adhoc-2f4ea1c1-548's root cause. #7002 (merged, incident-closing receipt: CI run 29848758856 on main — memory governor stayed within the 17.2GB cap, 16.1GB peak, zero eviction storm, zero exit-137, fix proven by execution) was the symptom mitigation — it stoppedread_host_budget_bytes()from re-sampling a noisyMemAvailablefallback on every cache eviction. That fallback was only reachable because srv3 runners had no discoverable private cgroup forread_host_budget_bytes()to resolve viacgroup memory.max/memory.high— this PR is that missing convergence, and its own displaced-cost citation is that same incident-closing receipt: once srv3 runners carry the same slice1 cgroup knobs srv1 already has, the host budget resolves through the real cgroup signal there too, and #7002's[floor-drain] degraded_budget_sourcediagnostic line should stop appearing on srv3.Also fixed, in passing: a genuine
.daglexer defect this file tripped over (a{immediately followed by non-whitespace containing a bare$-word inside anawkstring literal triggered the string-interpolation fast-path and produced a parser panic never caught because the file had never actually been compiled before this change) — worked around with a one-character space fix; not previously reported as a toolchain issue.First-execution safety
The apply arm (
systemctl set-property) is unreachable except viahost_converge_slice1_converge_unit's own control flow: live enumeration + cross-check must succeed before any unit is attempted, and per-unit the liveMemoryMaxread +Driftedverdict +host_converge_slice1_usage_guard's own liveMemoryCurrentread (returningSafe) must all succeed beforeset-propertyfires. So the first live invocation against srv3 is necessarily read-only up through several fail-closed gates before any write is reachable — no separate wet witness step is required by construction. Documented in the module's scope disposition.Review fixes (review 40879, cursor/composer-2.5)
gunbc converge --host srv3previously fell through to the old genericconverge_apply_for_hostpath, so the entire multi-unit machinery in this PR was dead code. Fixed:converge_cli_routes_to_slice1now matches srv1 and srv3, dispatching srv3 to a newconverge_cli_slice1_srv3_receipt()that callshost_converge_slice1_converge_srv3()and aggregates its per-unit outcomes into aConvergeCliReceipt.HostConvergeSlice1UsageGuardOutcome/HostConvergeSlice1UnitUsageGuardRefusedcarried bareIntand re-derived the current-vs-target comparison instead of reusingrunner_unit_live_read.runner_unit_usage_exceeds_bytes. Fixed: fields are nowstd.measure.ByteSize, andhost_converge_slice1_usage_guarddelegates the comparison to that existing typed authority.list-unitsread with zero active units collapsed tonone, same as a transport failure, so a real zero-unit state never reached the typedHostEnumerationMismatch. Fixed:host_converge_slice1_enumerate_unitsnow returnsPresent { value: [] }on a successful empty read, reservingnonefor genuine transport failure.Test plan
claim_batch --source-root dag --source-root src/v2 --entry dag/gunbc/host_converge_slice1.dag --functions host_converge_slice1_fixture_receipt,host_converge_slice1_receipt_is_valid,host_converge_slice1_cross_check_enumeration,host_converge_slice1_usage_guard— whole-tree resolve of the full ~285-module import closure, zero parse/type errors.claim_batch ... --entry dag/test/claim/host_converge_slice1_witness_test.dag --functions host_converge_slice1_wires_srv1_memory_cap,host_converge_slice1_wires_srv3_multi_unit_generalization— PASS on both, including hermetic fixture-based witnesses for the srv3 paths: read-onlyenumerate_unitsscript shape, cross-check mismatch/match routing againstdeclared_runner_count(), and usage-guard-refusal → typed-retryable-reason routing (now ByteSize-typed).claim_batch ... --entry dag/test/claim/fleet_converge_cli_witness_test.dag --functions fleet_converge_cli_witness_holds— PASS, including newwitness_converge_cli_srv3_routes_to_slice1_not_realize_frontier.[floor-drain] degraded_budget_sourceline stops appearing on srv3 runners — will report once observed.Closes incident
adhoc-2f4ea1c1-548.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com