Skip to content

host_converge_slice1: generalize per-slot MemoryMax convergence from srv1 to srv3 (multi-unit) - #7009

Merged
briansrls merged 18 commits into
mainfrom
session/royal-dove-562
Jul 22, 2026
Merged

briansrls merged 18 commits into
mainfrom
session/royal-dove-562

Conversation

@briansrls

@briansrls briansrls commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Generalizes gunbc.host_converge_slice1's per_slot_memory_max_bytes cgroup-knob convergence (systemd MemoryMax, live systemctl show/set-property, independent read-back + reapply-noop proof) from srv1's single hardcoded probe unit to srv3's full runner set. srv3 units are live-enumerated (systemctl list-units over runner_instance_glob, never a guessed/hardcoded unit list) and cross-checked against declared_runner_count() — a mismatch refuses typed rather than silently converging a subset. Per-unit outcomes are independent (one unit's refusal never blocks its siblings, per DESIGN §5's per-cause-typed-refusal/siblings-continue discipline), and a pre-apply MemoryCurrent usage guard refuses per-unit (typed, retryable) rather than applying a cap below live usage, so this can never SIGKILL an in-flight job. gunbc converge --host srv3 now actually routes into this machinery via the CLI (see Review fixes below) — it was not reachable at all in the initial revision.

Fixes fleet incident adhoc-2f4ea1c1-548's root cause. #7002 (merged, incident-closing receipt: CI run 29848758856 on main — memory governor stayed within the 17.2GB cap, 16.1GB peak, zero eviction storm, zero exit-137, fix proven by execution) was the symptom mitigation — it stopped read_host_budget_bytes() from re-sampling a noisy MemAvailable fallback on every cache eviction. That fallback was only reachable because srv3 runners had no discoverable private cgroup for read_host_budget_bytes() to resolve via cgroup memory.max/memory.high — this PR is that missing convergence, and its own displaced-cost citation is that same incident-closing receipt: once srv3 runners carry the same slice1 cgroup knobs srv1 already has, the host budget resolves through the real cgroup signal there too, and #7002's [floor-drain] degraded_budget_source diagnostic line should stop appearing on srv3.

Also fixed, in passing: a genuine .dag lexer defect this file tripped over (a { immediately followed by non-whitespace containing a bare $-word inside an awk string literal triggered the string-interpolation fast-path and produced a parser panic never caught because the file had never actually been compiled before this change) — worked around with a one-character space fix; not previously reported as a toolchain issue.

First-execution safety

The apply arm (systemctl set-property) is unreachable except via host_converge_slice1_converge_unit's own control flow: live enumeration + cross-check must succeed before any unit is attempted, and per-unit the live MemoryMax read + Drifted verdict + host_converge_slice1_usage_guard's own live MemoryCurrent read (returning Safe) must all succeed before set-property fires. So the first live invocation against srv3 is necessarily read-only up through several fail-closed gates before any write is reachable — no separate wet witness step is required by construction. Documented in the module's scope disposition.

Review fixes (review 40879, cursor/composer-2.5)

  • srv3 unreachable from CLI — confirmed valid; gunbc converge --host srv3 previously fell through to the old generic converge_apply_for_host path, so the entire multi-unit machinery in this PR was dead code. Fixed: converge_cli_routes_to_slice1 now matches srv1 and srv3, dispatching srv3 to a new converge_cli_slice1_srv3_receipt() that calls host_converge_slice1_converge_srv3() and aggregates its per-unit outcomes into a ConvergeCliReceipt.
  • Int vs ByteSize fork — confirmed valid; HostConvergeSlice1UsageGuardOutcome/HostConvergeSlice1UnitUsageGuardRefused carried bare Int and re-derived the current-vs-target comparison instead of reusing runner_unit_live_read.runner_unit_usage_exceeds_bytes. Fixed: fields are now std.measure.ByteSize, and host_converge_slice1_usage_guard delegates the comparison to that existing typed authority.
  • Empty enumeration conflated with absent read — confirmed valid; a successful list-units read with zero active units collapsed to none, same as a transport failure, so a real zero-unit state never reached the typed HostEnumerationMismatch. Fixed: host_converge_slice1_enumerate_units now returns Present { value: [] } on a successful empty read, reserving none for genuine transport failure.

Test plan

  • claim_batch --source-root dag --source-root src/v2 --entry dag/gunbc/host_converge_slice1.dag --functions host_converge_slice1_fixture_receipt,host_converge_slice1_receipt_is_valid,host_converge_slice1_cross_check_enumeration,host_converge_slice1_usage_guard — whole-tree resolve of the full ~285-module import closure, zero parse/type errors.
  • claim_batch ... --entry dag/test/claim/host_converge_slice1_witness_test.dag --functions host_converge_slice1_wires_srv1_memory_cap,host_converge_slice1_wires_srv3_multi_unit_generalization — PASS on both, including hermetic fixture-based witnesses for the srv3 paths: read-only enumerate_units script shape, cross-check mismatch/match routing against declared_runner_count(), and usage-guard-refusal → typed-retryable-reason routing (now ByteSize-typed).
  • claim_batch ... --entry dag/test/claim/fleet_converge_cli_witness_test.dag --functions fleet_converge_cli_witness_holds — PASS, including new witness_converge_cli_srv3_routes_to_slice1_not_realize_frontier.
  • Success criterion (post-merge, live): the [floor-drain] degraded_budget_source line stops appearing on srv3 runners — will report once observed.

Closes incident adhoc-2f4ea1c1-548.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

Brian Searls and others added 9 commits July 21, 2026 15:25
2026-07-21 fleet incident (redded main + every PR floor from ~12:17Z on
srv3-06/07): #6972's enforce_typed_cache_entry_cap re-derived its entry cap
from read_host_budget_bytes() on every cache insert. On CI runners no
private cgroup memory.max/memory.high is discoverable, so the derivation
fell through to live /proc/meminfo MemAvailable — a host-wide signal shared
across co-resident sessions. Re-reading it per insert let the cap chase host
noise within a single run (e.g. 701->682->...->204->325), and each eviction's
recompute-on-miss added pressure exactly when pressure was already high — a
thrashing feedback loop ending in claim_executor SIGKILL (exit 137).

Fix, per signed direction: typed_module_cache_cap_derivation() factors the
existing env-override/budget-read logic out of typed_module_cache_max_entries
(kept as the pure per-call form for existing tests) and adds a `degraded`
flag. MultiEntryIndex gets a `typed_module_cache_cap: OnceCell<usize>` field;
the new typed_module_cache_cap(index) accessor samples the derivation exactly
once per index lifetime and, on that first call, logs a typed, counted
`[floor-drain] degraded_budget_source` line when the source isn't a private
cgroup limit (an honesty arm, not a widened failure -- the cap still derives
from whatever source was found). enforce_typed_cache_entry_cap and
emit_floor_drain_receipt now go through this accessor instead of re-deriving
per call. No floor-pin added; no env-var escape hatch.

Witnesses: typed_module_cache_cap_sampled_once_per_index_stays_stable_despite_signal_drift
proves the cap holds fixed across a live signal move within one index's
lifetime. uncached_derivation_tracks_moving_signal_the_sampled_once_accessor_must_not
is the RED control, reproducing the pre-fix oscillation shape (cap values
drawn from the incident's own log: 300/227/294/204/325) via the still-live
uncached derivation, so a regression that reverts the runtime call sites back
to the uncached form is caught.

Evidence trail: srv3-05 11:50Z run 29827692954 and srv3-06 12:17Z run
29829512709 both show the eviction-storm-then-Killed(137) signature; the
wedged 3.5h+ run 29829313521 was cancelled after pulling final evidence
(build job succeeded, ci job's floor-run step never emitted a single
[floor-drain] line before being starved by host contention) -- noted here so
the cancellation isn't mistaken for a hidden failure.

Contained to the #6972 mechanism + its two call sites, per manager sign-off;
no PR-beta/SpacePacked dissolve-on work included.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-blocking review observation from quick-carp-521 on #7002: the
degraded-source check string-scans read_host_budget_bytes' display label.
Fine for now (it returns (Option<u64>, String)), but if that fn ever grows
a typed source enum, this check should ground on it instead of re-parsing
the label — a one-line note so it doesn't cement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rst-execution safety

awk '{print $1}' triggered a string-interpolation lexer fast-path on the
bare $ inside an unspaced brace, producing a parser panic never caught
because the file had never been compiled. Space-padded the braces.

Also records why the apply arm already satisfies first-execution safety
by construction: enumeration + cross-check + per-unit live read + usage
guard must all succeed before set-property is reachable, so the first
live invocation against srv3 is necessarily read-only up to several
fail-closed gates.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title WAIT FOR DM BRIEF FROM quick-carp-521 BEFORE ACTING. FLEET INCIDENT: CI floor OOM-kill class — claim_executor exit 137 after typed_cache_eviction storm (cap 294->227, 2300+ evictions), redding main + every PR floor since 12:17Z on srv3-06 AND srv3-07. Diagnose the differential (12:14Z merge corpus g host_converge_slice1: generalize per-slot MemoryMax convergence from srv1 to srv3 (multi-unit) Jul 21, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review July 21, 2026 17:45
@cursor

cursor Bot commented Jul 21, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

…enumeration handling

Addresses review 40879 (cursor/composer-2.5) on PR #7009:
- Finding 1: gunbc converge --host srv3 now routes into
  host_converge_slice1_converge_srv3()/converge_host() via a new
  converge_cli_slice1_srv3_receipt() aggregator, instead of silently
  falling through to the generic converge_apply_for_host frontier.
  The srv3 multi-unit enumeration/cross-check/usage-guard machinery
  was previously unreachable from any CLI entrypoint.
- Finding 2: HostConvergeSlice1UsageGuardOutcome and
  HostConvergeSlice1UnitUsageGuardRefused now carry ByteSize instead
  of bare Int, and host_converge_slice1_usage_guard reuses
  runner_unit_usage_exceeds_bytes instead of re-deriving the
  current-vs-target comparison.
- Finding 3: host_converge_slice1_enumerate_units now returns
  Present{value: []} on a successful read with zero active units,
  distinct from none (transport/read failure), so a real zero-unit
  state reaches the typed HostEnumerationMismatch refusal instead of
  being conflated with a generic HostRefused.

Adds witness coverage for the srv3 routing predicate and the
ByteSize-typed usage-guard-refused legacy-result mapping.
@gunbai-bot

gunbai-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Addressed all three findings from review 40879 (cursor/composer-2.5) with code fixes, commits e8b68e1/df7f88f/51a8e32 on this branch:

Finding 1 (srv3 unreachable from CLI) — confirmed valid. gunbc converge --host srv3 now routes through a new converge_cli_slice1_srv3_receipt() in fleet_converge_cli.dag, which calls host_converge_slice1_converge_srv3() and aggregates its per-unit outcomes (via new host_converge_slice1_host_result_applied_count/_all_valid/_refusal_reason helpers in host_converge_slice1.dag) into a ConvergeCliReceipt. converge_cli_routes_to_slice1 now matches both srv1 and srv3, dispatching to the right receipt builder. Added witness_converge_cli_srv3_routes_to_slice1_not_realize_frontier to fleet_converge_cli_witness_test.dag.

Finding 2 (Int vs ByteSize fork) — confirmed valid. HostConvergeSlice1UsageGuardOutcome's current_bytes/target_cap_bytes and HostConvergeSlice1UnitUsageGuardRefused's matching fields are now std.measure.ByteSize, not bare Int. host_converge_slice1_usage_guard now builds a minimal RunnerUnitMemoryLiveRead from the live MemoryCurrent read and delegates the actual current-vs-target comparison to runner_unit_live_read.runner_unit_usage_exceeds_bytes instead of re-deriving it via byte_size_count → Int.

Finding 3 (empty enumeration conflated with absent read) — confirmed valid. host_converge_slice1_enumerate_units now returns Present { value: [] } on a successful list-units read with zero matching lines, reserving none for genuine transport failure (result.success == false). A real zero-active-units state now reaches host_converge_slice1_cross_check_enumeration and produces the typed HostConvergeSlice1HostEnumerationMismatch (carrying declared_count and the empty observed_units) instead of the generic HostRefused.

Full-tree claim_batch closure (283-285 modules, ~6300 resolved items) is green across both touched witness suites after the fixes.

— sent from royal-dove-562

@gunbai-bot

gunbai-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Addressed the naming note from review 40904: renamed HostConvergeSlice1HostConverged → HostConvergeSlice1HostEnumerated in dag/gunbc/host_converge_slice1.dag. The variant only carries the enumerated per-unit outcome list — actual convergence is decided downstream by host_converge_slice1_host_result_all_valid — so the old name asserted a verdict the type didn't itself guarantee. All 5 in-tree usage sites were contained to that one file; rebuilt and reran both host_converge_slice1_witness_test.dag witnesses green after the rename.

reviews 40887, 40895, 40901 raised no actionable findings (their "considered and dropped" notes were already consistent with the current design), so no further action was needed for those.

— sent from royal-dove-562

Brian Searls and others added 2 commits July 21, 2026 21:23
…ual_site_map_2026-07-21.md

The doc landed on main via #7023 with no doc-graph link, which is a
pre-existing main-red (main failed at 50bb7e9 and d36515b too) that
bled into this PR's CI once it merged main. Mirrors the existing
bind: pattern in this file for curated_cargo_frontier_probe_report.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

CI failure at f565bee root-caused: doc_graph_has_no_orphan_docs failed on docs/probes/emitter_residual_site_map_2026-07-21.md — a probe doc that landed on main via #7023 with no doc-graph link (bind: row). This is a pre-existing main-red, not introduced by this PR's own diff: main itself is failing CI at both 50bb7e9d (the #7023 commit) and the current tip d36515b6, confirmed via gh run list --branch main. It only surfaced here because this branch merged main.

Fixed in 29bbcdd58a — added a bind: docs/probes/emitter_residual_site_map_2026-07-21.md provenance row in dag/tools/self_host_curated_probe_cargo.dag, mirroring the existing pattern for the sibling curated_cargo_frontier_probe_report.md bind row in the same file. Verified locally: reimplemented the doc-graph reachability walk and confirmed 0 orphans, then ran the actual witness via claim_batch — doc_graph_has_no_orphan_docs, doc_graph_has_no_dangling_links, doc_graph_universe_is_nonempty all PASS.

— sent from royal-dove-562

Brian Searls and others added 2 commits July 21, 2026 22:56
# Conflicts:
#	dag/tools/self_host_curated_probe_cargo.dag
…r_count gap with named dissolution trigger

host_converge_slice1_cross_check_enumeration's host param labels the
mismatch record only; declared_runner_count() is fleet-wide by design
(one global budget, no per-host divergence exists yet). Modeling a
per-host budget now would be speculative per DESIGN §6. Recorded as a
named, triggered gap on the scope-disposition Terminal note instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

review 41089's finding is valid: host_converge_slice1_cross_check_enumeration's host param only labels the HostConvergeSlice1EnumerationMismatch record — the comparison itself goes through declared_runner_count(), which is fleet-wide (gunbc_runner_pool_budget is one global ByteSize, no per-host budget exists in the model today).

I didn't build a per-host declared-count model for this, though — every fleet host today shares one declared count, so that would be speculative modeling ahead of a real second-host divergence (DESIGN §6: don't build for a host that doesn't yet diverge). Instead I landed the gap as a named, triggered note on the scope-disposition Terminal doc string in dag/gunbc/host_converge_slice1.dag (commit 2ac9512f98): the day a second host's declared runner count genuinely differs from srv3's, declared_runner_count gets a host/HostIdentity param backed by a real per-host budget row, and this cross-check wires straight to it. Verified green via claim_batch on both host_converge_slice1_witness_test.dag entries.

— sent from royal-dove-562

@briansrls
briansrls merged commit 49bdd93 into main Jul 22, 2026
3 checks passed
@briansrls
briansrls deleted the session/royal-dove-562 branch July 22, 2026 03:03
briansrls pushed a commit that referenced this pull request Jul 22, 2026
…n the operator TODO

Rosters the 3 unrostered non-fold-residue sites that have kept the
scheduled cold falsifier red since 2026-07-16 (alert #7032), each with
its class reason and dissolution trigger per the #6691 shape:
host_converge_slice1 host_result_all_valid / host_result_applied_count
(legacy-result-bridge projections, #7009) and host_hygiene_reaper
action_kills_build_cache_server (two-axis discriminating predicate,
#7026). Receipts: nfr_roster_receipt unrostered=0 stale=0 (was red with
exactly these 3 sites before the edit - the discriminating control),
non_fold_residue_clean_holds evaluates true by execution, fmt clean.

Sheet: falsifier row marked landed, wave-reds row credits #7082 for the
drift close, new CI-ergonomics lane row (operator ask 2026-07-22) under
item 2. ROADMAP.md regenerated via main_wet at the fixed point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg
briansrls added a commit that referenced this pull request Jul 22, 2026
… + wave-red receipts and CI-ergonomics lane on the operator TODO (#7083)

* roadmap: heal the #7079-reopened ROADMAP drift + record wave-red receipts in the operator TODO

#7079's branch predated #7081, so its drift cleanup hand-deleted the
silent-ibex-417 paragraph from the generated ROADMAP.md while #7081 had
just re-homed that paragraph into roadmap_authority.dag - post-merge
main drifts the opposite way (authority has it, committed md does not)
and the drift gate reds the next main run that reaches batch 4. This
regenerates ROADMAP.md from the authority, closing the drift.

Sheet updates: wave-red receipts row (drift-gate red healed by #7081;
stale CommitWitnessClaim roster row from #7060 hotfixed by #7079 incl.
the #7064 orphan; the third collision above), the stale-roster
stabilization-loop instance, and #7076/#7008 marked merged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg

* falsifier green: nfr roster burn-down (3 rows) + CI-ergonomics lane on the operator TODO

Rosters the 3 unrostered non-fold-residue sites that have kept the
scheduled cold falsifier red since 2026-07-16 (alert #7032), each with
its class reason and dissolution trigger per the #6691 shape:
host_converge_slice1 host_result_all_valid / host_result_applied_count
(legacy-result-bridge projections, #7009) and host_hygiene_reaper
action_kills_build_cache_server (two-axis discriminating predicate,
#7026). Receipts: nfr_roster_receipt unrostered=0 stale=0 (was red with
exactly these 3 sites before the edit - the discriminating control),
non_fold_residue_clean_holds evaluates true by execution, fmt clean.

Sheet: falsifier row marked landed, wave-reds row credits #7082 for the
drift close, new CI-ergonomics lane row (operator ask 2026-07-22) under
item 2. ROADMAP.md regenerated via main_wet at the fixed point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fdkaGGLUKpLRwwqxp5sLg

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant