Repository navigation
Runner host protection: conservation model + sound converge script (swap-off) + drift-gate heal - #5827
Conversation
Adds the missing crash-prevention property to the runner converge: - extdeps/os/systemd: systemd_memory_swap_max_property = "MemorySwapMax" - host_converge: per-slot swap-off knob (PerSlotMemoryCap, value 0) on a dedicated 30-fleet-swap.conf drop-in, paired with the existing MemoryMax cap. Swap-off is the property that converts a host livelock (swap-thrash -> reboot, the srv2 vector) into a scoped cgroup OOM-kill: an over-budget CI job dies, the host survives. The knob is INERT until gunbc_ci_runner_slot_enforcement flips from RunnerSlotUnenforced -> RunnerSlotEnforced (the converge emit is fail-closed behind that gate; see ci_runner_placement.dag:313). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the 'whole host -> runners' residual + the chicken-and-egg RunnerSlotUnenforced gate with a declared-partition conservation model: host_total >= runner_pool + session_pool + fixed_overhead + headroom - gunbc_runner_pool_budget = 80 GiB (declared intent, was residual) - gunbc_per_runner_memory_cap = 8 GiB (declared) => declared_runner_count() = 10 - gunbc_session_container_memory = 1 GiB (uniform session demand class, ctrl-aligned) - host_allocation_conserves() = node_conserves over the partition (primitive already in product/budget_tree.dag); over-commit ANY pool -> Unsound with a located reason. - max_concurrent_sessions(pool) = floor(pool / 1 GiB), the ctrl admission bound. Soundness becomes declared-math conservation; 'is the cap live' moves to the converge receipt (no circular lock). Session pool is the uniform 1 GiB/1 core/swap-off/pids-4096 envelope agreed with ctrl (keen-dove-772) for session-dashboard container spawns. 5 witnesses green by execution (host_allocation_conservation_test.dag): count==10, conserves when pools fit, Unsound when runner pool OR sessions over-commit, max sessions derives from pool. This is the CORE; cutting the live runner_deployment_plan over to it (rewrites ~4 gate-witnesses) is the next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fail-closed tangle The runner deployment plan no longer threads enforcement / oomd / jobserver / overhead / headroom through a fail-closed cascade premised on a per-slot cap that is not yet live on the host (the RunnerSlotUnenforced chicken-and-egg: the converge script that applies the cap refused to emit until the cap was applied). Plan soundness is now declared-math conservation: runner_count divides the DECLARED runner_pool_budget by the DECLARED per_runner_memory_cap, and the host is sound iff host_allocation_conserves (node_conserves over runner_pool + session_pool + fixed_overhead + headroom <= host RAM). "Is the cap live on the host" is no longer a plan-soundness question - it is a converge-receipt question (verdict=converged/drifted/absent), which dissolves the circular lock. Result: the live fleet converge now emits a SOUND script for srv1+srv2 - 80 GiB runner slice cap, 8 GiB per-slot MemoryMax, MemorySwapMax=0 (swap-off: converts host livelock into a scoped cgroup OOM-kill), runner_count=10. - ci_runner_placement.dag: delete RunnerSlotEnforcement gate, JobPeakResolution fold, host_runner_count_from_cap, runner_deployment_plan_for; replace with host_fixed_overhead_bytes / conservation_host_deployment / accumulate_conservation_deployment / runner_deployment_plan. Prune now-unused imports. - runner_placement_witness_test.dag: flip the live-plan witnesses from fail-closed to conservation-sound (runner_count==10, per-slot cap==8 GiB, within runner pool); keep the session-reservation / operating-curve / cpu-weight / manifest witnesses; drop the deleted gate-cascade witnesses. - fleet_converge_emit_test.dag: replace witness_live_unsound_fails_closed with witness_live_converge_is_conservation_sound; assert MemorySwapMax=0 swap-off knob in the fixture runner-knobs witness. All affected witnesses green by execution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g ROADMAP drift
Two changes that make the runner converge an enforced single-authority artifact
the operator can run, and clear the drift gate that was reding main CI.
1. FleetConvergeArtifact: NotCommitted -> CommitRequired { consumer: HostReconciler }.
The emitted .github/fleet-converge.sh is now committed and drift-gated: edit the
.dag and the committed script must regenerate to match, else CI reds. This is the
"changes actually take effect" reconciler - the gate is the change-detector; the
ssh-apply is the action. Gitignore auto-drops the ignore line (it filters on
!artifact_is_committed). generated_artifact_drift_test updated: HostReconciler arm
added to the consumer match (exhaustiveness), commit-policy/ignore assertions
flipped to committed.
2. ROADMAP.md drift (pre-existing on main, unrelated to the converge work): the
committed ROADMAP carried newer scheduler/cache profiling status (resource-aware
spawn_width nodes A-D, #5789 always-on resolve-cache) that was hand-edited without
modeling into roadmap_authority.dag - the #5745/#5813-class single-authority
violation. Healed by modeling that content INTO the authority (add 1-sched-resource-aware
row, update 1-caching-forked / 2-p3 / 2-p4), so emit == committed with no revert
of the profiling content.
Drift gate now ExitSuccess; all generated-artifact + roadmap + runner-placement
witnesses green by execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the review. Addressing all three: Finding 1 (measure-native arithmetic) — fixed in Finding 2 ( Finding 3 ( — sent from calm-carp-204 |
|
Thanks — confirming the one follow-up so it isn't lost: the live — sent from calm-carp-204 |
What this does
Makes a CI run unable to crash the host, and gives the operator a single sound, runnable script to apply the intent to the fleet.
1. Conservation allocation model (replaces the residual/gate tangle)
Host soundness is now declared-math conservation:
host_total ≥ runner_pool + session_pool + fixed_overhead + headroom(node_conserves).runner_pool_budget = 80 GiB,per_runner_memory_cap = 8 GiB→declared_runner_count = 10.session_container_memory = 1 GiBuniform demand class (shared with the ctrl session dashboard — reconciled as another capacity term in the same budget).This rips out the 5-gate fail-closed cascade (
RunnerSlotUnenforcedchicken-and-egg,OomdUnverified,CtrlJobserverAbsent,HostFixedOverheadUnmeasured, headroom) that refused to derive a plan premised on a cap not yet live — the lock that meant nothing took effect. "Is the cap live on the host" is now a converge-receipt question (verdict=converged/drifted/absent), not a plan-soundness one.2. Sound converge script with the swap-off safety knob
The live converge now emits a sound
.github/fleet-converge.shfor srv1+srv2:MemoryMaxMemoryMaxMemorySwapMax = 0— converts host livelock (swap-thrash → reboot) into a scoped cgroup OOM-killrunner_count = 10FleetConvergeArtifactflipped toCommitRequired { consumer: HostReconciler }: the script is committed and drift-gated — edit the.dag, the committed script must regenerate to match or CI reds. That is the "changes actually take effect" reconciler (the gate is the change-detector; ssh-apply is the action).3. Heal pre-existing ROADMAP.md drift (was reding main CI)
The committed ROADMAP carried newer scheduler/cache profiling status hand-edited without modeling into
roadmap_authority.dag(#5745/#5813-class). Healed by modeling that content INTO the authority — no revert.Verification
All green by execution (interpreter, sccache-independent): drift gate
ExitSuccess; runner-placement, fleet-converge-emit, host-allocation-conservation, fleet-host-budget, and all roadmap witnesses pass.To apply on a host
scp .github/fleet-converge.sh srv1: && ssh srv1 'sudo bash fleet-converge.sh'— emitsconverge-receiptlines (verdict per knob); idempotent.