Skip to content

Runner host protection: conservation model + sound converge script (swap-off) + drift-gate heal - #5827

Merged
briansrls merged 9 commits into
mainfrom
session/runner-swap-off
Jun 25, 2026
Merged

briansrls merged 9 commits into
mainfrom
session/runner-swap-off

Conversation

@briansrls

@briansrls briansrls commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Makes a CI run unable to crash the host, and gives the operator a single sound, runnable script to apply the intent to the fleet.

1. Conservation allocation model (replaces the residual/gate tangle)

Host soundness is now declared-math conservation:
host_total ≥ runner_pool + session_pool + fixed_overhead + headroom (node_conserves).

  • runner_pool_budget = 80 GiB, per_runner_memory_cap = 8 GiB → declared_runner_count = 10.
  • session_container_memory = 1 GiB uniform demand class (shared with the ctrl session dashboard — reconciled as another capacity term in the same budget).

This rips out the 5-gate fail-closed cascade (RunnerSlotUnenforced chicken-and-egg, OomdUnverified, CtrlJobserverAbsent, HostFixedOverheadUnmeasured, headroom) that refused to derive a plan premised on a cap not yet live — the lock that meant nothing took effect. "Is the cap live on the host" is now a converge-receipt question (verdict=converged/drifted/absent), not a plan-soundness one.

2. Sound converge script with the swap-off safety knob

The live converge now emits a sound .github/fleet-converge.sh for srv1+srv2:

  • 80 GiB runner slice MemoryMax
  • 8 GiB per-slot MemoryMax
  • MemorySwapMax = 0 — converts host livelock (swap-thrash → reboot) into a scoped cgroup OOM-kill
  • runner_count = 10

FleetConvergeArtifact flipped to CommitRequired { consumer: HostReconciler }: the script is committed and drift-gated — edit the .dag, the committed script must regenerate to match or CI reds. That is the "changes actually take effect" reconciler (the gate is the change-detector; ssh-apply is the action).

3. Heal pre-existing ROADMAP.md drift (was reding main CI)

The committed ROADMAP carried newer scheduler/cache profiling status hand-edited without modeling into roadmap_authority.dag (#5745/#5813-class). Healed by modeling that content INTO the authority — no revert.

Verification

All green by execution (interpreter, sccache-independent): drift gate ExitSuccess; runner-placement, fleet-converge-emit, host-allocation-conservation, fleet-host-budget, and all roadmap witnesses pass.

To apply on a host

scp .github/fleet-converge.sh srv1: && ssh srv1 'sudo bash fleet-converge.sh' — emits converge-receipt lines (verdict per knob); idempotent.

Brian Searls and others added 8 commits June 25, 2026 22:09
Adds the missing crash-prevention property to the runner converge:
- extdeps/os/systemd: systemd_memory_swap_max_property = "MemorySwapMax"
- host_converge: per-slot swap-off knob (PerSlotMemoryCap, value 0) on a
  dedicated 30-fleet-swap.conf drop-in, paired with the existing MemoryMax cap.

Swap-off is the property that converts a host livelock (swap-thrash -> reboot,
the srv2 vector) into a scoped cgroup OOM-kill: an over-budget CI job dies, the
host survives. The knob is INERT until gunbc_ci_runner_slot_enforcement flips
from RunnerSlotUnenforced -> RunnerSlotEnforced (the converge emit is fail-closed
behind that gate; see ci_runner_placement.dag:313).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the 'whole host -> runners' residual + the chicken-and-egg
RunnerSlotUnenforced gate with a declared-partition conservation model:

  host_total  >=  runner_pool + session_pool + fixed_overhead + headroom

- gunbc_runner_pool_budget = 80 GiB (declared intent, was residual)
- gunbc_per_runner_memory_cap = 8 GiB (declared)  => declared_runner_count() = 10
- gunbc_session_container_memory = 1 GiB (uniform session demand class, ctrl-aligned)
- host_allocation_conserves() = node_conserves over the partition (primitive already
  in product/budget_tree.dag); over-commit ANY pool -> Unsound with a located reason.
- max_concurrent_sessions(pool) = floor(pool / 1 GiB), the ctrl admission bound.

Soundness becomes declared-math conservation; 'is the cap live' moves to the converge
receipt (no circular lock). Session pool is the uniform 1 GiB/1 core/swap-off/pids-4096
envelope agreed with ctrl (keen-dove-772) for session-dashboard container spawns.

5 witnesses green by execution (host_allocation_conservation_test.dag): count==10,
conserves when pools fit, Unsound when runner pool OR sessions over-commit, max
sessions derives from pool.

This is the CORE; cutting the live runner_deployment_plan over to it (rewrites ~4
gate-witnesses) is the next step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fail-closed tangle

The runner deployment plan no longer threads enforcement / oomd / jobserver /
overhead / headroom through a fail-closed cascade premised on a per-slot cap
that is not yet live on the host (the RunnerSlotUnenforced chicken-and-egg: the
converge script that applies the cap refused to emit until the cap was applied).

Plan soundness is now declared-math conservation: runner_count divides the
DECLARED runner_pool_budget by the DECLARED per_runner_memory_cap, and the host
is sound iff host_allocation_conserves (node_conserves over runner_pool +
session_pool + fixed_overhead + headroom <= host RAM). "Is the cap live on the
host" is no longer a plan-soundness question - it is a converge-receipt question
(verdict=converged/drifted/absent), which dissolves the circular lock.

Result: the live fleet converge now emits a SOUND script for srv1+srv2 -
80 GiB runner slice cap, 8 GiB per-slot MemoryMax, MemorySwapMax=0 (swap-off:
converts host livelock into a scoped cgroup OOM-kill), runner_count=10.

- ci_runner_placement.dag: delete RunnerSlotEnforcement gate, JobPeakResolution
  fold, host_runner_count_from_cap, runner_deployment_plan_for; replace with
  host_fixed_overhead_bytes / conservation_host_deployment /
  accumulate_conservation_deployment / runner_deployment_plan. Prune now-unused
  imports.
- runner_placement_witness_test.dag: flip the live-plan witnesses from
  fail-closed to conservation-sound (runner_count==10, per-slot cap==8 GiB,
  within runner pool); keep the session-reservation / operating-curve /
  cpu-weight / manifest witnesses; drop the deleted gate-cascade witnesses.
- fleet_converge_emit_test.dag: replace witness_live_unsound_fails_closed with
  witness_live_converge_is_conservation_sound; assert MemorySwapMax=0 swap-off
  knob in the fixture runner-knobs witness.

All affected witnesses green by execution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g ROADMAP drift

Two changes that make the runner converge an enforced single-authority artifact
the operator can run, and clear the drift gate that was reding main CI.

1. FleetConvergeArtifact: NotCommitted -> CommitRequired { consumer: HostReconciler }.
   The emitted .github/fleet-converge.sh is now committed and drift-gated: edit the
   .dag and the committed script must regenerate to match, else CI reds. This is the
   "changes actually take effect" reconciler - the gate is the change-detector; the
   ssh-apply is the action. Gitignore auto-drops the ignore line (it filters on
   !artifact_is_committed). generated_artifact_drift_test updated: HostReconciler arm
   added to the consumer match (exhaustiveness), commit-policy/ignore assertions
   flipped to committed.

2. ROADMAP.md drift (pre-existing on main, unrelated to the converge work): the
   committed ROADMAP carried newer scheduler/cache profiling status (resource-aware
   spawn_width nodes A-D, #5789 always-on resolve-cache) that was hand-edited without
   modeling into roadmap_authority.dag - the #5745/#5813-class single-authority
   violation. Healed by modeling that content INTO the authority (add 1-sched-resource-aware
   row, update 1-caching-forked / 2-p3 / 2-p4), so emit == committed with no revert
   of the profiling content.

Drift gate now ExitSuccess; all generated-artifact + roadmap + runner-placement
witnesses green by execution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title CI profiling Runner host protection: conservation model + sound converge script (swap-off) + drift-gate heal Jun 25, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review June 25, 2026 23:16
@gunbai-bot

gunbai-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for the review. Addressing all three:

Finding 1 (measure-native arithmetic) — fixed in 10f1c91. session_pool_bytes now uses measure_scale_fraction_floor(m, num, den) and declared_runner_count / max_concurrent_sessions use measure_fit_count_floor(capacity, each), so the magnitudes stay inside the ByteSize carrier instead of unwrap→arith→rewrap. The Nat results coerce at the Int return boundary exactly as the prior division did, so no consumer churn. You were right that this PR is the moment to consume the dissolved family rather than repeat it.

Finding 2 (session_pool: byte_size(0) on the live path) — intentional, follow-up confirmed. The runner-placement plan reserves the full declared runner_pool (80 GiB) and checks conservation against it; session pressure is bounded by the separate sessions.slice cap (its own live term in the converge), not by the runner-count derivation, so charging it here would double-count. The active-session count also isn't known at plan-derivation time — it's a live ctrl signal. Wiring a real session_pool term here is exactly the ctrl session-envelope integration (the next phase, coordinating with keen-dove-772): the host_allocation_conserves algebra already accepts the term and sessions_subtract_from_the_same_budget proves it bites, so the seam is ready for that consumer. byte_size(0) is the conservative-correct value until then (full runner pool reserved; sessions live under their own cap in headroom).

Finding 3 (converge_runner_width advisory stderr on increase) — by design. The receipt verdict is the authority: an increase yields effective < desired → drifted → host_failed=1, the fail-closed signal. The stderr line is operator-facing UX only (a runner increase needs the registration/install path, not set-property), which is why it's advisory and the receipt carries the truth — consistent with the "EMIT is the single authority" header (the knobs and receipts are what the model emits). The script is generated, so this is a model-level choice, not a hand-edit; not a defect, no change.

— sent from calm-carp-204

@gunbai-bot

gunbai-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Thanks — confirming the one follow-up so it isn't lost: the live session_pool: byte_size(0) on the runner-placement path is intentional (the runner plan reserves the full declared runner_pool; sessions are bounded by their own sessions.slice cap and the active count is a live ctrl signal not known at plan-derivation time). The host_allocation_conserves algebra already accepts the term and sessions_subtract_from_the_same_budget proves it bites, so the seam is ready for the ctrl session-envelope wiring (next phase, with keen-dove-772). Detail in my earlier reply on the first review. No code change needed for the APPROVE findings.

— sent from calm-carp-204

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant