CI floor OOM: cap split (PR-A) + honest width fold (PR-B)

docs/plans/budget-tree.md

@@ -51,22 +51,23 @@ The carrier is merged (#5582), but per §3/§5 above the tree is descriptive unt
 
 The original consumer-0 tree (§6) gave a single run-pool **all** of `host_ram − fixed_overhead`. That is wrong on a host that also runs the dashboard agent sessions: the runner pool and the dashboard sessions co-reside and over-commit the host together. B5 makes the co-tenancy structural.
 
-**Two-pool split (divide-once).** `host_ram` is divided **once** into three siblings: `fixed_overhead` (a `Guaranteed` claim) · `gha_runner_pool` · `dashboard_session_pool`. Conservation is structural (`node_conserves` recurses): `fixed_overhead + gha_runner_pool + dashboard_session_pool ≤ host_ram`. The runner pool is now the **derived remainder** (the cap): `gha_runner_pool = host_ram − fixed_overhead − dashboard_session_pool` (saturating, so an over-large dashboard reservation fails closed to a zero runner pool — the tree then refuses to conserve rather than silently over-commit). `dashboard_session_pool = per_container_cap × concurrency` (`per_container_cap` is the same single-authority container `memory.max` = `gunbc_ci_host_container_memory_cap` ~31.27 GiB shared with the floor-runner hard cap; interim concurrency = 2). deep-otter-528 **reads** `srv1_dashboard_session_pool_budget()` to drive dashboard admission; this session is the **single author** of the file, deep-otter does not edit it.
+**Two-pool split (divide-once).** `host_ram` is divided **once** into three siblings: `fixed_overhead` (a `Guaranteed` claim) · `gha_runner_pool` · `dashboard_session_pool`. Conservation is structural (`node_conserves` recurses): `fixed_overhead + gha_runner_pool + dashboard_session_pool ≤ host_ram`. The runner pool is now the **derived remainder** (the cap): `gha_runner_pool = host_ram − fixed_overhead − dashboard_session_pool` (saturating, so an over-large dashboard reservation fails closed to a zero runner pool — the tree then refuses to conserve rather than silently over-commit). `dashboard_session_pool = per_container_cap × concurrency` (`per_container_cap` = `gunbc_ci_dashboard_session_container_cap` ~31.27 GiB, inter-run session pool only — NOT the floor-runner OOM cap; interim concurrency = 2). deep-otter-528 **reads** `srv1_dashboard_session_pool_budget()` to drive dashboard admission; this session is the **single author** of the file, deep-otter does not edit it.
 
 *Follow-up (the session-class unlock, deep-otter lever #2, not in this PR).* Pinning every dashboard session to the 31.27 GiB floor-cap is the blunt-cap over-reservation #1799 already fixed for placement: the bulk of dashboard sessions are light agent/reviewer sessions (~2–4 GiB real, worst-observed ~9.5 GiB), not floor jobs. The unlock is a **session-class field** (light ~12 GiB vs floor-running ~31.27) so the session reservation is per-class; at light ≈ 12 GiB, concurrency 5–6 fits the session pool *and* the runner pool keeps ~49 GiB. It needs (a) deep-otter's longer-window light-session worst-observed and (b) a bright-stag/crisp-carp ruling on the session-vs-CI allocation, so it lands as a fast-follow with the accessor names unchanged. Interim concurrency = 2 preserves the runner pool until then.
 
 **Two numbers, two roles (the floor-runner leaf).** The `ci_run` leaf carries the per-run memory profile in two distinct fields — the k8s request-vs-limit shape — and deep-otter derives a different consumer from each:
 
-- **run-pool RESERVATION** = `srv1_ci_run_reservation()` = *typical* whole-run peak + margin (B4-measured ~10.8 GiB @ width 4 = `11592347648` bytes, run 28003119550, × 6/5 ≈ 12.96 GiB). This is the bin-packing "request": the `ci_run` leaf appropriation, what `R = floor(gha_runner_pool / reservation)` (`measure_fit_count_floor`) is derived from, and what the `.dag`-floor spawn-width reads (`srv1_floor_memory_budget()`). deep-otter derives the **run-pool reservation** from this.
-- **per-container HARD CAP** = `srv1_ci_run_container_hard_cap()` = the real container `memory.max` (`gunbc_ci_host_container_memory_cap` = `33578549248`, ~31.27 GiB = totalMem/4). This is the OOM-kill "limit" — the *physical* kernel ceiling, so it carries **no margin** (you cannot reserve headroom above what the host enforces). deep-otter derives the **per-container cap** from this. The two are never collapsed (`reservation < hard_cap`, a witnessed invariant).
+- **run-pool RESERVATION** = `srv1_ci_run_reservation()` = *typical* whole-run peak + margin (B4-measured ~10.8 GiB @ width 4 = `11592347648` bytes, run 28003119550, × 6/5 ≈ 12.96 GiB). This is the bin-packing "request": the `ci_run` leaf appropriation, what `R = floor(gha_runner_pool / reservation)` (`measure_fit_count_floor`) is derived from. deep-otter derives the **run-pool reservation** from this.
+- **within-run OOM CAP** = `srv1_floor_memory_budget()` = `srv1_ci_run_container_hard_cap()` = `gunbc_ci_runner_cgroup_memory_cap` (`8589934592`, 8 GiB, actions-runner@ MemoryMax). This is the OOM-kill limit for floor parallelism — the *physical* kernel ceiling for CI runner jobs, so it carries **no margin** (you cannot reserve headroom above what the host enforces). The `.dag`-floor spawn-width fold reads this cap (NOT the 31.27 GiB dashboard session cap).
+- **dashboard SESSION CAP** = `gunbc_ci_dashboard_session_container_cap` (`33578549248`, ~31.27 GiB = totalMem/4). Inter-run session pool sizing only; witnessed `runner_cap < session_cap`. Retired fused `gunbc_ci_host_container_memory_cap` nickname (PR-A #5910).
 
-A third measured field validates the cap is not a false-killer: **worst-observed** = `srv1_ci_run_worst_observed()` = the worst per-job cgroup `memory.peak` deep-otter's srv2 monitor has caught (`gunbc_ci_floor_runner_worst_max` = `32434110464`, ~30.2 GiB, actions-runner@srv2-09, a rust-gate floor job with rustc children). `reservation < worst_observed ≤ hard_cap` is witnessed: the worst real job (30.2) fits *under* the kill ceiling (31.27) — by only ~1 GiB. That ~1 GiB is a **risk to retire by shrinking the footprint, not by raising a cap the host can't afford**; if a future worst exceeds the container cap the witness goes red, which is the honest signal that jobs are about to OOM-kill.
+A third measured field calibrates **INTER-run** co-residence (distinct from the within-run 8 GiB cap): **worst-observed** = `srv1_ci_run_worst_observed()` = the worst per-job whole-run cgroup `memory.peak` deep-otter's srv2 monitor has caught (`gunbc_ci_floor_runner_worst_max` = `32434110464`, ~30.2 GiB, actions-runner@srv2-09, rust-gate floor job with rustc children). `reservation < worst_observed ≤ dashboard_session_cap` is witnessed: the worst whole-run job (30.2) fits under `gunbc_ci_dashboard_session_container_cap` (31.27 GiB) — by only ~1 GiB. That headroom is a **risk to retire by shrinking the footprint**, not by raising a cap the host cannot afford; if a future worst exceeds the session cap the witness goes red. A ~30 GiB whole-run peak does **not** fit the 8 GiB runner cgroup — that is expected; within-run floor parallelism is serialized (`spawn_width=1`) against the runner cap, not the session cap.
 
 **Construction-first, not validation.** `R = floor(gha_runner_pool / reservation)` is *derived*, so an over-wide run count is unwritable; the leaf appropriation is the *measured* typical peak + margin, so the floor's spawn-width is governed by a real number, not a `totalMem / R` guess. Conservation across the three siblings is the residue lens (`node_conserves`) over the constructed tree.
 
-**Reservation-fit vs hard-guarantee (the valve is load-bearing).** `srv1_runs_fit_in_pool()` = `floor(pool / reservation)` = 3 is the *bin-packing* count — it assumes typical usage, so a correlated burst of rust-gate jobs each spiking to the ~30 GiB worst would over-commit (3 × 30 > 49.45 GiB pool). The *hard-guarantee* count `srv1_runs_hard_guaranteed()` = `floor(pool / hard_cap)` = 1 is the number that never over-commits even if every job hits the container cap. `1 ≤ 3` is witnessed. The gap is covered by the host-RSS valve (deep-otter's admission backstop) — i.e. the reservation-based R is **valve-backstopped, not a hard guarantee**, exactly as the dashboard session pool is. The actuator that *enforces* R by start/stopping runner services (B5's "GHA-runner-pool memory cap") is a live-fleet apply and stays **operator-fenced** (§5); until it lands deep-otter holds the runner count at R=3 by hand, and the convergence proof is that the hand-walked count already equals the derived R.
+**Reservation-fit vs hard-guarantee (the valve is load-bearing).** `srv1_runs_fit_in_pool()` = `floor(pool / reservation)` = 3 is the *bin-packing* count — it assumes typical usage, so a correlated burst of rust-gate jobs each spiking to the ~30 GiB worst would over-commit (3 × 30 > 49.45 GiB pool). The *hard-guarantee* count `srv1_runs_hard_guaranteed()` = `floor(pool / worst_observed)` = 1 is the number that never over-commits even if every concurrent run hits the measured worst whole-run peak (not the 8 GiB within-run runner cap). `1 ≤ 3` is witnessed. The gap is covered by the host-RSS valve (deep-otter's admission backstop) — i.e. the reservation-based R is **valve-backstopped, not a hard guarantee**, exactly as the dashboard session pool is. The actuator that *enforces* R by start/stopping runner services (B5's "GHA-runner-pool memory cap") is a live-fleet apply and stays **operator-fenced** (§5); until it lands deep-otter holds the runner count at R=3 by hand, and the convergence proof is that the hand-walked count already equals the derived R.
 
-**Values — provenance.** typical/reservation ← **B4 (quick-lynx-78) verified** (`11592347648`, run 28003119550); worst-observed ← **deep-otter-528's srv2 cgroup `memory.peak` monitor** (`32434110464`); hard cap ← the measured container `memory.max`. The two-driver invariant (parent): the reservation side is driven by typical claim_executor batches (B4 measures it), the cap/worst side by rust-gate rustc *children* (only deep-otter's rust-gate-present monitor sees it; B4's no-rust-gate run under-reports it). NOTE: B4's #5619 dedup moves the *typical* self-RSS only ~0.57 GiB (8.7→8.1, NOT the earlier-retracted 8.7→4.2) and never touches the cap side. NOTE 2: `gunbc_ci_floor_runner_margin_num/den` (6/5 = 20%) was documented here as applying to the reservation, but its only code consumer was `within_run_spawn_budget()` (RETIRED in #5831 — was computing the wrong budget for within-run spawn-width, not the reservation). The margin constants (`gunbc_ci_floor_runner_margin_num/den`) and `gunbc_ci_floor_whole_run_peak_rss_bytes` are now deleted alongside `within_run_spawn_budget()` (no remaining consumers); the reservation reconciliation with deep-otter-528 is a separate runway.
+**Values — provenance.** typical/reservation ← **B4 (quick-lynx-78) verified** (`11592347648`, run 28003119550); worst-observed ← **deep-otter-528's srv2 cgroup `memory.peak` monitor** (`32434110464`); within-run runner cap ← `gunbc_ci_runner_cgroup_memory_cap` (8 GiB); dashboard session cap ← `gunbc_ci_dashboard_session_container_cap` (31.27 GiB). The two-driver invariant (parent): the reservation side is driven by typical claim_executor batches (B4 measures it), the cap/worst side by rust-gate rustc *children* (only deep-otter's rust-gate-present monitor sees it; B4's no-rust-gate run under-reports it). NOTE: B4's #5619 dedup moves the *typical* self-RSS only ~0.57 GiB (8.7→8.1, NOT the earlier-retracted 8.7→4.2) and never touches the cap side. NOTE 2: `gunbc_ci_floor_runner_margin_num/den` (6/5 = 20%) was documented here as applying to the reservation, but its only code consumer was `within_run_spawn_budget()` (RETIRED in #5831 — was computing the wrong budget for within-run spawn-width, not the reservation). The margin constants (`gunbc_ci_floor_runner_margin_num/den`) and `gunbc_ci_floor_whole_run_peak_rss_bytes` are now deleted alongside `within_run_spawn_budget()` (no remaining consumers); the reservation reconciliation with deep-otter-528 is a separate runway.
 
 Carriers: [dsl/gunbc/ci_budget_tree.dag](../../dsl/gunbc/ci_budget_tree.dag) (the two-pool tree + two-number leaf accessors), [dsl/gunbc/ci_floor_measurement.dag](../../dsl/gunbc/ci_floor_measurement.dag) (the measured carriers), witness [dsl/test/claim/ci_budget_tree_witness_test.dag](../../dsl/test/claim/ci_budget_tree_witness_test.dag).
 

dsl/gunbc/ci_budget_tree.dag

@@ -15,16 +15,17 @@ import gunbc.fleet_intent { srv1_offer, srv2_offer }
 import gunbc.ci_floor_measurement {
   HostFixedOverhead, HostFixedOverheadMeasured, HostFixedOverheadUnmeasured,
   gunbc_ci_host_fixed_overhead,
-  gunbc_ci_host_container_memory_cap,
-  gunbc_ci_session_reservation_mode
+  gunbc_ci_runner_cgroup_memory_cap,
+  gunbc_ci_dashboard_session_container_cap,
+  gunbc_ci_session_reservation_mode,
 }
 import gunbc.ci_runner_placement {
   SessionSliceResolution, SessionSliceResolved, SessionSliceUnresolved,
   resolve_session_slice
 }
-import gunbc.fleet_host_budget { gunbc_fleet_oomd_evidence, gunbc_per_runner_memory_cap }
+import gunbc.fleet_host_budget { gunbc_fleet_oomd_evidence }
 
-data gunbc_ci_budget_tree_deconfliction_disposition: Disposition = Terminal { reason: "DESIGN §3 deliberate carrier split: this BudgetNode governs INTER-run co-residence (host_ram partitioned as overhead + sessions_slice + runner_slice, conserving by construction via byte_size_saturating_sub). srv1_floor_memory_budget returns the measured per-runner cgroup cap (gunbc_per_runner_memory_cap = 8 GiB) -- the real binding constraint for within-run parallelism. Do NOT conflate with inter-run budget accounting; do NOT reintroduce within_run_spawn_budget (retired -- it used whole-run-peak x margin as the budget, which exceeded the real runner cgroup cap and caused OOM). See gunbc_ci_footprint_granularity_disposition." }
+data gunbc_ci_budget_tree_deconfliction_disposition: Disposition = Terminal { reason: "DESIGN §3 deliberate carrier split: this BudgetNode governs INTER-run co-residence (host_ram partitioned as overhead + sessions_slice + runner_slice, conserving by construction via byte_size_saturating_sub). srv1_floor_memory_budget returns gunbc_ci_runner_cgroup_memory_cap (8 GiB, within-run OOM authority) -- the real binding constraint for within-run parallelism. gunbc_ci_dashboard_session_container_cap (31.27 GiB) is inter-run session pool only. Do NOT conflate. See gunbc_ci_footprint_granularity_disposition." }
 
 fn host_overhead_bytes() -> ByteSize {
   match gunbc_ci_host_fixed_overhead {
@@ -127,9 +128,17 @@ fn srv1_fixed_overhead_bytes() -> ByteSize {
 }
 
 fn srv1_floor_memory_budget() -> ByteSize {
-  gunbc_per_runner_memory_cap
+  gunbc_ci_runner_cgroup_memory_cap
 }
 
 fn srv1_ci_run_container_hard_cap() -> ByteSize {
-  gunbc_ci_host_container_memory_cap
+  gunbc_ci_runner_cgroup_memory_cap
+}
+
+fn srv1_dashboard_session_container_cap() -> ByteSize {
+  gunbc_ci_dashboard_session_container_cap
+}
+
+fn witness_floor_budget_equals_runner_cgroup_cap() -> Bool {
+  byte_size_count(b: srv1_floor_memory_budget()) == byte_size_count(b: gunbc_ci_runner_cgroup_memory_cap)
 }