feat(OMN-15255): composite runner readiness — one typed fleet view (6 signals, ALL-must-pass) + quarantine gate that stops the healthy-but-idle bounce storm

[jonahgabriel]

What / Why

Friction report §Recommended-4 / F-04 (docs/tracking/friction/2026-07-27-weekend-friction-report.md): "'Online' currently means registered, not ready to execute the governed workload."

Live measurement, 2026-07-27T16:40Z (OMN-15233 canary comment): the GitHub org registry read {total: 64, online: 64, busy: 0} while 53 of 64 containers read docker-unhealthy. Two surfaces, opposite verdicts, and no third surface adjudicating — the adjudication was a human diffing gh api .../actions/runners, docker ps, and per-container _diag mtimes by hand.

This PR makes node_runner_fleet_health_compute that third surface.

Composite readiness — a conjunction, not a precedence chain

readiness is READY only when all six independently-probed signals PASS. Every signal is evaluated every tick, including the passing ones:

Signal	PASS when
github_registration	registry reports online
docker_health	container running + health healthy (or none — image declares no healthcheck)
diag_heartbeat	_diag age ≤ RUNNER_HEALTH_MAX_DIAG_AGE_SECONDS (4500s, brackets the measured ~50-53 min token-refresh cycle — OMN-15233)
listener_topology	exactly one Runner.Listener, zero at PPID 1
container_stability	RestartCount ≤ CRASHLOOP_RESTART_THRESHOLD (5)
disk_capacity	host disk used < RUNNER_READINESS_MAX_DISK_USED_PERCENT (90)

The pre-existing state field is unchanged and still a first-match-wins precedence pick. It does not evaluate container health or listener topology at all — so the two legitimately disagree, and that disagreement is the F-04 finding. A GitHub-online runner with a fresh heartbeat, an unhealthy container and two listeners is state=HEALTHY and readiness=NOT_READY. Asserted directly in test_online_plus_unhealthy_container_plus_duplicate_listener.

Fleet rollup on the verdict: ready_count (the real capacity number, not online_count), not_ready_count, readiness_unknown_count, fleet_ready, quarantined_runners, bounce_eligible_runners, and per-signal readiness_signal_rollups (which surface disagrees, about how many runners).

Quarantine ≠ restart — the false-positive storm class

Two fail directions, deliberately opposite:

• Readiness fails CLOSED. UNKNOWN is not READY; an unprobeable runner is not counted as capacity.
• The bounce gate fails SAFE. A restart is recommended only when the runner is NOT_READY and sources were determinate and the runner is not github_busy and at least one failing signal is something a force-recreate can actually fix.

Consequences, each a test:

• GitHub-offline with a single non-orphaned listener + fresh heartbeat + healthy container → quarantined, never bounced (OMN-14057 status-lag corroboration, mechanized).
• Host disk over ceiling → quarantined, never bounced — recreating a container frees no host disk.
• Busy runner → never bounced, regardless of signals.
• Indeterminate source → never bounced, even with a confirmed FAIL.

Net-negative surface

Deleted: the four state-keyed RESTART_RUNNER branches in _recommend_for_assessment (CRASH_LOOPING 0.9 / LISTENER_ZOMBIE 0.85 / OFFLINE_IDLE 0.6 / WEDGED 0.5). Bounce-eligibility is now the single producer of a restart recommendation. Four independently-tunable confidence heuristics over the same facts is how one misread threshold (the retired 900s heartbeat window) became a fleet-wide restart storm with no second condition anywhere able to veto it.

Retired, rollout-gated (stated, not claimed): the manual three-surface comparison procedure in docs/runbooks/runner-fleet-listener-liveness.md is superseded by the fleet view, but remains the operator procedure until the extended probe is deployed. The runbook says so explicitly rather than implying the view is live.

Follow-up filed, not taken here: deleting collector_runner_health._classify_runner + EnumRunnerHealthState (the legacy CLI collector's duplicate classification) once the node is live-proven. Not taken in this PR because that surface is the one currently running.

Seams (OMN-14208)

Producer node_runner_health_snapshot_effect → consumer node_runner_fleet_health_compute, matched field-by-field:

Field	Type	Producer	Consumer
ModelRunnerFleetRunnerFact.docker_health	str ("" = unknown)	handler_runner_fleet_snapshot	_docker_health_signal
.listener_process_count	int | None	same	_listener_topology_signal
.orphaned_listener_count	int | None	same	same
ModelRunnerFleetSnapshot.host_disk_used_percent	float | None	_fetch_host_disk_used_percent	_disk_capacity_signal

Driven end-to-end by the real handler pair — test_pre_rollout_probe_output_degrades_to_unknown (producer side) and test_unprobed_readiness_facts_read_unknown_not_pass (consumer side) are the cross-boundary pair proving the unknown path, not two independent unit suites.

Rollout is inert by construction

The new facts only exist once the effect's extended SSH probe runs against the fleet. Until then all three signals report UNKNOWN → readiness UNKNOWN → nothing quarantined, nothing bounced, no recommendation emitted. Shipping this ahead of rollout cannot change fleet behavior. test_unprobed_readiness_facts_read_unknown_not_pass asserts exactly that.

No host mutation in this PR. No container recreated, no host file written, no cron changed, omninode-pc untouched — the OMN-15233 fan-out lane owns the runner-fleet host files concurrently. runner-monitor.sh is deliberately not edited: it is rsynced to the host by deploy-runners.sh, so a change on dev would land on the host as a side effect of that lane's sync. Wiring the bash monitor to consume this verdict is the rollout follow-up.

test_no_mutating_docker_verb_reaches_the_runner_host asserts against the command string actually dispatched over SSH (not the source text) that no docker restart|stop|rm|kill|start|compose|tag|commit or --force-recreate can reach the host.

RED → GREEN

Same 38 tests, held constant:

Baseline	Result
origin/dev src (tests applied, source reverted)	38 failed / 0 passed
This branch	38 passed

Falsifiability inside the change, not just against the old tree: test_the_retired_900s_threshold_reproduces_the_storm pins the retired threshold via monkeypatch and the identical healthy-idle fixture flips to 8/8 bounce-eligible with 8 restart recommendations, while test_idle_fleet_past_the_retired_900s_window_is_ready_and_unbounced (5 idle ages, 1000–4400s) reads 8/8 READY and zero restarts at the corrected default. test_a_genuinely_dead_listener_still_bounces (9000s) proves the recalibration did not disarm the check.

Per-signal RED discipline: test_single_failing_signal_makes_the_runner_not_ready is parametrized over six cases, each failing exactly one signal with the other five asserted PASS — the only shape that proves a signal is load-bearing rather than decorative.

Gates — run on .200 (ssh stickybeatz-studio), rule 11a

Patch-transfer discipline: patch sha256 2bc913cd…46e1a4 verified identical on both hosts; git diff --cached --stat identical (18 files, 1704 insertions, 47 deletions). Commit + push executed on .200.

• ruff check src/ tests/ — All checks passed
• ruff format --check — 4542 files already formatted
• mypy src/omnibase_infra/ — Success, 2622 source files
• pre-commit run --files <all changed> — all Passed/Skipped, no file rewritten by a hook
• scripts/check-env-reads.sh --base origin/dev — exit 0 (the disk-threshold read lands in the already-allowlisted handler_runner_fleet_health_evaluate.py; no new allowlist entry added — OMN-15234 still owns narrowing that matcher)
• Governed selector → {"selected_paths":["tests/"],"is_full_suite":true,"full_suite_reason":"shared_module"} → full suite running on .200; result posted to the ticket.

Contracts bumped to 1.1.0 on both touched nodes (new output fields = minor).

Not claimed

1. proof_class: receipt-bound, NOT deployed. No live fleet readback. The six signals have never been evaluated against real probe output — only against fixtures.
2. No live rollout. Follow-up owns deploying the extended probe and wiring the monitor's auto-bounce to bounce_eligible_runners.
3. Two Recommended-4 signals are absent, not stubbed: toolchain/image-contract and the tiny-governed-job probe. The signal enum extends without a model change.
4. No independent adversarial verifier agent has run against this change yet (rule 3).

Refs: OMN-15255 (child/related OMN-13915), OMN-15233, OMN-15177, OMN-14057, OMN-15234.

No merges, no --auto, no drafts. No OCC file modified by this branch.

Evidence-Ticket: OMN-15255
Evidence-Source: OCC#5159

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 43 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cfea07d8-d930-4d96-a15c-392196b83025

📥 Commits

Reviewing files that changed from the base of the PR and between 1517e66 and 32db718.

📒 Files selected for processing (18)

• docs/runbooks/runner-fleet-listener-liveness.md
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/contract.yaml
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/handlers/handler_runner_fleet_health_evaluate.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/__init__.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/enum_readiness_signal_outcome.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/enum_runner_readiness_signal.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/enum_runner_readiness_state.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/model_readiness_signal_rollup.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/model_runner_fleet_health_verdict.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/model_runner_health_assessment.py
• src/omnibase_infra/nodes/node_runner_fleet_health_compute/models/model_runner_readiness_signal.py
• src/omnibase_infra/nodes/node_runner_health_snapshot_effect/contract.yaml
• src/omnibase_infra/nodes/node_runner_health_snapshot_effect/handlers/handler_runner_fleet_snapshot.py
• src/omnibase_infra/nodes/node_runner_health_snapshot_effect/models/model_runner_fleet_runner_fact.py
• src/omnibase_infra/nodes/node_runner_health_snapshot_effect/models/model_runner_fleet_snapshot.py
• tests/unit/nodes/node_runner_fleet_maintain/test_runner_fleet_snapshot_effect.py
• tests/unit/nodes/node_runner_fleet_maintain/test_runner_readiness_composite_omn15255.py
• tests/unit/nodes/node_runner_fleet_maintain/test_runner_readiness_facts_effect_omn15255.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonah/omn-15255-composite-runner-readiness

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)

[jonahgabriel]

PAIR_INCOMPATIBLE with #2502 (OMN-15234) — flagged, half fixed here

Both PRs edit handler_runner_fleet_health_evaluate.py, model_runner_health_assessment.py, that node's contract.yaml, and tests/unit/nodes/node_runner_fleet_maintain/test_runner_fleet_health_compute.py. Both are individually green. That is the OMN-14208 shape, so stating the seams explicitly rather than letting the second merge discover them.

1. check-env-reads.sh allowlist — MECHANICAL, fixed on this side (4bc8caf03)

#2502 deletes this file's APPROVED_INFIX_PATTERNS entry and replaces it with per-name grandfathering ("this env-var name was already read in this file before the change"). RUNNER_READINESS_MAX_DISK_USED_PERCENT was a new name, so it would not be grandfathered and this PR's required check-env-reads gate would have gone RED the moment #2502 landed — regardless of merge order, because #2502 lands the narrowed matcher on dev.

Resolved here by making the disk ceiling a plain module constant. No new env-var name, no dependency on the allowlist entry #2502 removes, and no widening of an allowlist the other lane is deliberately narrowing (rule 10). #2502 needs no change for this half.

2. LISTENER_ZOMBIE restart corroboration — SEMANTIC, needs a decision, NOT taken unilaterally

#2502 adds _zombie_restart_corroboration: LISTENER_ZOMBIE → RESTART_RUNNER now additionally requires composite corroborating evidence.

This PR deletes the LISTENER_ZOMBIE restart branch outright (along with CRASH_LOOPING 0.9 / OFFLINE_IDLE 0.6 / WEDGED 0.5) and routes every restart recommendation through one rule — bounce-eligibility — which is strictly stronger than corroboration: determinate sources and not github_busy and at least one failing signal a force-recreate can actually fix.

Landing both as written leaves two mechanisms enforcing one rule, which is the condition that produced the storm in the first place (independently-tunable heuristics, no single veto). Textual conflict in _recommend_for_assessment is also guaranteed.

Proposed resolution — sequencing belongs to whoever owns the merges, not to either lane:

• Land fix(OMN-15234): narrow check-env-reads to per-name grandfathering (drop the OMN-15233 allowlist entry) + require composite evidence for LISTENER_ZOMBIE restarts #2502 first (it owns the gate narrowing, and this PR no longer depends on the allowlist entry it removes).
• Then rebase feat(OMN-15255): composite runner readiness — one typed fleet view (6 signals, ALL-must-pass) + quarantine gate that stops the healthy-but-idle bounce storm #2500 and delete _zombie_restart_corroboration, keeping bounce-eligibility as the single producer. Nothing is lost: every case corroboration would catch, bounce-eligibility already catches, plus the busy-runner and non-remediable-cause cases corroboration does not cover.
• Both nodes' contract.yaml version bumps will conflict; whoever rebases second takes the higher version.

If the opposite order is preferred, #2502 should drop _zombie_restart_corroboration before merging instead — but one of the two mechanisms has to go.

No merge, no --auto, no rebase performed from this lane.

[jonahgabriel]

Red-check triage — neither red is caused by this diff

Tests (Split 2/15) → CI Tests Gate → Test-Failure Ratchet Gate → CI Summary

Root-caused, not written off as transient (it is the second occurrence on this PR, so "flake" is not an available diagnosis).

All 12 failures are docker compose ... config returning exit 1 in tests/integration/infra/test_{judge,prod_runtime,stability_test_runtime,dev_runtime}_compose_render.py. This diff touches no compose file, no env file, and no docker path.

Falsification attempt, run against an independent lane: PR #2502 (OMN-15234, diff = scripts/check-env-reads.sh + the fleet-health handler/model/tests, no compose or env file) fails the identical 12 tests in its own Tests (Split 2/15) — run 30290604529, job on GitHub-hosted runner 1002215469. Two unrelated diffs, same failures, different runner instances (1002215203, 1002216466, 1002215469) ⇒ the cause is on dev, not in either branch.

Already filed and owned: OMN-15263 — OMN-15173's DEV_REDPANDA_ADVERTISE_HOST :? fail-fast (landed in #2495 at 14:43Z) makes every compose render fail closed. Filed 18:03Z by the OMN-15234 lane. No duplicate opened from here.

This PR is merge-blocked on OMN-15263 landing, not on anything in its own diff.

deploy-gate / deploy-gate

Correct, expected, and being fixed the sanctioned way — not skip-tokened.

DEPLOY GATE FAILED: PR touches runtime paths but no cited ticket has deploy DoD evidence.
Tickets found but declaring no falsifiable deploy probe: ['OMN-15255']

OCC#5159 adds the falsifiable deploy probe (plus its own self-bind after occ-preflight rejected the first push with pr_ticket_mismatch). Falsifiability verified in both directions in-session: exit 0 at product head 4bc8caf03, exit 1 at dev. It reads the product source over the contents API — not the receipt or contract this PR authors.

Known follow-up recorded on OCC#5159: the probe pins the branch head because #2500 has not merged; it should be re-pinned to the squash commit on dev after merge (reference_never_pin_a_feature_branch_head).

[jonahgabriel]

OMN-15263 fix is up: #2504 — your Tests (Split 2/15) red is not your diff.

The 12 docker compose config CalledProcessError failures in tests/integration/infra/test_{prod_runtime,judge,stability_test_runtime}_compose_render.py come from #2495 (OMN-15173) switching DEV_REDPANDA_ADVERTISE_HOST to the compose :? fail-fast form while adding it to only one of four render fixtures. Redpanda's command: block is interpolated regardless of --profile, so every layered render exits non-zero on a runner that has Docker.

#2504 supplies the var in the three hermetic fixtures (compose itself untouched — the fail-fast stays) and generalizes the OMN-5240 coverage gate to all registered render fixtures, fail-closed on unregistered ones.

Action for this PR: once #2504 lands on dev, rebase (or re-run the failed jobs on a base that includes it) and the Split-2/15 → CI Tests Gate → Test-Failure Ratchet Gate → CI Summary cascade clears. Nothing to change here.

Refs: OMN-15263, OMN-15173 (#2495), #2504