[jonahgabriel]

Summary

OMN-18411 -- caps the .201 self-hosted omnibase-ci runner fleet at 60 (down from 88). A CI burst drove one-minute load to 100.9 on the 32-core host, with two runner containers pinned at 200% CPU. Idle runners cost nothing, so the registered runner count is the worst-case concurrent-job ceiling on a fixed-core box.

• config/runner_fleet.yaml: expected_count/burst_count 88 -> 60.
• docker/docker-compose.runners.yml: removed the omninode-runner-61..88 service and runner-N-creds volume definitions (28 blocks), so a future deploy-runners.sh run cannot orphan-remove the live fleet inconsistently or resurrect the removed runners.
• Updated the two tests that hardcoded the prior fleet size (tests/unit/observability/runner_health/test_runner_fleet_config.py, tests/ci/test_runner_kubeconfig_mount.py's FLEET_SERVICE_COUNT) to assert 60, and fixed a pre-existing mypy --strict no-any-return finding in the latter file while touching it.

Out of scope, untouched: omninode-deploy-runner, the two customer-plane runners, every .201 compose lane (omnibase-infra, -stability-test, -judge, -lakshman), any host other than .201.

Correction against the dispatch premise: live readback at execution time showed 88 omnibase-ci runners registered, not the "92" first cited -- config/runner_fleet.yaml already read 88/88. The live fleet was brought down to 60 by removing the 28 highest-numbered idle runners (omninode-runner-61..88), not 32.

Live fleet change (already executed, out-of-band, matching this PR's repo state)

No full-fleet --force-recreate was run (that costs ~3h and was explicitly out of scope). Each of the 28 removals was: a fresh busy:false readback immediately before acting, docker stop on the host, gh api -X DELETE orgs/OmniNode-ai/actions/runners/<id>, then docker rm + docker volume rm for the container and its creds volume. Two runners (71, then later re-checked) were transiently busy and were skipped and retried once idle; no busy runner was removed.

Before:

$ gh api orgs/OmniNode-ai/actions/runners --paginate --jq '[.runners[]|select(.labels[].name=="omnibase-ci")]|length'
88
$ ssh omninode-pc.tail75df5e.ts.net "docker ps --format '{{.Names}}' | grep -c '^omninode-runner-'"
88

After:

$ gh api orgs/OmniNode-ai/actions/runners --paginate --jq '[.runners[]|select(.labels[].name=="omnibase-ci")]|length'
60
$ ssh omninode-pc.tail75df5e.ts.net "docker ps --format '{{.Names}}' | grep -c '^omninode-runner-'"
60
$ gh api orgs/OmniNode-ai/actions/runners --paginate --jq '.total_count'
63   # 60 omnibase-ci + omninode-deploy-runner + 2 customer-plane, unchanged

Readback at +5min and +15min after the last removal: fleet held steady at 60/60 both times (no auto-bounce resurrection, since the host's own copy of the two edited files was pushed ahead of the removals so the monitor's seq 1 $EXPECTED_RUNNERS never considered runners 61-88).

Test plan

• uv run pytest tests/unit/observability/runner_health/ tests/ci/test_runner_kubeconfig_mount.py tests/ci/test_incident_replay_omn18031.py tests/ci/test_runner_route_decision.py tests/ci/test_customer_plane_runner_credential_free_omn18392.py -q -- 245 passed
• uv run ruff format/ruff check clean; mypy --strict clean on touched files
• pre-commit run --all-files (via commit) -- all applicable hooks passed
• Live readback above: fleet at 60/60, stable at +5min and +15min
• gh pr checks -- pending, will confirm green before calling this done

Ledger: docs/tracking/ROLLING_WORK_LEDGER.md:8380 (OPERATOR-CONSENT row, both scope lists, verbatim operator quote).

Evidence-Ticket: OMN-18411
Evidence-Source: OCC#9733

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 8
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings demoted from threads (anchor rejected)

• [CRITICAL] hostile-reviewer (glm-review)

  No deregistration or teardown path for removed runners 61-88 | The diff deletes services runner-61 through runner-88 and their credential volumes from the compose file, but contains no migration, deregistration, or teardown step. The live host previously drifted ahead of the repo (per the config comment on OMN-15978), which establishes that the live fleet is provisioned out-of-band. Stopping the compose project will leave runners 61-88 registered with GitHub as offline runners and their containers and named

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Capping repo config does not shrink the live fleet; drift risk inverted | The stated motivation for OMN-15978 was that the host was scaled to 88 before the repo config caught up. This diff updates the repo to 60 but the live host presumably still runs 88. The test asserts the repo compose matches a 60-runner fleet, but nothing in the diff reconciles the host. Until the host is scaled down, the repo and live fleet diverge in the opposite direction from the previous incident, and deploy-runners.sh semantics

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  In-flight job loss when runners 61-88 are removed | The test docstring notes the deploy script can 'orphan-remove live runners'. If the fleet is converged from 88 to 60 while jobs are executing on runners 61-88, those jobs are killed mid-run. The diff contains no drain mechanism, job-completion wait, or labels-based quarantine for the removed runners. | Evidence: "deploy-runners.sh` cannot orphan-remove live runners beyond 60" (updated docstring), with no drain or graceful-shutdown logic present in the diff

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Capacity ceiling conflated with throughput; queueing impact unanalyzed | The comment claims idle runners cost nothing so 88 registered runners is what allowed load 100.9. That reasoning is incomplete: load reached 100.9 only because at least 60 concurrent jobs plus system overhead saturated the box, which means queued demand exceeded 60. Capping to 60 does not eliminate the overload condition on the next burst; it converts overload into queue growth on the GitHub side. The diff contains no analysis of expec

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Each runner mounts the host Docker socket | The deleted blocks show every runner mounts /var/run/docker.sock directly. This is pre-existing in the base template, but the diff rewrites the fleet documentation asserting load-based reasoning without addressing that any job on any runner gains host-root-equivalent access, which makes the 'fixed-core box' capacity planning secondary to the fact that CI burst content controls the host. Changing fleet topology without noting this is a missed review point on the ch

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Magic number 60 duplicated across three files | The cap appears in runner_fleet.yaml, FLEET_SERVICE_COUNT in test_runner_kubeconfig_mount.py, and hardcoded assertions in test_runner_fleet_config.py. The kubeconfig test hardcodes 60 rather than deriving it from config/runner_fleet.yaml, so a future cap change requires synchronized edits in at least three places; the prior 88-to-60 change demonstrates this already happened once. | Evidence: FLEET_SERVICE_COUNT = 60 in tests/ci/test_runner_kubeconfig_mount.py;

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  No test that deploy script tolerates the 60-cap transition | The updated test asserts compose matches 60 but does not exercise the deploy script's behavior when the live host reports more than expected_count runners during the transition. The docstring claims the script 'cannot orphan-remove live runners beyond 60', yet no test covers the 61-88 removal path or its interaction with in-flight jobs. | Evidence: "so deploy-runners.sh cannot orphan-remove live runners beyond 60" asserted only in prose. | Fix:

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Stale named volumes and containers for runners 61-88 will consume host resources | Removing volume declarations from compose does not remove existing runner-61-creds through runner-88-creds volumes or the containers. Until pruned, they persist as unmanaged state that docker compose down will no longer track, contributing disk usage and possible subnet-pool consumption if containers keep running, which interacts with the Docker subnet-pool alerting threshold referenced in the same config file. | Eviden

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Critical findings: 0
Major findings: 3
Total findings: 7
Models succeeded: glm-review

Note: Fewer than 2 reviewer models succeeded. Degraded results are informational (OMN-8468/OMN-8524) and do not block merge. Error: cli_review exit 2 (fewer than 2 models succeeded — partial/total outage)

---

Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface	Meaning	Blocks merge?
Review threads	Per-finding, posted by the reviewer	No (informational)
Hostile Review Thread Gate	Deterministic: unresolved hostile-reviewer threads exist	Fails until resolved (not yet a required context)
degraded verdict	Fewer than 2 models succeeded (infra)	No

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 5
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (glm-review)

  No removal path for registered runners 61-88 on the live host | The diff removes services 61-88 from compose and lowers expected_count to 60, but the referenced deploy script's behavior is only described in the test docstring as 'orphan-remove live runners beyond 60'. Nothing in the diff changes the deploy script itself, the runner deregistration flow, or documents cleanup of the GitHub-registered runners 61-88, their containers, and their named credential volumes (runner-61-creds .. runner-88-creds remain

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Capacity rationale is internally inconsistent; cap does not bound load | The comment claims 'idle runners cost nothing, so 88 registered runners is what let a burst reach that load.' A one-minute load of 100.9 on 32 cores implies roughly 100 concurrent runnable jobs, which requires 88 running (not idle) runners. Capping registration at 60 still permits 60 concurrent jobs on a 32-core host, a ~1.9x oversubscription; the change reduces the ceiling but does not bound load below saturation. No per-runner CPU/me

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  32% fleet reduction with no queue-depth impact analysis | Reducing concurrent capacity from 88 to 60 increases job queue wait time during burst periods. The diff contains no measurement of current peak concurrency utilization, no CI queue metrics baseline, and no follow-up plan to detect queueing regressions. The only signal cited (one-minute load 100.9) is a single incident. | Evidence: expected_count: 88 -> 60; no accompanying metric, alert, or ticket reference addressing CI queue latency impact. | Fix: A

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Config-consistency test does not assert yaml and fleet config agree | test_runner_fleet_config_loads_from_repo_config hardcodes 60 and test_runner_compose_reconciled_to_capacity_capped_60_fleet hardcodes 60 independently. The suite was already structured this way, but this diff touches both hardcoded values again; nothing ties FLEET_SERVICE_COUNT, expected_count, and the compose service count to a single source, so the next scale change can desynchronize silently. | Evidence: FLEET_SERVICE_COUNT = 60 in tes

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Untouched trailing-volume assertion now depends on implicit ordering | The retained assertion definition["volumes"][-1] == runner-N-creds... passes only because every remaining service lists the creds mount last. This diff edits services 1-60's neighborhood (deleting 61-88) without any test change covering runner-60's definition specifically; correct today, but the invariant is fragile and the diff reduces the fleet without verifying runner-60's own definition is complete and not truncated by the deletion b

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 1
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings not anchored to a changed file

• [MAJOR] hostile-reviewer (glm-review)

  Orphaned named volumes retain runner credentials after scale-down | The diff removes services runner-61 through runner-88 and their volume declarations, but named volumes declared with name: are not deleted by docker compose up or down unless -v/--remove-orphans --volumes is used. Volumes runner-61-creds through runner-88-creds will persist on the host containing runner registration tokens or credentials. The diff contains no decommissioning or cleanup step. | Evidence: Removal of `runner-61-c

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  GitHub runner registrations for 61-88 are not de-registered | Removing compose services stops the containers, but the corresponding GitHub Actions runner registrations remain in the org/runner group as offline entries. The plan's own reconciliation history (OMN-15978) shows the fleet was previously scaled ahead of the repo; the reverse drift now occurs: live registered runners 61-88 may still accept jobs until containers are stopped, and stale offline registrations accumulate unless a de-registration workfl

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  No test asserts runner indices are a suffix-free contiguous range independent of the cap | The updated test hardcodes range(1, 61) and FLEET_SERVICE_COUNT = 60 in a second file. The fleet size now exists in at least three places (config yaml, kubeconfig test constant, fleet config test assertions). The next resize requires editing all three; a drift between them produces test failures only after the fact. The changed lines perpetuate rather than fix this. | Evidence: tests/ci/test_runner_kubeconfig_moun

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Test edit to _parse_sync_paths return annotation is unrelated churn | The _parse_sync_paths change (binding the call to an annotated local then returning it) does not alter behavior and is unrelated to the fleet cap. It inflates the diff and obscures review scope. | Evidence: - return module.parse_sync_paths(DEPLOY_SCRIPT.read_text(encoding="utf-8"))

  • result: list[str] = module.parse_sync_paths(...)
  • return result | Fix: Revert or split into a separate change with justification (e.g., a linter

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Volume-to-service correspondence not asserted after removal | The test counts 60 credential volumes and 60 services separately but does not assert the volume names are exactly the set implied by the service indices. If a stale volume entry (e.g., runner-61-creds) were left behind, len(volume_names) == 60 could pass only if a required volume were also missing, and the mismatch would be undetected as a pairing check. | Evidence: "volume_names = {name for name in compose["volumes"] if re.fullmatch(r"runner-\

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (glm-review)

  Capacity regression unquantified; no evidence 60 is sufficient | The change caps concurrent-job ceiling from 88 to 60 based on a single observed load spike (100.9 one-minute load). No analysis of queue depth, job arrival rate, or CI latency impact is presented. If sustained demand exceeds 60 concurrent jobs, jobs queue and CI wall-clock latency regresses silently. The causal claim that registered-runner count drove the load is plausible but unverified; the 100.9 reading may have been transient CPU saturatio

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 6
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (glm-review)

  Cap does not actually bound host load | The stated remediation for the load-100.9 incident is that 'the registered runner count is the worst-case concurrent-job ceiling on a fixed-core box.' Every runner mounts /var/run/docker.sock, so a single runner job can spawn an unbounded number of containers, threads, or sub-jobs. Concurrent jobs are a lower bound on host work, not an upper bound. The change reduces the ceiling but does not establish a load bound, and no resource limits (cpus, pids_limit, mem_limit)

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  No lifecycle plan for the 28 removed runners | The diff removes services 61 through 88 from compose and their named volumes, but says nothing about deregistering the corresponding runners from GitHub, removing their containers, or pruning the orphaned named volumes (runner-61-creds through runner-88-creds) on the live host. Stale runner registrations will remain 'online' until they time out, distorting the fleet count the new expected_count of 60 is reconciled against, and dangling volumes with runner crede

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Scale-down race against in-flight jobs | The updated test asserts deploy-runners.sh 'cannot orphan-remove live runners beyond 60.' The diff contains no change to deploy-runners.sh and no evidence it handles jobs currently executing on runners 61-88 at deploy time. Removal of a runner service while a job is mid-flight either kills the job or leaves a zombie runner; the test comment implies removal behavior that is not shown or verified. | Evidence: so deploy-runners.sh cannot orphan-remove live runners bey

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Orphaned credential volumes persist after service removal | Named volumes runner-61-creds through runner-88-creds are removed from compose, but external named volumes are not deleted by removing their declarations. On the existing host these volumes, containing runner registration credentials, survive as orphans with no rotation or expiry path defined. | Evidence: - runner-88-creds:

  • name: omninode-runner-88-creds | Fix: Add a one-time cleanup step (docker volume rm) for the 28 orphaned volumes, or ver

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Fleet count duplicated across three files | The fleet size is now hand-maintained in config/runner_fleet.yaml, the compose file, and FLEET_SERVICE_COUNT in the test. This is the third reconciliation incident (72, 88, now 60) caused by exactly this duplication. A test-only constant guards drift but the constant itself must be manually bumped, and this diff proves the process fails under time pressure. | Evidence: FLEET_SERVICE_COUNT = 60 | Fix: Generate compose services and the test constant from runner_flee

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  No alert or threshold change accompanying the cap | The incident trigger was host one-minute load of 100.9 on 32 cores, yet the diff only changes the runner count. No load-based alert, admission control, or job-throttling hook is added, so the sole detection mechanism remains an operator noticing load after the fact. Docker subnet-pool alerting exists (OMN-12566); host load has no analogue. | Evidence: capped 88 -> 60 (OMN-18411) after a CI burst drove one-minute load to 100.9 on the 32-core host | Fix: Add

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).