Repository navigation
feat(OMN-13942): runner-fleet-maintain workflow — Increment 1 (detection core, read-only) - #2206
Merged
jonahgabriel merged 4 commits intoJul 5, 2026
Merged
Conversation
…ion core, read-only) Adds a canonical ONEX node workflow for runner-fleet health detection, closing the OMN-13932 buildx/codeload blind spot. Strictly read-only: no path in this PR restarts a runner, cancels a run, force-recreates a container, or dequeues/re-enqueues a merge-queue entry. New nodes (mirroring the same-repo node_merge_sweep_* trio and the omnimarket node_redeploy_orchestrator gate-then-effect-then-terminal-event topology): - node_runner_fleet_maintain_orchestrator (ORCHESTRATOR_GENERIC, new): coordinates the tick — start -> snapshot-gather -> health-evaluate -> terminal completed event. No I/O. - node_runner_fleet_health_compute (COMPUTE_GENERIC, new): pure/deterministic classifier. Ports collector_runner_health.py's crash-loop/offline logic, OMN-13915's _diag heartbeat-freshness rule, and OMN-13109's SILENT-WEDGE heuristic. Adds two NEW checks (BUILDX_UNAVAILABLE, CODELOAD_THROTTLED) that close the OMN-13932 gap. Emits recommended_actions that are recorded/ surfaced only — nothing here executes them. - node_runner_health_snapshot_effect (EFFECT_GENERIC): completed the OMN-6091 stub in place (same contract/topics declared since OMN-6082) with real gh-api + SSH/Docker gather logic, incl. the two new probes. Trigger: reuses OMN-13915's runner-fleet-canary 15-min GitHub-hosted schedule per design — no new parallel schedule. Also: - Register both new nodes in [project.entry-points."onex.nodes"] and regenerate the dispatch-selection-parity oracle and topic enums. - Add runtime_profiles: [effects] to all three new/completed contracts (ValidatorRuntimeProfiles gate). - Allowlist the 8 new infra-owned handler paths in infra-node-allowlist.txt (runner fleet is infra's domain object, not a market business workflow). - Baseline the orchestrator's inherent 3-hop payload_type_match ARCH-004 finding (same accepted shape as 6 sibling orchestrators already baselined). - Exempt two pattern-validator false positives (workflow_name, target_id — neither is an internal entity/UUID) with documented reasons. - Allowlist the external-trigger start topic in the subscribe-wiring-health script (triggered by the canary schedule, not a Kafka publisher). Increment 2 (recovery effects, FSM reducer, grant/cooldown gate) is design-only per the task — not built here; a follow-up ticket will be filed once this increment is reviewed. Evidence-Ticket: OMN-13942
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (46)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…er-image lock - Update ARCH-004 imperative-orchestrator baseline frozen-list tests to accept the new node_runner_fleet_maintain_orchestrator entry (OMN-13942): add it to EXPECTED_OWNER_TICKETS and bump risk-2 count 6->7. - Regenerate docker/runners/runner-image.lock.json: the two new node entry-points in pyproject.toml changed the manifest_digest (and shared_env_digest), so the bound runner-image identity was stale. Fixes Split 1/15 identity tests and runner-image-build-smoke. Evidence-Source: OCC#3607 Evidence-Ticket: OMN-13942
jonahgabriel
enabled auto-merge
July 5, 2026 03:31
test_release_backmerge_preserves_runner_identity_lock froze the pre-regen identity/shared_env digests. The two new node entry-points in pyproject.toml rebind the runner-image identity (binding-not-label by design), so update the frozen values to the regenerated, build-proven (runner-image-build-smoke pass) digests bc29a1.../d11587... Evidence-Source: OCC#3607 Evidence-Ticket: OMN-13942
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 5, 2026
jonahgabriel
enabled auto-merge
July 5, 2026 04:51
jonahgabriel
disabled auto-merge
July 5, 2026 05:13
jonahgabriel
enabled auto-merge
July 5, 2026 05:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a canonical ONEX node workflow for GitHub Actions self-hosted runner-fleet health detection, per the operator directive to move runner-fleet maintenance from ad-hoc cron/bash into a node-based workflow. Strictly read-only — no path in this PR restarts a runner, cancels a run, force-recreates a container, or dequeues/re-enqueues a merge-queue entry. Increment 2 (recovery effects, FSM cooldown reducer, grant-gated mutation) is design-only per the task and intentionally not built here.
Closes the live OMN-13932 blind spot (buildx loss + codeload throttling degraded the fleet 2026-07-04 with zero automated detection).
Nodes (mirrors the same-repo
node_merge_sweep_*trio + omnimarket'snode_redeploy_orchestratorgate-then-effect-then-terminal-event topology)node_runner_fleet_maintain_orchestrator(ORCHESTRATOR_GENERIC, new) — coordinates one tick: start → snapshot-gather command → health-evaluate command → terminalrunner-fleet-maintain-completedevent. No I/O.node_runner_fleet_health_compute(COMPUTE_GENERIC, new) — pure/deterministic classifier. Input: a facts-onlyModelRunnerFleetSnapshot. Output:ModelRunnerFleetHealthVerdict(per-runnerEnumRunnerFleetHealthState, fleet aggregates,recommended_actionsthat are recorded/surfaced, never executed). Portscollector_runner_health.py::_classify_runner, OMN-13915's_diagheartbeat-freshness rule, and OMN-13109's SILENT-WEDGE heuristic. Adds two new checks closing OMN-13932:BUILDX_UNAVAILABLE,CODELOAD_THROTTLED.node_runner_health_snapshot_effect(EFFECT_GENERIC) — completed the OMN-6091 stub in place (same contract/topics declared since OMN-6082, not forked) with real gh-api + SSH/Docker gather logic: runner registrations, Docker RestartCount,_diagheartbeat age, queue/run-age facts, and the two new probes (buildx availability, codeload-throttle log-signature grep). Read-only.Trigger: reuses OMN-13915's
runner-fleet-canary15-min GitHub-hosted schedule per the design — no third parallel schedule.Gates fixed along the way (all genuine, none bypassed)
[project.entry-points."onex.nodes"]and regenerated the dispatch-selection-parity oracle (tests/fixtures/dispatch_parity/baseline-selection-v2.json) and the topic enum (enum_omnibase_infra_topic.py) — both auto-generated, diff is additive-only.runtime_profiles: [effects]to all three contracts (ValidatorRuntimeProfilesgate, OMN-12957) — no allowlist entries needed.scripts/ci/infra-node-allowlist.txt(runner fleet is infra's domain object per the task's explicit layering call, not a market business workflow).payload_type_matchARCH-004 (H2) finding inarchitecture-handshakes/imperative-orchestrator-baseline.yaml— the exact same accepted shape as 6 sibling orchestrators already baselined (thin decision handlers, no decorative FSM). Verified the changed-node ratchet (--check-changed --ratchet, the actual pre-commit-blocking gate) passes clean.validate.py patternsfalse positives invalidation_exemptions.yaml:workflow_name(GitHub's own workflow display name, not an internal entity) andtarget_id(polymorphic runner-name-or-run-id string, not a UUID).scripts/check_subscribe_wiring_health.py's_EXTERNAL_PUBLISHER_ALLOWLIST(triggered by the canary GHA schedule, not a Kafka publisher — same category as the existingbuild-loop-start/skill.merge-sweepentries).Tests
tests/unit/nodes/node_runner_fleet_maintain/(18 tests, all passing):test_runner_fleet_health_compute.py— pure classification against fixture fleet states: healthy, saturated (mirrors the live 2026-07-04 incident: all-online zero-idle), crash-loop + listener-zombie, offline, zombie-run-candidate →CANCEL_RUN, buildx-unavailable, codeload-throttled, buildx-precedence-over-codeload, fleet-wide wedge, and a determinism check.test_runner_fleet_snapshot_effect.py— gh-api/SSH/Docker gather fully mocked (no live calls): forward-pass reconciliation, reverse-pass stale-registration, index-bounds filtering, buildx-unavailable probe, partial-source-failure degradation.test_runner_fleet_maintain_workflow_integration.py— full handler-chain wiring (start → gather → evaluate → completed), mirroringtest_merge_sweep_canary's pattern.No test in this PR restarts, cancels, or mutates anything real.
Verification
uv run ruff format/check,uv run mypy src/omnibase_infra— clean on all new/changed files.onex spdx fix --check src— clean.scripts/validate.py {architecture,contracts,patterns,unions,imports,any_types,declarative_nodes,migration_freeze}— clean.scripts/check_contract_topic_parity.py,scripts/validation/check_topic_literals.py— clean.bash scripts/ci/reject-node-implementations.sh --all— PASS.ValidatorRuntimeProfiles) — clean, no allowlist debt added.scanner_imperative_orchestrator_ratchet --check-changed --ratchet(the actual blocking gate) — clean.tests/integration/runtime/test_dispatch_selection_parity.py(10/10),tests/unit/scripts/test_check_subscribe_wiring_health.py(14/14),tests/unit/contracts/test_batch_validate_node_contracts.py,tests/ci/test_topic_config_contract_parity.py,tests/unit/observability/runner_health/,tests/integration/observability/,tests/unit/nodes/test_merge_sweep_canary/— all passing.uv run pytest tests/ -n auto -m "not slow": 24775 passed, 23 failed, 78 errors, 443 skipped — every failure/error reproduces on local-env gaps unrelated to this change (no live Postgres/Kafka/Docker/Infisical on this machine:test_db_handler_integration,test_dlq_tracking_integration,test_adapter_protocol_event_publisher_kafka(78 errors),test_provision_infisical, etc.), excepttests/ci/test_runner_image_identity.py::test_recorded_identity_matches_recomputed_identitywhich is a pre-existing stale Docker-image-lock drift onorigin/devat this branch's base commit (e74105c8) — verified zero diff touchesdocker/runners/runner-image.lock.jsonorscripts/ci/runner_image_identity.pyin this PR.@pytest.mark.slowtest_comprehensive_infra_scan(ARCH-004 full-repo scan,continue-on-error: truein CI, documented ~14min CI runtime) exceeds local wall-clock budget on this machine; not blocking (informational-only CI step).No mutation
Confirmed by design and by test:
HandlerRunnerFleetSnapshotperforms read-onlygh api/ssh/docker inspectcalls only (nodocker restart, nogh run cancel, nodocker compose up --force-recreate, no queue dequeue/re-enqueue anywhere in the dependency graph of these three nodes).node_runner_fleet_remediate_effectand the FSM cooldown reducer are Increment 2, design-only, not built.Evidence-Ticket: OMN-13942
Evidence-Source: OCC#3607
Evidence-Ticket: OMN-13942