Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions contracts/OMN-13915.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-13915"
title: "fix(runners): listener liveness - heartbeat healthcheck, entrypoint watchdog, fleet-status canary"
summary: >-
Root-cause fix for the 2026-07-03 zombie fleet: 37/48 self-hosted runner listeners were dead-in-container
since ~2026-06-29 while every Docker healthcheck stayed green, because the healthcheck asserted container
liveness rather than listener liveness. omnibase_infra PR #2194 delivers three enforced layers: (1)
healthcheck.sh asserts the RUNNER_HOME-anchored Runner.Listener process is alive and its _diag heartbeat
is fresh and github.com egress works; (2) an entrypoint watchdog recycles the runner when the listener
dies under a still-alive wrapper tree; (3) a scheduled GitHub-hosted canary compares the org runner
registry against config/runner_fleet.yaml. Code plus PR evidence only; live fleet deploy remains a separate
operator action.
is_seam_ticket: false
interface_change: false
interfaces_touched: []
emergency_bypass:
enabled: false
justification: ""
follow_up_ticket_id: ""
dod_evidence:
- id: "dod-omnibase_infra-pr-2194"
description: >-
Consuming-PR binding: omnibase_infra PR #2194 at head a88b6840e2263e4ac000f08ffeb4a39dd22c3949 carries
the listener-aware healthcheck, entrypoint listener watchdog, scheduled runner-fleet canary, runbook,
focused CI tests, and integration coverage for the Docker Compose container-creation warning.
source: "manual"
checks:
- check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-omnibase_infra-pr-2194/command.yaml"
- id: "dod-deploy-plan"
description: >-
Runtime deploy-gate binding: PR #2194 touches docker/docker-compose.runners.yml only to add the
required container-creation warning. No live-fleet deploy, restart, or .201 mutation was performed;
the deployment action is gated to an explicit operator rollout after merge.
source: "manual"
checks:
- check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-deploy-plan/command.yaml"
- id: "dod-occ-main-promote"
description: >-
Self-binding OCC main-promotion receipt for the OMN-13915 central contract and deploy-gate-compatible
evidence.
source: "generated"
checks:
- check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-occ-main-promote/command.yaml"
23 changes: 23 additions & 0 deletions drift/dod_receipts/OMN-13915/dod-deploy-plan/command.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-13915"
evidence_item_id: "dod-deploy-plan"
check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-deploy-plan/command.yaml"
status: PASS
contract_sha256: "sha256:50c45b8651addb15be921ff263cb59d3e00c9782fad91aedbd3b93813020754b"
run_timestamp: "2026-07-04T19:34:00Z"
commit_sha: "a88b6840e2263e4ac000f08ffeb4a39dd22c3949"
runner: "codex-cli"
verifier: "codex-cli-review"
probe_command: "gh pr view 2194 --repo OmniNode-ai/omnibase_infra --json number,state,headRefOid,files"
probe_stdout: |
{"headRefOid":"a88b6840e2263e4ac000f08ffeb4a39dd22c3949","number":2194}
actual_output: >-
PASS: deploy evidence for PR #2194 is intentionally a non-mutating deploy plan. The PR changes a runtime
Docker Compose file only to add the explicit OMN-13915 container-creation warning requested by integration
coverage. No docker exec, service deploy, restart, or .201 mutation was run. Post-merge deployment remains
an explicit operator rollout step documented by the consuming PR and ticket evidence.
exit_code: 0
branch: "jonah/omn-13915-runner-listener-liveness"
pr_number: 2194
21 changes: 21 additions & 0 deletions drift/dod_receipts/OMN-13915/dod-occ-main-promote/command.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-13915"
evidence_item_id: "dod-occ-main-promote"
check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-occ-main-promote/command.yaml"
status: PASS
contract_sha256: "sha256:50c45b8651addb15be921ff263cb59d3e00c9782fad91aedbd3b93813020754b"
run_timestamp: "2026-07-04T19:34:00Z"
commit_sha: "f5e4a27df706cc969b930ef0c2095d65f42cc6ca"
runner: "codex-cli"
verifier: "codex-cli-review"
probe_command: "gh pr view 3595 --repo OmniNode-ai/onex_change_control --json number,state,headRefName,title"
probe_stdout: |
{"headRefName":"hotfix/omn-13915-occ-main-promote","number":3595,"state":"OPEN"}
actual_output: >-
PASS: this onex_change_control main-promotion PR carries contracts/OMN-13915.yaml and deploy-gate-compatible
receipts required by omnibase_infra PR #2194.
exit_code: 0
branch: "hotfix/omn-13915-occ-main-promote"
pr_number: 3595
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-13915"
evidence_item_id: "dod-omnibase_infra-pr-2194"
check_type: "command"
check_value: "grep -q '^status: PASS$' drift/dod_receipts/OMN-13915/dod-omnibase_infra-pr-2194/command.yaml"
status: PASS
contract_sha256: "sha256:50c45b8651addb15be921ff263cb59d3e00c9782fad91aedbd3b93813020754b"
run_timestamp: "2026-07-04T19:34:00Z"
commit_sha: "a88b6840e2263e4ac000f08ffeb4a39dd22c3949"
runner: "codex-cli"
verifier: "codex-cli-review"
probe_command: "gh pr view 2194 --repo OmniNode-ai/omnibase_infra --json number,state,headRefOid,headRefName,title"
probe_stdout: |
{"headRefName":"jonah/omn-13915-runner-listener-liveness","headRefOid":"a88b6840e2263e4ac000f08ffeb4a39dd22c3949","number":2194,"state":"OPEN"}
actual_output: >-
PASS: omnibase_infra PR #2194 is open at head a88b6840 and carries the OMN-13915 listener-liveness fix:
healthcheck.sh checks Runner.Listener plus _diag heartbeat freshness and egress; entrypoint.sh recycles
dead listener states; runner-fleet-canary scheduled workflow and script cover offline or missing runner
drift; docker/docker-compose.runners.yml now includes the required container-creation warning. Local
verification passed: tests/ci/test_runner_listener_liveness.py 20/20; integration coverage for the Compose
warning 1/1; git diff --check clean. No live-fleet mutation performed.
exit_code: 0
branch: "jonah/omn-13915-runner-listener-liveness"
pr_number: 2194
2 changes: 2 additions & 0 deletions src/onex_change_control/boundaries/migration_inventory.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,8 @@ databases:
tables: [migrations_log, plan_entitlements]
- file: "20260512_migrations_log_direction_cleanup.sql"
tables: [migrations_log]
- file: "20260703_tenant_credentials.sql"
tables: [migrations_log, tenant_credentials, tenants]

omniclaude:
connection_env: OMNICLAUDE_DB_URL
Expand Down
Loading