Repository navigation
feat(OMN-20594): the protected delegation lane's chain canary reaches the operator on a red, and the lab alarm backstops it - #4577
Conversation
… the operator on a red, and the lab alarm backstops it The h201 dev lane is the protected delegation lane (operator in session 2026-10-05). chain-canary.yml (C15) now posts to the operator Slack channel when a scheduled run's verdict changes (red after non-red, recovery after red), through the SLACK_BOT_TOKEN and SLACK_CHANNEL_ID secrets the fleet canary already alerts through. scripts/lab_alarm.py gains the delegation_chain_canary condition, which reads the C15 verdict through the OMN-18866 workflow-verdict reader (scheduled runs, 3h window) and raises once when it is red or silent. The lane manifest and the generated lane boundary name the dev lane as the protected delegation lane and list its guards. Onex-Lane: deleg-guard-9143 Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…tered listing The filtered run listing intermittently omits the newest runs (OMN-20277); read live, it named a stale failure as the previous verdict on one of two calls. Read the unfiltered listing and filter and order it here. Onex-Lane: deleg-guard-9143 Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…work and parse errors it expects Onex-Lane: deleg-guard-9143 Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…string Onex-Lane: deleg-guard-9143 Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…12903) * evidence(OMN-20594): author OCC companion for OmniNode-ai/omnibase_infra#4577 OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 09cb2f2434ea3f718110d0cac7b93b058527f872. * evidence(OMN-20594): self-bind OCC#12903 + rebind contract_sha256 --------- Co-authored-by: omnimarket-bot <bot@omninode.ai>
There was a problem hiding this comment.
Hostile Reviewer — adversarial findings (OMN-17492)
Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 13
Nit-level findings suppressed: 0
The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.
Below quorum: 13 finding(s) raised by one model only (OMN-18479)
These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.
- [MAJOR]
.github/workflows/chain-canary.yml, alert step env block(qwen3-review) — Slack token and channel ID exposed in process environment | The alert step passes SLACK_BOT_TOKEN and SLACK_CHANNEL_ID into the step's env block. In GitHub Actions, step-level env values are visible i - [MAJOR]
.github/workflows/chain-canary.yml, previous_conclusion function(qwen3-review) — previous_conclusion sorts by created_at string, not ISO-8601 timestamp | The functionprevious_conclusionsorts admitted runs bystr(run.get("created_at", ""))in reverse. The GitHub API returns ` - [MAJOR]
tests/ci/test_chain_canary_workflow.py, test_alert_posts_only_on_a_change_of_verdict(qwen3-review) — decide() treats 'cancelled' as no-op, but a cancelled run after a red should not suppress a subsequent red alert | Thedecidefunction returnsNonewhenjob_status == "cancelled", meaning no ale - [MAJOR]
.github/workflows/chain-canary.yml, read_receipt function(qwen3-review) — read_receipt uses open() without closing the file handle | Theread_receiptfunction opens a file withopen(path, encoding="utf-8").read()without using a context manager. In CPython, the file is - [MAJOR]
scripts/lab_alarm.py(gpt-oss-review) — ModelAlarmConfig.load now unconditionally accesses new keys | The load() method reads payload["chain_canary_workflow"], payload["chain_canary_branch"] and payload["chain_canary_max_age_minutes"] witho - [MAJOR]
scripts/lab_alarm.py(gpt-oss-review) — Missing import for subprocess in exception handling | evaluate_delegation_chain_canary catches subprocess.SubprocessError, but the file does not import the subprocess module. If the reader raises a Su - [MAJOR]
scripts/lab_alarm.py(gpt-oss-review) — Potential mismatch in condition count expectations | The addition of DELEGATION_CHAIN_CANARY changes the total number of evaluated conditions from six to seven. Other components (e.g., serialization, - [MINOR]
.github/workflows/chain-canary.yml, main function(qwen3-review) — GitHub API listing fetches 50 runs unfiltered, then filters client-side | The alert step fetches up to 50 workflow runs from the GitHub API and filters them in Python. The comment in `previous_conclus - [MINOR]
tests/ci/test_chain_canary_workflow.py(qwen3-review) — No test for the case where the GitHub API returns an empty workflow_runs list | The testtest_alert_without_slack_secrets_warns_and_never_passes_silentlyuses an opener that returns `{"workflow_runs - [MINOR]
.github/workflows/chain-canary.yml and scripts/lab_alarm.py(qwen3-review) — The alert step duplicates the workflow-verdict reader logic that lab_alarm.py already has | The alert step in the workflow fetches the previous run's conclusion via the GitHub API and makes its own ed - [MINOR]
tests/ci/test_chain_canary_workflow.py(qwen3-review) — No test for the Slack post failure path in the alert step | The testtest_alert_red_posts_one_message_naming_the_verdict_and_runtests the success path. There is no test for the case where the Slack - [MINOR]
tests/ci/test_lab_alarm_omn18867.py(qwen3-review) — The lab_alarm config test does not verify that the new keys are required, not optional | TheModelAlarmConfig.loadmethod usespayload["chain_canary_workflow"](direct key access), which will rais - [MINOR]
.github/workflows/chain-canary.yml(gpt-oss-review) — Alert step masks Slack posting failures | The new workflow step prints a warning and returns exit code 0 when the Slack API call fails. This causes the GitHub Actions job to succeed despite the operat
✅ Hostile Reviewer — REVIEWEDCritical findings: 0
Semantics (OMN-17492 — the model finds, thread resolution gates)
Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492) |
|
Lab: head=6870aee9354a host=Mac Studio (launching host) lane=h201 dev (read-only, nothing restarted or redeployed) command=evaluate_delegation_chain_canary(read_chain_canary_verdict) against OmniNode-ai/omnibase_infra chain-canary.yml@dev observed=OK, run 37293363538 concluded success 1.35h ago (3h window); positive control with a 30-minute window on the same history observed=ALARM. The alert step's own inline program, run against the real run listing with no Slack secrets, decided RED for a failed job after the green 09:56Z run and no alert for a green one, three times each. Demo-path delegation from the Studio (onex delegate --lane dev --locus deployed-lane, run e3e13ff5) answered from local-heavy-reasoning at 11:34Z. |
Summary
OMN-20594. Operator in session 2026-10-05: delegation must work in the lab and stay working on at least one lane. The h201 dev lane (compose project
omnibase-infra, the Tech Week demo lane) becomes the declared protected delegation lane, and its one real end-to-end delegation probe, the chain canary (C15,chain-canary.yml), now reaches the operator instead of failing silently. OMN-16773 deferred alerting; until this PR a red C15 was a failed run nobody watched.Wiring only, of surfaces that already exist: the Slack secrets the fleet canary and dev-lane-liveness alert through, the lab alarm (launchd, hourly, consent-gated Slack delivery), and the OMN-18866 workflow-verdict reader that staging delivery binds C15 with. No new script, workflow, node or allowlist.
Changes
.github/workflows/chain-canary.yml: a last step, scheduled runs only, posts to the operator channel when the verdict changes (red after a non-red, recovery after a red); a lane that stays red posts once. It reads the previous scheduled run from the unfiltered run listing, because the filtered listing omits the newest runs intermittently (OMN-20277; observed live while building this). Absent secrets, an unreadable history or a refused post are a::warning::on the run, never silent. The job gainsactions: readfor that read.scripts/lab_alarm.py+config/lab_alarm.json: a seventh condition,delegation_chain_canary, reads the C15 verdict through the workflow-verdict reader (scheduled runs, 180-minute window, the same bound staging delivery uses). Red, stale or absent raises once and recovers once through the existing edge-triggered delivery; an unreadable run surface is INDETERMINATE (and the existing stale-indeterminate condition watches it).deploy/lane-census/lane-manifest.yaml+scripts/generate_claude_lane_block.py: the dev entry and the generated lane boundary name the dev lane as the protected delegation lane and list its guards, so a lane reading the lane table sees it before it touches the lane.Not in this PR, and why:
scripts/deploy-agent/deploy_agent/idle_converge.py; its committed-schedule tests pin the converge gap). That is a deploy-design change and goes to the operator as a decision.Test plan
uv run pytest tests/ci/test_chain_canary_workflow.py tests/ci/test_lab_alarm_omn18867.py tests/unit/scripts/test_generate_claude_lane_block.py tests/ci/test_check_lab_probe_windows.py tests/ci/test_lane_projection_writer_coverage_omn17562.pyand the lane-boundary suites: 284 passed, 1 skipped;scripts/deploy-agent/tests/unit/test_idle_converge_omn19509.py: 24 passed (cadence untouched).mypy --strict scripts/lab_alarm.pyclean; ruff clean; the pre-commit hooks pass on commit.run 37293363538 concluded success 1.35h ago, 3h window), and the positive control with a 30-minute window returned ALARM on the same history. The alert step's own program, run against the real run listing with no Slack secrets, decided RED for a failed job after the green 09:56Z run and nothing for a green one, three times each, and printed the warning that the operator was not told.Runtime startup gate
Type safety checklist
except Exception— the lab alarm and the alert step catch named exception typesStrict-gate ordering
Related issues
OMN-20594 (parent OMN-16025). Related: OMN-16773, OMN-18866, OMN-20277, OMN-19509, OMN-20593 (backup delegation lane).
Evidence-Ticket: OMN-20594
Evidence-Source: OCC#12903