Skip to content

feat(OMN-20594): the protected delegation lane's chain canary reaches the operator on a red, and the lab alarm backstops it - #4577

Merged
jonahgabriel merged 4 commits into
devfrom
jonah/omn-20594-protected-delegation-lane
Oct 5, 2026
Merged

jonahgabriel merged 4 commits into
devfrom
jonah/omn-20594-protected-delegation-lane

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

OMN-20594. Operator in session 2026-10-05: delegation must work in the lab and stay working on at least one lane. The h201 dev lane (compose project omnibase-infra, the Tech Week demo lane) becomes the declared protected delegation lane, and its one real end-to-end delegation probe, the chain canary (C15, chain-canary.yml), now reaches the operator instead of failing silently. OMN-16773 deferred alerting; until this PR a red C15 was a failed run nobody watched.

Wiring only, of surfaces that already exist: the Slack secrets the fleet canary and dev-lane-liveness alert through, the lab alarm (launchd, hourly, consent-gated Slack delivery), and the OMN-18866 workflow-verdict reader that staging delivery binds C15 with. No new script, workflow, node or allowlist.

Changes

  • .github/workflows/chain-canary.yml: a last step, scheduled runs only, posts to the operator channel when the verdict changes (red after a non-red, recovery after a red); a lane that stays red posts once. It reads the previous scheduled run from the unfiltered run listing, because the filtered listing omits the newest runs intermittently (OMN-20277; observed live while building this). Absent secrets, an unreadable history or a refused post are a ::warning:: on the run, never silent. The job gains actions: read for that read.
  • scripts/lab_alarm.py + config/lab_alarm.json: a seventh condition, delegation_chain_canary, reads the C15 verdict through the workflow-verdict reader (scheduled runs, 180-minute window, the same bound staging delivery uses). Red, stale or absent raises once and recovers once through the existing edge-triggered delivery; an unreadable run surface is INDETERMINATE (and the existing stale-indeterminate condition watches it).
  • deploy/lane-census/lane-manifest.yaml + scripts/generate_claude_lane_block.py: the dev entry and the generated lane boundary name the dev lane as the protected delegation lane and list its guards, so a lane reading the lane table sees it before it touches the lane.

Not in this PR, and why:

  • A 15-minute C15 cadence. Every minute would then fall inside a C15 exclusion window, which closes all idle-converge windows of this lane (OMN-19509, scripts/deploy-agent/deploy_agent/idle_converge.py; its committed-schedule tests pin the converge gap). That is a deploy-design change and goes to the operator as a decision.
  • Rollback when a deploy breaks delegation. None exists for delegation (the deploy agent's post-deploy verification checks health, not delegation; the signed rollback declaration is manual). Proposed under OMN-20594 for after Tech Week, since it needs a deploy-agent update on h201.

Test plan

  • RED first: the new tests failed before the change (14 alert tests; the lab alarm suite failed at import; 2 lane tests).
  • uv run pytest tests/ci/test_chain_canary_workflow.py tests/ci/test_lab_alarm_omn18867.py tests/unit/scripts/test_generate_claude_lane_block.py tests/ci/test_check_lab_probe_windows.py tests/ci/test_lane_projection_writer_coverage_omn17562.py and the lane-boundary suites: 284 passed, 1 skipped; scripts/deploy-agent/tests/unit/test_idle_converge_omn19509.py: 24 passed (cadence untouched). mypy --strict scripts/lab_alarm.py clean; ruff clean; the pre-commit hooks pass on commit.
  • Lab readback (read-only, nothing on h201 restarted or redeployed): the new condition read live against the real C15 history returned OK (run 37293363538 concluded success 1.35h ago, 3h window), and the positive control with a 30-minute window returned ALARM on the same history. The alert step's own program, run against the real run listing with no Slack secrets, decided RED for a failed job after the green 09:56Z run and nothing for a green one, three times each, and printed the warning that the operator was not told.
  • Not runtime-affecting: no runtime image, compose or contract path changes; the dev lane and its deploy agent are not touched.

Runtime startup gate

  • N/A — this PR does not touch any of the above

Type safety checklist

  • No new bare except Exception — the lab alarm and the alert step catch named exception types

Strict-gate ordering

  • No — not a gate PR (an alert and an alarm condition; neither blocks a merge or a deploy).

Related issues

OMN-20594 (parent OMN-16025). Related: OMN-16773, OMN-18866, OMN-20277, OMN-19509, OMN-20593 (backup delegation lane).

Evidence-Ticket: OMN-20594
Evidence-Source: OCC#12903

… the operator on a red, and the lab alarm backstops it

The h201 dev lane is the protected delegation lane (operator in session
2026-10-05). chain-canary.yml (C15) now posts to the operator Slack channel
when a scheduled run's verdict changes (red after non-red, recovery after
red), through the SLACK_BOT_TOKEN and SLACK_CHANNEL_ID secrets the fleet
canary already alerts through. scripts/lab_alarm.py gains the
delegation_chain_canary condition, which reads the C15 verdict through the
OMN-18866 workflow-verdict reader (scheduled runs, 3h window) and raises once
when it is red or silent. The lane manifest and the generated lane boundary
name the dev lane as the protected delegation lane and list its guards.

Onex-Lane: deleg-guard-9143
Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…tered listing

The filtered run listing intermittently omits the newest runs (OMN-20277);
read live, it named a stale failure as the previous verdict on one of two
calls. Read the unfiltered listing and filter and order it here.

Onex-Lane: deleg-guard-9143
Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
…work and parse errors it expects

Onex-Lane: deleg-guard-9143
Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
Comment thread tests/ci/test_chain_canary_workflow.py Fixed
…string

Onex-Lane: deleg-guard-9143
Onex-Session: b7c9bc237e84415cbab7ccb1cb67b8ce
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 5, 2026
…12903)

* evidence(OMN-20594): author OCC companion for OmniNode-ai/omnibase_infra#4577

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 09cb2f2434ea3f718110d0cac7b93b058527f872.

* evidence(OMN-20594): self-bind OCC#12903 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 13
Nit-level findings suppressed: 0

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 13 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [MAJOR] .github/workflows/chain-canary.yml, alert step env block (qwen3-review) — Slack token and channel ID exposed in process environment | The alert step passes SLACK_BOT_TOKEN and SLACK_CHANNEL_ID into the step's env block. In GitHub Actions, step-level env values are visible i
  • [MAJOR] .github/workflows/chain-canary.yml, previous_conclusion function (qwen3-review) — previous_conclusion sorts by created_at string, not ISO-8601 timestamp | The function previous_conclusion sorts admitted runs by str(run.get("created_at", "")) in reverse. The GitHub API returns `
  • [MAJOR] tests/ci/test_chain_canary_workflow.py, test_alert_posts_only_on_a_change_of_verdict (qwen3-review) — decide() treats 'cancelled' as no-op, but a cancelled run after a red should not suppress a subsequent red alert | The decide function returns None when job_status == "cancelled", meaning no ale
  • [MAJOR] .github/workflows/chain-canary.yml, read_receipt function (qwen3-review) — read_receipt uses open() without closing the file handle | The read_receipt function opens a file with open(path, encoding="utf-8").read() without using a context manager. In CPython, the file is
  • [MAJOR] scripts/lab_alarm.py (gpt-oss-review) — ModelAlarmConfig.load now unconditionally accesses new keys | The load() method reads payload["chain_canary_workflow"], payload["chain_canary_branch"] and payload["chain_canary_max_age_minutes"] witho
  • [MAJOR] scripts/lab_alarm.py (gpt-oss-review) — Missing import for subprocess in exception handling | evaluate_delegation_chain_canary catches subprocess.SubprocessError, but the file does not import the subprocess module. If the reader raises a Su
  • [MAJOR] scripts/lab_alarm.py (gpt-oss-review) — Potential mismatch in condition count expectations | The addition of DELEGATION_CHAIN_CANARY changes the total number of evaluated conditions from six to seven. Other components (e.g., serialization,
  • [MINOR] .github/workflows/chain-canary.yml, main function (qwen3-review) — GitHub API listing fetches 50 runs unfiltered, then filters client-side | The alert step fetches up to 50 workflow runs from the GitHub API and filters them in Python. The comment in `previous_conclus
  • [MINOR] tests/ci/test_chain_canary_workflow.py (qwen3-review) — No test for the case where the GitHub API returns an empty workflow_runs list | The test test_alert_without_slack_secrets_warns_and_never_passes_silently uses an opener that returns `{"workflow_runs
  • [MINOR] .github/workflows/chain-canary.yml and scripts/lab_alarm.py (qwen3-review) — The alert step duplicates the workflow-verdict reader logic that lab_alarm.py already has | The alert step in the workflow fetches the previous run's conclusion via the GitHub API and makes its own ed
  • [MINOR] tests/ci/test_chain_canary_workflow.py (qwen3-review) — No test for the Slack post failure path in the alert step | The test test_alert_red_posts_one_message_naming_the_verdict_and_run tests the success path. There is no test for the case where the Slack
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (qwen3-review) — The lab_alarm config test does not verify that the new keys are required, not optional | The ModelAlarmConfig.load method uses payload["chain_canary_workflow"] (direct key access), which will rais
  • [MINOR] .github/workflows/chain-canary.yml (gpt-oss-review) — Alert step masks Slack posting failures | The new workflow step prints a warning and returns exit code 0 when the Slack API call fails. This causes the GitHub Actions job to succeed despite the operat

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — REVIEWED

Critical findings: 0
Major findings: 7
Total findings: 13
Models succeeded: qwen3-review,gpt-oss-review
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492)

@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Lab: head=6870aee9354a host=Mac Studio (launching host) lane=h201 dev (read-only, nothing restarted or redeployed) command=evaluate_delegation_chain_canary(read_chain_canary_verdict) against OmniNode-ai/omnibase_infra chain-canary.yml@dev observed=OK, run 37293363538 concluded success 1.35h ago (3h window); positive control with a 30-minute window on the same history observed=ALARM. The alert step's own inline program, run against the real run listing with no Slack secrets, decided RED for a failed job after the green 09:56Z run and no alert for a green one, three times each. Demo-path delegation from the Studio (onex delegate --lane dev --locus deployed-lane, run e3e13ff5) answered from local-heavy-reasoning at 11:34Z.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants