feat(OMN-16773): schedule the delegation chain canary so a dead chain is caught in hours, not weeks

[jonahgabriel]

OMN-16773 — the delegation chain gets a canary

The 13-class delegation matrix existed only as a recorded manual recipe. Nothing ran it.

On 2026-08-23 an omnimarket contract change gave node_delegation_routing_reducer a db_io block; the runtime's _prepare_handler_wiring selects the projection dispatch arm on db_io.db_tables alone, so a typed def-B handler began receiving a raw dict and every delegation went to the platform quarantine sink. The chain was dead for four days and was discovered on 2026-08-27 only because a human happened to fire the recipe by hand while closing an unrelated ticket (OMN-16767). The quarantine sink was at ~8,878,924 records at that moment, watched by nothing.

delegation-seam-gate.yml (OMN-14771) was green throughout, and correctly so. It drives the delegation seam end-to-end over InMemoryTransport, where the wiring is constructed by the test. No in-memory seam test can observe a deployed wiring arm choosing the wrong dispatch path. That gate proves the seam is right in the tree; this one proves the chain is alive in the lane. Neither substitutes for the other.

What lands

node_chain_canary_effect (EFFECT_GENERIC, omnibase_infra) — fires ONE delegation through the recorded recipe (POST {probe_url}/skill, command_name=node_delegate_skill_orchestrator, payload shape from omnidash/server/routes.ts:216-234, task class from omnidash/shared/contracts/delegation-task-types.json), asserts a terminal inside a declared budget, then scans the quarantine sink tail for that run's own correlation id.

chain-canary.yml — every 2h on omnibase-deploy, receipt into the job summary, non-zero exit on any non-GREEN verdict.

tests/ci/test_chain_canary_workflow.py — the PR-time guard. The canary has no pull_request trigger (it publishes a real command onto the lane), so nothing else would notice its wiring rotting.

Design decisions worth reviewing

• The probe correlation id is minted in the handler and is not a request field. A canary whose id can be pinned by its caller is a canary whose runs can be confused with each other.
• Verdicts are ranked, not collapsed into "red". QUARANTINED outranks TERMINAL_MISSING because in the OMN-16767 incident both are true and only the first names the defect. A canary reporting "timed out" there would have sent someone to look at latency instead of the dispatch seam.
• SKIPPED_NOT_CONFIGURED is never reported as CLEAN, and a configured-but-unrunnable quarantine check fails the run. The entire reason this ticket exists is that a check nobody ran looked exactly like a check that passed.
• ok=true with no terminal is RED (OMN-16027: publish_envelope() is fail-open, so a cheerful accept proves nothing).
• probe_url is required with no default (Rule 8). A canary that guesses its own target can report green about a lane nobody meant to probe.
• Zero retries. A retry would let an intermittently dead chain report green. The 2h cadence is the retry.

Live evidence — the canary works, and its first result is RED

Run against the .201 dev lane 2026-08-27 (lane serving 0.38.11, the pre-fix rev; OMN-16767's fix #2937 is still OPEN/BLOCKED):

{"verdict": "terminal_missing", "success": false,
 "detail": "no terminal event inside the 45000 ms budget after 46409 ms: ingress returned ok=false (dispatch_timeout: Local runtime ingress timed out after 45000 ms)",
 "probe_correlation_id": "da4834a2-7b26-4134-a482-c4f7cdc0388f",
 "ingress_error_code": "dispatch_timeout", "terminal_event": "",
 "quarantine_status": "clean", "quarantine_records_scanned": 300}

That RED is the deliverable. A red canary on a known-dead chain is the product working.

Correlation-exact broker trace across four probes, posted in full to OMN-16767 — it also shows the failure has moved one hop earlier since the 15:32Z reproduction: all four probes reached onex.cmd.omnimarket.delegate-skill.v1 (offsets 20-23), none reached delegation-routing-request.v1, zero terminals, and the quarantine HWM did not advance by a single record (8,878,933 before and after). The orchestrator is no longer consuming its command topic at all. The canary reported terminal_missing + clean rather than falsely claiming quarantined — the verdict discrimination is doing real work.

Two defects in the quarantine leg were found by running it live rather than by reasoning about it, and both are documented at their fix sites:

1. partitions_for_topic() returns None for a topic the consumer never subscribed to, even when topics() lists it — the scan fail-closed on a topic that plainly existed at 8.9M records.
2. aiokafka surfaces CancelledError out of consumer.stop(); catching only Exception let it escape and destroyed an otherwise complete scan result.

Scope

Dev lane only (omnibase-infra, the pre-authorized fully-mutable test platform). No claim is made about stability-test, judge, or prod, and the workflow must not be pointed at them without its own ticket — the probe publishes a real delegation command, which is a lane mutation.

Two allowlist entries, both with precedent

• scripts/check-env-reads.sh — third instance of the documented "RuntimeLocal single-shot compute path has no config-prefetch/overlay seam for a node's own kill switch" case, alongside node_evidence_autoclose_sweep_effect (OMN-16106) and node_sync_revert_watchdog_effect (OMN-16536). The comment flags that three instances of one rationale means the matcher should learn the shape rather than take a fourth path.
• scripts/ci/infra-node-allowlist.txt — lane and runner topology are infra-exclusive, not a portable omnimarket business-domain capability.

Deliberately not in this PR

• The OMN-16690 AC5 hook-chain probe. Its hook-event-capture gateway entry is currently re-fenced, and OMN-16690's own body states it is un-fenced "only when this ticket is deployed AND the canary shows a row". OMN-16690 is still In Progress. The fence blocks the probe, so it is not built here.
• The other 12 delegation classes. One class proves the chain is alive; 13 is a matrix run.
• Alerting on red — the failing workflow run is the phase-1 signal.
• Aggregate quarantine-sink depth. That is OMN-16769 (In Progress in a parallel lane); this canary's quarantine leg is correlation-scoped and does not overlap it.

Pre-push evidence note (stated, not hidden)

The governed selector escalated to a full suite on reason=test_infrastructure (this PR adds test files) and refused to run it locally at 1.25x-core load. Its PREFERRED remediation — let GitHub-hosted CI run the full suite on this sha, then re-push — is structurally unavailable for a new branch: no ci.yml run can exist for a sha that has never been pushed. The .201 gate-runner has capacity but its container has no ssh client, so the push cannot originate there. A single-use, sha-bound, receipted override grant was minted and consumed (prepush_override_consumed, context degraded-capacity: heavy fail-closed full-suite escalation, nonce 6ac205ce211f, head 8c150e622cff) so the escalated suite ran here rather than being skipped. It ran green: 24,002 passed / 40 skipped on the escalated pass, plus 8,014 passed / 52 skipped / 1 xfailed on the impacted-subset pass (tests/gates/ tests/nodes/ tests/unit/). No --no-verify, no skip token, no bypass flag. The over-escalation class is already filed as OMN-16745; the new-branch bootstrap gap is a residual worth its own ticket.

Ticket: OMN-16773

Evidence-Ticket: OMN-16773
Evidence-Source: OCC#7328

[coderabbitai]

Warning

Review limit reached

Next included review available in 21 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 131 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1a88fabc-5525-4abe-b72b-524eab3ecd66

📥 Commits

Reviewing files that changed from the base of the PR and between c015bb0 and 73b53a7.

📒 Files selected for processing (20)

• .github/workflows/chain-canary.yml
• docker/runners/runner-image.lock.json
• pyproject.toml
• scripts/check-env-reads.sh
• scripts/ci/infra-node-allowlist.txt
• src/omnibase_infra/cli/skill_mapping.yaml
• src/omnibase_infra/nodes/node_chain_canary_effect/__init__.py
• src/omnibase_infra/nodes/node_chain_canary_effect/contract.yaml
• src/omnibase_infra/nodes/node_chain_canary_effect/handlers/__init__.py
• src/omnibase_infra/nodes/node_chain_canary_effect/handlers/handler_chain_canary.py
• src/omnibase_infra/nodes/node_chain_canary_effect/models/__init__.py
• src/omnibase_infra/nodes/node_chain_canary_effect/models/enum_chain_canary_verdict.py
• src/omnibase_infra/nodes/node_chain_canary_effect/models/enum_quarantine_check_status.py
• src/omnibase_infra/nodes/node_chain_canary_effect/models/model_chain_canary_request.py
• src/omnibase_infra/nodes/node_chain_canary_effect/models/model_chain_canary_result.py
• src/omnibase_infra/nodes/node_chain_canary_effect/node.py
• tests/ci/test_chain_canary_workflow.py
• tests/fixtures/dispatch_parity/baseline-selection-v2.json
• tests/unit/nodes/node_chain_canary_effect/__init__.py
• tests/unit/nodes/node_chain_canary_effect/test_handler_chain_canary.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8000 ] unreachable — preflight short-circuit (no models available)

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review (OMN-8468/OMN-8524)

[jonahgabriel]

Superseded during manual merge sweep by #2955. The code head is unchanged at 73b53a7; replacement branch is ticket-named (jonah/omn-16773-chain-canary-dev-lane) because Receipt Gate identity binding fails on the original branch name jonah/omn-chain-canary-dev-lane.

[jonahgabriel]

Closed as superseded by #2955 with the same repaired head and ticket-named branch for Receipt Gate identity binding.