[jonahgabriel]

Summary

Ticket: OMN-19520 (parent epic OMN-16776, platform observability). Found while working OMN-19513.

The work_events projection on the .201 dev lane failed every session-started event for four days, and nothing named it. omninode_internal.consumer_flow_windows counted every failure (handler_errors, messages_dlq). The existing alert effect, node_consumer_flow_stall_alert_effect, reads exactly that table, but it fires only on a run of confirm_windows (2) consecutive STALLED windows that carry failure evidence. Session-started is sparse (one event every few minutes, 30 s windows), so each failure sat alone between IDLE windows. The run rule graded it PENDING_CONFIRMATION (not delivered) or NO_ALERT. It posted only when two failures happened to land in adjacent windows, and those posts were titled STALLED, which read as a flow-rate blip.

The runtime /health dimension projection_dlq_saturation also missed it: it sums per projection across all four hook topics and needs a 100 percent DLQ ratio, so three healthy topics diluted it to zero (curl :8085/health on the lab read dlq_saturated_projection_count 0, 2026-09-25T01:40Z). This PR does not change that dimension. The alert effect is the surface that reads per-(consumer group, topic) windows and already delivers.

Change

It reuses the existing effect, its window read and the node_slack_publish_effect delivery. There is no new daemon.

• alert_policy.handler_error_windows: 1 in the contract. It is contract data with no Python default, and the AST scan now covers it.
• New outcome FAIL_HANDLER_ERRORS. It fires when at least that many windows of the trailing read (history_windows, 16) have handler_errors > 0, whether or not they are consecutive or the newest. None is an unobserved counter and never counts.
• A run that already reached confirm_windows keeps FAIL_CONFIRMED_STALL, and its recovery still grades RECOVERING, so the same incident is not announced twice.
• The idempotency key is <bucket>|<group>|<topic>|HANDLER_ERRORS, bucketed on the newest failing window. Each of the 16 re-evaluations that still see one failure hands the publish ledger the same key, so it becomes one post per consumer, topic and 6 h bucket.
• The Slack text opens HANDLER ERRORS, followed by the group, the topic, N of the last M windows failed messages in the handler, the totals (handler_errors, dlq, in, out), the failing-window span and the correlation id.
• The payload gains handler_error_windows and observed_windows. consecutive_windows is relaxed to ge=0, because a handler-error payload may have no trailing run.
• Four existing tests pinned the anti-flap behaviour using a failing STALLED window as the example. They now use STARVED, which carries no handler error, so the anti-flap rule stays pinned for the state it is meant for. A single failing window now fires, and that is the point of the change.

Proof

RED first. Twelve new tests, plus a widened windows-source assertion. With origin/dev node source checked out over the committed tree (rule 17 method), the 11 tests selected by -k handler_error ran 9 failed and 2 passed. The 2 that pass are the negative controls (zero and unobserved counters), which must pass on both sides. On HEAD, 75 of 75 pass across the stall-alert decision, thresholds, windows-source and self-assembly suites, the golden chain, and the OMN-16875 producer/consumer pin.

Positive and negative control on lab data. I exported the consumer_flow_windows rows of local.omnimarket.projection_work_events.consume.1.0.0 from the .201 dev lane Postgres for 2026-09-21 to 2026-09-24, 9,793 windows per topic. The replay runs the shipped decide_stall_alert on the trailing 16 windows at every window, which is what the node sees on each heartbeat.

source	topic	failing windows	failing windows covered by a post	posts after dedupe
origin/dev	session-started	197	34	6 (all titled STALLED)
this PR	session-started	197	197	22
origin/dev	tool-executed (healthy)	0	0	0
this PR	tool-executed (healthy)	0	0	0

Volume over the last 24 h, lab data. The same replay covered every key with a handler error in the last 24 h (13 keys) plus 8 healthy FLOWING keys. Each erroring key had every failing window covered, 26 posts in total. The healthy keys produced 0. The notifications channel took 1,030 stall posts over the retained log, so this adds about 2.5 percent. It also names four more projections that fail every event and never reached a post of their own kind: projection_baselines, projection_baselines_roi, projection_baselines_quality and projection_dod_verdict.

Local gates. ruff format and check are clean, mypy --strict over the node is clean, and the full pre-commit suite passed at commit.

Lab runtime (rule 24). This is not yet exercised on the lab runtime. 201-dev-runtime is under a ledger HOLD by another lane until 02:31Z, and deploy-agent-dev is down (ledger MSG from runtime-train-83 at 01:36:24Z). The lab proof above is the shipped decision function over the lab's own recorded windows. OMN-19520 AC5 is the post-deploy readback: a FAIL_HANDLER_ERRORS decision on the evaluation topic, and its delivery on slack-published. Rule 24(d): this touches no tenant, RLS or reader-authorization path, and onex-api is not involved.

Sequencing

Like omnimarket#2878, this branch carries the post-release version bump commit from omnimarket#2874 (0.4.216 to 0.4.217, cherry-picked unchanged). The release-identity pre-commit hook refuses any src/ change on dev without it. Land #2874 first. Updating this branch from dev then drops the duplicate.

Evidence-Ticket: OMN-19520
Evidence-Source: OCC#11191

[github-actions]

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule	Description
ARCH-TOPIC-001	No hardcoded Kafka topic strings in handler code
ARCH-DI-001	No event_bus=None bypass
ARCH-DI-002	No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003	No reinvented DI containers
ARCH-TOPIC-002	contract.yaml topics follow `onex.{cmd

---

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

[github-actions]

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 13
Models succeeded: qwen3-review,qwen3-review-b

---

Gate semantics

Verdict	Meaning	Blocks merge?
passed	>=2 models succeeded, no critical/error findings	No
blocked	CRITICAL or ERROR findings found	Yes
degraded	Fewer than 2 models succeeded (infra unavailable/timeout)	Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)