[jonahgabriel]

OMN-16778 — a projection nobody reads is not observability

Phase 1 of epic OMN-16776, the other half of OMN-16777 (merged to dev as 59885562). Execution gate OPEN (operator directive 2026-08-27).

OMN-16777 made a stalled consumer visible. Visibility is not notification. OMN-14440 is the precedent this repeats if the projection ships alone: a job that fired every 30 minutes for three months, failed every time, and wrote 348KB of errors nobody read. And on 2026-08-27 the delegation chain died, quarantined 16 commands in about a minute, and nothing fired — the outage was found by hand, hop by hop, by someone running an unrelated verification.

node_consumer_flow_stall_alert_effect is the half that speaks.

What it does

Consumes onex.evt.omnimarket.projection-consumer-flow-applied.v1 — the flow projection's own declared terminal event, no new transport — and publishes onex.cmd.omnimarket.slack-publish.v1 when one (consumer_group, topic) holds STALLED or STARVED for the contract-declared number of consecutive windows.

Outcome	Severity	Publishes
FAIL_CONFIRMED_STALL	FAIL	yes
PENDING_CONFIRMATION	WARN	only if deliver_warnings
WARN_MISSED_WINDOW	WARN	only if deliver_warnings
RECOVERING	WARN	no
NO_ALERT	NONE	no

Four decisions that carry the weight

Thresholds are contract data, and there is no code default to fall back on. ModelStallAlertPolicy declares no field defaults; load_stall_alert_policy raises StallAlertPolicyError on a missing or malformed alert_policy block rather than substituting a constant. AC2 is falsified by any threshold literal in a .py file, so test_no_threshold_literal_lives_in_python walks the AST of every Python file in the node and fails on a bare int/bool assigned to any threshold name — in an assignment, an annotated assignment, or a keyword argument. Grep would have missed a default buried in a signature.

UNKNOWN neither fires nor clears. It breaks an alerting run rather than extending it. Counting an unobserved window as a continuation would let a runtime that stopped heartbeating manufacture a confirmed alert out of nothing — the mirror image of the false-green OMN-16777 closed. It surfaces instead as its own WARN_MISSED_WINDOW outcome, so it is neither silently healthy nor silently stalled (AC5). The observation model keeps the counters int | None; nothing here coerces an unobserved window to 0.

Asymmetric hysteresis, asserted rather than assumed. confirm_windows: 2 before firing, clear_windows: 8 before calling it recovered, with a test that fails if the second is ever set at or below the first. This is the damping already proven for the .201 host reporter under OMN-16789 — before it had that damping it produced the alert flap the operator asked to have stopped. An alert channel that flaps is muted within a day, at which point this node is worth exactly as much as the silence it replaced.

No state file, and no second dedupe mechanism. The renotify period is enforced by bucketing the idempotency key on window_start // renotify_after_seconds, so node_slack_publish_effect's existing durable ledger collapses a repeat inside the 6h bucket. A standing stall re-posts at most once per period, and this node remembers nothing between invocations. The consecutive-window count and the recovery state are both derived from the window history itself, so there is no per-key state to go stale.

Delivery is proven, not asserted

Delivery rides the canonical node_slack_publish_effect (OMN-13733 consolidated every alert caller onto it) — Slack Web API chat.postMessage, bot token resolved through contract_secret_ref. SLACK_WEBHOOK_URL is deliberately unused: re-probed 2026-08-27 on both ~/.omnibase/.env and .201:/data/omninode/omnibase_infra/.env, it returns HTTP 404 / no_service, and it is being retired under OMN-15600.

The chain was driven end to end once, in-session, against the live channel — the real stall handler decided, build_slack_command rendered, and HandlerSlackPublishEffect resolved the secret and POSTed:

success=True ts=1787885222.308769 error_code=None

One message, clearly labelled :test_tube: OMN-16778 DELIVERY PROOF — synthetic, ignore., with the synthetic consumer group named as such in the payload. Per the ticket's own closeout note the assertion is on .ok, not on HTTP 200 — Slack returns 200 for channel_not_found and not_in_channel. The proof script was a scratch file outside the repo; nothing under scripts/** is added by this diff.

RED before GREEN

With the decision collapsed to the pre-change behaviour — fire on any single alerting window, and treat UNKNOWN as a continuation of the run:

FAILED test_a_single_stalled_window_does_not_fire
FAILED test_unknown_window_warns_and_does_not_fire_a_stall_alert
FAILED test_an_unknown_window_breaks_a_stall_run_instead_of_extending_it
FAILED test_raising_the_declared_confirm_window_suppresses_the_same_history
4 failed, 12 passed

Restored: 22 passed.

Two boundary decisions, both from this repo's own rules

CLAUDE.md: do not make one node import another node's private handler or model package.

• EnumConsumerFlowState is promoted to omnimarket.models — two nodes share it now, and duplicating a verdict enum is precisely the drift OMN-16777's own writer comment warns about ("the SQL writer and the in-memory writer cannot drift into disagreeing about what STALLED means"). Moved, not shimmed.
• The Slack command is a documented wire mirror (ModelStallAlertSlackCommand), not an import. These two nodes meet on a topic, not on a call edge, so the wire shape is the contract. It is extra="forbid", and test_slack_command_is_accepted_by_the_publish_node round-trips the payload through the real ModelSlackPublish so the two shapes cannot drift apart in silence — the OMN-14490/OMN-14506 failure mode where every undeclared field was dropped without a word.

Gates

• state-coverage-gate (strict): both declared outputs are now pinned by name in the golden-chain test. PASS: 2 WARN: 0 FAIL: 0.
• test_market_node_runtime_dogfood: node_dirs 388 → 389, entry_points 385 → 386, each with its reason recorded beside the number.
• The relation inventory is left unchanged on purpose. Its reader set is derived by scanning node sources for relation-name tokens, and a docstring mentioning consumer_flow_windows recorded this node as a reader of a table it never touches. The docstring was reworded (with the reason written down) rather than the generated evidence being edited — this handler is pure and takes its history as input.

dod_evidence

• tests/test_omn16778_stall_alert_decision.py — 11 cases: confirmed stall fires (AC1), starved fires, single window silent, idle silent (AC3), flowing silent, UNKNOWN warns (AC5), UNKNOWN breaks a run, recovering silent, payload completeness (AC4), one idempotency key across the renotify window, two legs judged independently.
• tests/test_omn16778_stall_alert_thresholds_from_contract.py — 5 cases incl. the AST literal sweep (AC2) and fail-closed on a missing alert_policy.
• tests/test_golden_chain_consumer_flow_stall_alert.py — 6 cases walking every hop, incl. the ModelSlackPublish round-trip and an assertion that the publisher still posts via chat.postMessage with SLACK_BOT_TOKEN and no SLACK_WEBHOOK_URL.
• Focused run: 22 passed. Wider run incl. the OMN-16777 suites and the dogfood/inventory gates: 37 passed.
• uv run mypy src/omnimarket/nodes/node_consumer_flow_stall_alert_effect src/omnimarket/models/enum_consumer_flow_state.py --strict → Success: no issues found in 12 source files.
• uv run ruff check src/ tests/ → All checks passed!
• pre-commit full pass at commit time: all gates Passed (state-coverage-gate, projection contract-access gate OMN-16690, golden-chain coverage, dependency health, contract topic graph, unimported-handler check).
• Governed pre-push impacted-test selector: 16511 passed, 116 skipped on the tree that this branch was cut from, then the two ratchets above were fixed and re-run.

Evidence-Ticket: OMN-16778
Evidence-Source: OCC#7371

[coderabbitai]

Warning

Review limit reached

Next included review available in 11 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 131 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4ead8b95-8a77-4ae3-90f3-048c37787a64

📥 Commits

Reviewing files that changed from the base of the PR and between 5988556 and 5b847a9.

📒 Files selected for processing (21)

• pyproject.toml
• src/omnimarket/models/enum_consumer_flow_state.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/__init__.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/contract.yaml
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/handlers/__init__.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/handlers/build_slack_command.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/handlers/handler_consumer_flow_stall_alert.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/metadata.yaml
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/__init__.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/enum_stall_alert_outcome.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/enum_stall_alert_severity.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/model_stall_alert_decision.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/model_stall_alert_policy.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/model_stall_alert_request.py
• src/omnimarket/nodes/node_consumer_flow_stall_alert_effect/models/model_stall_alert_slack_command.py
• src/omnimarket/nodes/node_projection_consumer_flow/models/__init__.py
• src/omnimarket/nodes/node_projection_consumer_flow/models/model_consumer_flow_row.py
• tests/test_golden_chain_consumer_flow_stall_alert.py
• tests/test_market_node_runtime_dogfood.py
• tests/test_omn16778_stall_alert_decision.py
• tests/test_omn16778_stall_alert_thresholds_from_contract.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule	Description
ARCH-TOPIC-001	No hardcoded Kafka topic strings in handler code
ARCH-DI-001	No event_bus=None bypass
ARCH-DI-002	No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003	No reinvented DI containers
ARCH-TOPIC-002	contract.yaml topics follow `onex.{cmd

---

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

[github-actions]

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 0
Models succeeded: qwen3-review,qwen3-review-b

---

Gate semantics

Verdict	Meaning	Blocks merge?
passed	>=2 models succeeded, no critical/error findings	No
blocked	CRITICAL or ERROR findings found	Yes
degraded	Fewer than 2 models succeeded (infra unavailable/timeout)	Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)