[jonahgabriel]

Ticket: OMN-20318 (parent OMN-17163)

Problem

Workflow dlq-depth-monitor.yml failed 348 of 351 runs in 7 days. Run 36870466680 log is empty via gh run view --log (its job page shows the probe step green and the "Name the offenders and gate the run" step red with exit code 1). Runs 36874480284 (2026-10-01T14:12Z) and 36878227805 (14:40Z) logs show the monitor itself works: declared broker omninode-pc.tail75df5e.ts.net:19092, SASL_PLAINTEXT/SCRAM-SHA-256, 91 topics observed. It is a real alert, not auth, broker or topic breakage.

Offenders run 36878227805 (arrivals in 1800s window / bound 0 / retained):

• onex.dlq.omnibase-infra.quarantine.v1 182 / 190969
• onex.dlq.omnibase-infra.events.v1 91 / 99864
• onex.dlq.omnimarket.projection-delegation-malformed.v1 79 / 7786

Run 36874480284:

• delegation-malformed 204/7707
• quarantine 194/190787
• events 97/99773
• projection-savings-malformed.v1 2/375

I read the live dev-lane broker read-only with rpk (SCRAM, consume only). quarantine.v1: partition 0 log-start 11656197, high-watermark 11847226. The tail records are all ModelDlqReplayRunResult with total_processed 0, quarantined 0, event_type omnibase-infra.quarantine - the summary of a drain with nothing to do, not a quarantined message.

Cause

HandlerDlqReplay.handle returned the run summary as ModelHandlerOutput.for_compute(result=...). handler_wiring._normalize_handler_result turns a BaseModel result into output_events. node_dlq_replay_effect/contract.yaml event_bus.publish_topics lists only onex.dlq.omnibase-infra.quarantine.v1 (the quarantine producer's own destination), which _select_dispatch_result_output_topic uses as the fallback output topic. Every DLQ record triggers a whole drain, so every DLQ record wrote one summary to the terminal quarantine sink.

Fix

handle() logs the summary at INFO ("DLQ replay run finished: topics=... total=... completed=... quarantined=... failed=... pending=... dry_run=... halted_partitions=...") and returns ModelHandlerOutput.for_effect with no events and no result. Contract unchanged; the quarantine producer still publishes real quarantined messages directly.

Tests

• New tests/unit/nodes/node_dlq_replay_effect/test_omn20318_run_summary_not_published_to_quarantine.py (RED before the fix: output_events held a ModelDlqReplayRunResult; green after).
• Updated test_handler_dlq_replay.py::test_handle_envelope_logs_the_run_summary_and_returns_no_result and tests/integration/dlq/test_dlq_replay_dispatch_bound_omn17137.py::test_dispatch_returns_when_the_dependency_teardown_never_completes to read the summary from the log.
• 120 passed, 17 skipped (PostgreSQL absent) over tests/unit/nodes/node_dlq_replay_effect, tests/integration/dlq and test_omn18084_handler_error_leg_dlq_loop.py.
• mypy on node_dlq_replay_effect clean.

Lab Readback

Lab dev-202 lane, compose project omnibase-infra-dev-202, host h202, redpanda container omnibase-infra-dev-202-redpanda; not stability-test or judge. Ran the real handler through the real _make_dispatch_callback, then the real contract-derived result applier publishing over SCRAM to the lab broker, 3 dispatches, measuring onex.dlq.omnibase-infra.quarantine.v1 high-watermark.

• Control (lab image's own /app/src, pre-fix): output_events [ModelDlqReplayRunResult] x3, high-watermark 15 -> 18, delta 3, all 3 published to quarantine.v1.
• Fixed (this worktree's src mounted read-only over /app/src in the same image): output_events [] x3, high-watermark 18 -> 18, delta 0, nothing published.

Surfaces exercised: dispatch result coercion, contract result applier, broker publish. NOT exercised: a full runtime boot, the kernel, or the .201 broker itself; the monitor workflow was not re-run.

Remaining After This PR

Named, not fixed here:

(a) events.v1 91 per window: 392 of the 400 tail records sampled (186 skill-completed, 185 skill-started, 21 delegate-skill-failed) are failure_class gateway_egress_denied_record for tenant 79afa726-3852-464f-b7a4-d4b8b9c75ee7, denied topic prefix tenant-beta-gateway-canary-79afa7263852, direction outbound, written by the gateway forwarder (service_gateway_delivery GatewayEgressDeniedError quarantine path, OMN-17201) because the destination broker ACL denies that principal.

(b) projection-delegation-malformed.v1: omnimarket projection writer refuses quality-gate-result events with no tenant on the envelope (OMN-18139).

(c) projection-savings-malformed.v1: tenant a630ba90-7baa-4aec-b990-c34ec71ea2fb absent from tenant_registry_mirror (OMN-16831 / OMN-17446).

The workflow will stay red until (a)-(c) are resolved; the threshold, schedule and gating are untouched.

Evidence-Ticket: OMN-20318
Evidence-Source: OCC#12183

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 9
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 9 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

• [MAJOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:296-308 (qwen3-review) — Log message format string has 8 placeholders but 9 arguments | The logger.info call in handler_dlq_replay.py provides 9 positional arguments (topics, total, completed, quarantined, failed, pending, dr
• [MAJOR] tests/unit/nodes/node_dlq_replay_effect/test_omn20318_run_summary_not_published_to_quarantine.py (qwen3-review) — No test verifies that a non-empty drain does not publish to quarantine | The new test file test_omn20318_run_summary_not_published_to_quarantine.py only tests the empty-drain case (no messages). The b
• [MAJOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py (gpt-oss-review) — Handler output contract altered without downstream adaptation | The HandlerDlqReplay.handle method now returns ModelHandlerOutput[None] and omits the result field, while the runtime's dispatch n
• [MAJOR] tests/unit/nodes/node_dlq_replay_effect/test_omn20318_run_summary_not_published_to_quarantine.py (gpt-oss-review) — Missing pytest import in newly added unit test file | The file test_omn20318_run_summary_not_published_to_quarantine.py uses pytest.mark.unit and pytest fixtures but does not import the `pytes
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:299 (qwen3-review) — Log injection via topics_drained field | The log statement interpolates ','.join(run_result.topics_drained) directly into the log message. If topics_drained contains user-controlled or attacker-influe
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:296-308 (qwen3-review) — Log statement executes on every DLQ record even when drain is empty | The log.info call now executes on every invocation of handle(), which is triggered by every DLQ record. The ticket notes 182-194 a
• [MINOR] tests/integration/dlq/test_dlq_replay_dispatch_bound_omn17137.py:212-215 (qwen3-review) — Integration test relies on fragile log message substring matching | The integration test asserts that a log record contains 'DLQ replay run finished' and 'total=2'. This is brittle: any change to the
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:249-250 (qwen3-review) — Return type change from ModelHandlerOutput[ModelDlqReplayRunResult] to ModelHandlerOutput[None] is a breaking API change | The handle method's return type annotation changes from ModelHandlerOutput[Mo
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py (gpt-oss-review) — Missing required result argument in ModelHandlerOutput.for_effect | ModelHandlerOutput.for_effect is invoked without the result keyword, but the method signature in the codebase requires a `re

[github-actions]

✅ Hostile Reviewer — REVIEWED

Critical findings: 0
Major findings: 4
Total findings: 10
Models succeeded: qwen3-review,gpt-oss-review
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).

---

Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface	Meaning	Blocks merge?
Review threads	Per-finding, posted by the reviewer	No (informational)
Hostile Review Thread Gate	Deterministic: unresolved hostile-reviewer threads exist	Fails until resolved (not yet a required context)
degraded verdict	Fewer than 2 models succeeded (infra)	Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492)

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 9
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 9 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

• [MAJOR] tests/integration/dlq/test_dlq_replay_dispatch_bound_omn17137.py:212-215, tests/unit/nodes/node_dlq_replay_effect/test_handler_dlq_replay.py:369-372 (qwen3-review) — Log message uses r.message which is None for lazy FormattedRecord | The tests assert against r.message (the raw message string) rather than r.getMessage() (the formatted string). In pytest's LogCa
• [MAJOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:305-309 (qwen3-review) — for_effect() with no result may violate ModelHandlerOutput contract | The change switches from ModelHandlerOutput.for_compute(..., result=run_result) to ModelHandlerOutput.for_effect(...) with no
• [MAJOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py (gpt-oss-review) — Handler return type no longer matches contract expectations | HandlerDlqReplay.handle now returns ModelHandlerOutput[None] via ModelHandlerOutput.for_effect, removing the ModelDlqReplayRunResult paylo
• [MAJOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py (gpt-oss-review) — Potential runtime error due to missing result field in ModelHandlerOutput | ModelHandlerOutput.for_effect may expect a result argument or set a default that downstream code relies on. Omitting the r
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:301-308 (qwen3-review) — Log injection via topics_drained field | The log statement interpolates ",".join(run_result.topics_drained) directly into the log message. If any topic name in topics_drained contains newline char
• [MINOR] tests/unit/nodes/node_dlq_replay_effect/test_omn20318_run_summary_not_published_to_quarantine.py (qwen3-review) — No test verifies that a non-empty drain does not publish a summary event | The new test test_a_drain_with_nothing_to_do_hands_the_runtime_no_output_event only exercises the empty-drain path (zero me
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py:301-308 (qwen3-review) — Log line emitted on every trigger even when backlog is zero | The contract states that a run with a zero backlog probe does not start the consumer. However, the log statement in handle() is emitted
• [MINOR] src/omnibase_infra/nodes/node_dlq_replay_effect/handlers/handler_dlq_replay.py (gpt-oss-review) — Info‑level logging of internal DLQ replay metrics | The handler logs detailed run statistics (topics, counts, halted partitions) at INFO level. If log aggregation is accessible to users without approp
• [MINOR] tests/integration/dlq/test_dlq_replay_dispatch_bound_omn17137.py (gpt-oss-review) — Integration test now depends on log capture without asserting handler output | test_dispatch_returns_when_the_dependency_teardown_never_completes removes the assertion on output.result and only chec