Skip to content

feat(OMN-20730): red CI emits ci-run-failed and the lifecycle orchestrator triages it - #3565

Merged
jonahgabriel merged 6 commits into
devfrom
jonah/omn-20730-red-ci-bus-triage
Oct 8, 2026
Merged

jonahgabriel merged 6 commits into
devfrom
jonah/omn-20730-red-ci-bus-triage

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

First slice (S0) of retiring the landing controller script: red CI is wired into the existing PR lifecycle nodes over the bus, per OMN-20730 and the plan in knowledge-base-internal beta/plans/2026-10-08-landing-controller-retirement-into-lifecycle-nodes.md (OmniNode-ai/knowledge-base-internal#1136). This change emits a bus event on red CI that the sweep orchestrator triages at once.

What changed

  • node_pr_state_emit_effect (EFFECT node): new handler on onex.evt.omnimarket.pr-state-observed.v1 emits onex.evt.omnimarket.ci-run-failed.v1 once per repo, PR, head, and failing-check set for open, non-draft RED PRs, including peers sharing a check. Explicit topic on both routes.
  • node_pr_lifecycle_triage_compute: pure classify_ci_red operation using landing controller rules (shared_cause, pr_own, dev_head, runner; CI Summary dropped unless alone; hostile-review-only as runner capacity; base reds; timed_out/cancelled) and a cluster floor of 3 armed PRs.
  • node_pr_lifecycle_orchestrator: handler on ci-run-failed drops replayed event_id, reads head/base check runs via read-only gh GETs, decides once per repo/head/check, claims one owner per cause, and emits onex.cmd.omnimarket.pr-lifecycle-orchestrator-start.v1 with pr_numbers, fix_only true, and dry_run true while ci_red_triage.act is false.
  • Decisions emit on onex.evt.omnimarket.ci-red-triage-decided.v1; node_pr_lifecycle_state_reducer projects them into pr_lifecycle_ledger_entries (initial_state ci_red:<class>, final_state fix_dispatched or skipped).
  • Shared models in omnimarket.models.ci_red_triage. No new node, script, migration, gate, or baseline change.

Safety

  • ci_red_triage.act is false, so all event-triggered starts are dry runs (fix_only, action_mode report_only, merge kill switch on).
  • The deployed runtime containers set no KAFKA_AUTO_OFFSET_RESET and the runtime bus config defaults to latest, so new consumer groups start at the head of pr-state-observed and do not replay its 7-day backlog.
  • The runtime stamps event_type from the resolved topic, matching the start handler’s alias.

Tests

On h201 at head dccea3ae5: focused, contract, topic-graph, registry, routing, and related suites: 2320 passed, 8 skipped. ruff check and format clean; mypy --strict clean over 4350 files; runtime import-check clean over 487 entry points; node metadata dependency check PASS; unimported-handler check OK; every pre-commit hook passed.

Lab proof

Lab: host h201 (.201), lane dev (compose project omnibase-infra), head=dccea3ae593ad6b55cc9e3d289cb3ec885ecd4ce, command uv run python lab_proof.py over the 800 newest onex.evt.omnimarket.pr-state-observed.v1 records (offsets 55432..56231) read with rpk from omnibase-infra-redpanda, live read-only gh GETs, reducer projection into a scratch postgres:16 with the dev pr_lifecycle_ledger_entries DDL: 28 ci-run-failed (19 armed), 28 decisions, 16 dry-run owner starts with 16 distinct run ids, 28 ledger rows; replay of all 800 gave 0 events, 0 decisions, 0 starts.

  • The omniclaude branch-claim-check / branch-claim-check cause had 3 decisions and exactly 1 owner start (members up to 5); the Tests Gate cause had 2 decisions and 1 owner start (members up to 4); one dev_head decision came from a live base-branch read.
  • Detector latency p50 0.024 ms; triage with two gh GETs p50 2170 ms. Watcher lag from ci_read_at to the bus: p50 193 s, max 695 s.

Not done here

  • Deployed-lane readback of AC1 and AC2 waits for merge and dev-lane redeploy; the two new topics do not exist on the dev bus yet (auto-create is off) and are provisioned by the runtime from the contracts.
  • AC1’s 2-minute and AC2’s 5-minute bounds cannot be met from the PR-state source while the watcher’s own lag is p50 193 s (max 695 s); plan slice S1.
  • The PR-state event carries no run id, workflow, or completed_at; plan slice S1.
  • Claims are in memory; the first two PRs of a cluster below the floor get their own PR-own start before the cause forms, as in the controller, and a restart empties the detector index; plan slice S2.

Evidence-Ticket: OMN-20730
Evidence-Source: OCC#13326

- node_pr_state_emit_effect subscribes to pr-state-observed.v1 and emits
  ci-run-failed.v1 for open, non-draft RED PRs, attaching peers sharing
  the same failing check.
- Adds pure classify_ci_red in triage_compute using landing controller
  rules and a cluster floor of 3 armed PRs.
- Orchestrator deduplicates replayed event_id, claims one owner per
  cause, and emits start.v1 with pr_numbers, fix_only, and dry_run.
- Publishes decisions on ci-red-triage-decided.v1, projected into
  pr_lifecycle_ledger_entries by the state reducer.
- Shared models reside in omnimarket.models.ci_red_triage; no new
  nodes, scripts, or migrations are introduced.
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

⚠️ Architectural Review — WARNINGS

Errors: 0
Warnings: 1

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

ARCH-DI-002: Direct Handler instantiation (warning)\n\n```\nsrc/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_ci_red_triage.py:167: self._classifier = HandlerClassifyCiRed()

src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1392: inv_handler = HandlerPrLifecycleInventory()
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1410: triage_handler = HandlerPrLifecycleTriage()
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1428: reducer_handler = HandlerPrLifecycleStateReducer()
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1450: merge_handler = HandlerPrLifecycleMerge(
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1497: fix_handler = HandlerPrLifecycleFix(
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1532: prune_handler = HandlerWorktreePrune(git_adapter=GitWorktreeAdapter())
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:1548: arm_gate_handler = HandlerPrArmGate()
src/omnimarket/nodes/node_pr_lifecycle_orchestrator/handlers/handler_pr_lifecycle_orchestrator.py:3172: verify_handler = HandlerVerify()\n```\n\nHandler classes should be wired only in workflow_runner.py or adapter modules. Direct instantiation elsewhere bypasses the contract system.\n\n

Static architectural lint — no model inference (OMN-14176).

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 12
Models succeeded: qwen3-review,gpt-oss-review


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)



class ProtocolCiRedFactsReader(Protocol):
def read(self, event: ModelCiRunFailedEvent) -> ModelCiRedFacts: ...
@jonahgabriel
jonahgabriel enabled auto-merge (squash) October 8, 2026 15:57
onexbot-occ-writer Bot and others added 4 commits October 8, 2026 16:46
repo-evidence / dod-verify failed at 05180ce because the PR cites
OMN-20730 but carried no contracts/OMN-20730.yaml. The contract binds
AC1 (ci-run-failed for open non-draft RED PRs), AC2 (classification,
dry-run triage starts and the reducer ledger projection), AC3 (one owner
start per shared cause) and AC4 (duplicates and replays emit nothing) to
the PR's new unit tests. node_dod_verify returns verified at the head and
fails at merge base bfc73dd with the PR's tests laid over it.
Release Identity Gate was red on PR 3565 because dev (e0f9332) already carries 0.4.305.
jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 8, 2026
…13326)

* fix(omn-20730): add contract and PASS receipt for OmniNode-ai/omnimarket PR 3565

Adds contracts/OMN-20730.yaml and the PASS receipt drift/dod_receipts/OMN-20730/dod-OmniNode-ai-omnimarket-pr-3565/command.yaml as the change-control companion for OmniNode-ai/omnimarket PR 3565 (ticket OMN-20730). This resolves the Deploy Gate failure: "Tickets with no contract file: OMN-20730".

Evidence:
- dod_evidence probe reads handler_classify_ci_red.py at PR head a7df686beb1dc9132bf9707aecd9f003808389df via gh api and greps 'def classify_ci_red'.
- Captured with check_receipt_hardening.py --capture-probe: green exit 0 at the head, red exit 1 at merge-base bfc73dd884ac2559d36cc8476b568f1366a1da3c.
- yamlfmt, check-yamlfmt-contamination and check-receipt-hardening all Passed.

Compute-only change, no live deploy surface.

* evidence(OMN-18921): mint the owed OCC self-bind for PR #13326

---------

Co-authored-by: onexbot-occ-writer[bot] <onexbot-occ-writer[bot]@users.noreply.github.com>
…est_passes check

The test_passes check for the item dod-omn20730-ac3-ac4-owner-replay in contracts/OMN-20730.yaml named only file::test node ids. This caused test_every_repo_contract_binds_every_criterion in tests/ci/test_repo_evidence_caller_workflow.py to fail in CI Tests Split 2/20 because it found no token ending in .py.

Added the plain test_ci_red_triage.py path as a token. The test now passes (5 passed) and the bound tests pass (33 passed).
@jonahgabriel
jonahgabriel merged commit 1ddd8b4 into dev Oct 8, 2026
143 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-20730-red-ci-bus-triage branch October 8, 2026 19:36
jonahgabriel added a commit that referenced this pull request Oct 13, 2026
…t hook (#3863)

Ruling 2026-10-08T14:27:26Z: red CI emits a typed event on the bus and the sweep orchestrator triages it on receipt; polling stays as the backstop.

Nodes already merged (#3565): node_pr_state_emit_effect, node_pr_lifecycle_orchestrator, node_pr_lifecycle_state_reducer.

Added tests/ci/test_red_ci_bus_chain_omn20730.py: reads five contract legs (emitter consumes pr-state-observed and routes it to a detector handler; emitter publishes ci-run-failed; orchestrator consumes ci-run-failed and routes it to an existing triage handler module; orchestrator publishes ci-red-triage-decided; reducer consumes ci-red-triage-decided).

8 planted cuts, each refused by chain_violations; 12 tests total pass.

Added a lint step in ci.yml named "red CI bus chain stays connected (OMN-20730)" and a check-only pre-commit hook with id red-ci-event-chain.

contracts/OMN-20730.yaml gains AC9 and a dod_evidence entry dod-omn20730-ac9-bus-chain-gate; no existing criterion line was edited.

Co-authored-by: codex <codex@omninode.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants