Skip to content

feat(OMN-20415): landing decision groups red PRs by shared cause and dispatches one bounded cause worker - #3342

Merged
jonahgabriel merged 8 commits into
devfrom
jonah/omn-20415-landing-decision-shared-cause
Oct 4, 2026
Merged

jonahgabriel merged 8 commits into
devfrom
jonah/omn-20415-landing-decision-shared-cause

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Ticket OMN-20415, epic OMN-20413. Node: src/omnimarket/nodes/node_pr_landing_decision_compute (pure definition-B compute, handle(ModelLandingFacts) -> ModelLandingDecision). Contract version 0.1.0 -> 0.2.0. metadata.yaml unchanged (no new dependencies or capabilities).

Seam-first declaration

T1 verdict (OMN-20414, omnibase_internal#144, merged): TLC on h201, all 82 configurations meet their expectation, P1 to P14 and L1 to L6 hold, scenarios S21 to S28 are reached and every mutant is caught (tla/landing_controller/README.md, "TLC results"). The model found two gaps in plan section 2.4, LC-F2 and LC-F3; this PR implements both (below), and the plan is amended in knowledge-base-internal#1006. Plan R8 (T1 verdict cited) is met; the PR was a draft until the park-vs-fallback ruling below; it is ready for review on this head and handed to the landing controller.

What changed

Rules added: normalize_signature (timestamps, shas, #n, bracketed id lists and digit runs replaced; the generic "Process completed with exit code N" line skipped; whitespace collapsed; cut to 300 chars; signature = first 12 hex of sha256 over check name + normalized text; no annotation = unread, never clusters). Clusters: same repo + check + signature, >=3 members (>=2 in a floor-breached repo), follower checks and the hostile reviewer checks excluded, largest first, a cluster >=80% inside a chosen cause is absorbed carrying its pairs, cause key cause:/:, a PR in at most one cause, an unused rerun reruns first. A gate-suspended PR joins once gate_since is more than 60 minutes old; hold, draft, do_not_land never join; new owned suspension may join but never gets a per-PR worker and is never merged. A parked (escalation_exhausted) PR with a readable annotation and no covering cluster becomes a cause of one. One lease per cause key in the shared lease-id space; caps 4 cause workers fleet-wide, 2 per repo, 6 workers of any kind per repo (also applied to per-PR dispatch), inside max_workers; causes dispatch before per-PR workers, floor-breached repos first, then member count, then key. Member suppression: no per-PR worker for a member of a cause with a live lease, an open fix PR or a park. Budget: 2 attempts per park episode (a recorded result, or the deadline passing with the process alive); an exit with no result inside 600 s is a spawn failure, three park the cause. Park 12 h with exactly one escalate_operator action, dedupe key @<parked_until>; a RELEASE row dated after the park start ends it early. Fixer HOLD (repo or all): no dispatch in scope, every live lease in scope revoked through the existing kill sequence; merges, reruns, branch updates continue. cause_fix_submitted verified against the fix PR in the facts (open at fix_head, or merged), else invalid and the attempt is spent; after the fix merges one rerun per member head; cause_not_shared returns members to per-PR at rung 0 and blocks that key for that member set until a member head moves. A cause is owned only by a cause_owners entry (a CLAIM naming the cause), never by per-PR CLAIMs. All new facts, state and policy fields are optional with defaults.

Replay fixture derivation

Fixture tests/unit/nodes/node_pr_landing_decision_compute/fixtures/replay_2026_10_03_omnimarket.json: the landing controller facts snapshot of tick 725 (observed 2026-10-03T21:00:21Z, the input of that tick, same cluster as tick 722), trimmed to 25 omnimarket PRs, 5 live leases with probes, 27 records. Source sha256: landing-facts-snapshot.json 31610a0cd3c6052361126554bdb162bc9df33f0eb63da7b6a4faefbde83784ec, omnimarket-red-annotations.json 776bbf7a1624c24e883ae9c37bb5ad24240c734b976e31ddfbf5e85ea1b3d303. red_annotations: per red check, the failure-level annotations (at most 3, newline-joined) of the latest failing check-run copy on the snapshot head from the GitHub rollup read at about 21:05Z; red checks with no failing copy in that read are absent (unread). gate_since (not in the snapshot): completed_at of the first-created (lowest id) failing check-run copy of the PR's red checks on the snapshot head, read from the GitHub check-runs API by the ids the rollup read carries. The controller tick tail shows every such PR in its companion wait from tick 721 (20:36:14Z), so the true wait is between 24 minutes and that bound; test_replay_tick722_lower_bound_gate_still_forms_a_cause replays the lower bound (one cause, #3321 inside, only the 3 PRs with no gate). omnimarket#3340 (gate_since 20:12:19Z, 48 minutes) stays out. Author handles in annotation text replaced by role names; floor alarm state not in the snapshot, read as no breach.

RED proof

Replay result: one cause, key cause:OmniNode-ai/omnimarket:52c59723e1ad, 19 members, omnimarket#3321 a member, no per-PR dispatch_worker for any member, the cause dispatch is the first action, cause lease 5400 s on claude_opus; the live per-PR lease on member omnimarket#3338 runs to its own deadline.

Commits 7dfcbd5 (tests) then 1791b89 (implementation). With the node files checked out from origin/dev (identical to b10129a), test_landing_decision_causes.py: 45 failed, 1 passed (the passing one is the static fixture hygiene check). Per selector at b10129a: replay_tick722 3 failed 1 passed; cause_permutation 10 failed; cause_scenario 8 failed; budget 5 failed; fixer_hold 3 failed; escalated_singleton 3 failed. test_landing_decision_scenarios.py (S1-S20) 60 passed at b10129a and unchanged.

At head 1791b89 on h201: replay_tick722 4 passed; cause_permutation 10 passed; cause_scenario 8 passed; budget 5 passed; fixer_hold 3 passed; escalated_singleton 3 passed; whole node suite 235 passed (S1-S20 file unchanged); ruff format and ruff check clean; mypy --strict on the node clean; check_node_metadata_dependencies PASS.

Acceptance selectors

  • replay_tick722
  • cause_permutation
  • cause_scenario
  • budget
  • fixer_hold
  • escalated_singleton

Lab

Lab: host h201, lane rpf-OMN-20415, head 1791b89, command uv run pytest tests/unit/nodes/node_pr_landing_decision_compute -q in the lane worktree on h201, exit 0, observed 235 passed. onex-lab-run --via bus placement refused with exit 75 twice (h200 over the 0.75 load/core bar at 0.83 and 0.87; h201 at load 26.9 on 32 cores not offered).
Lab: host h201, lane rpf-OMN-20415, head 1791b89, command landing_decide_bridge.py --node-src src < replay facts (bridge fetched read-only from omniclaude-internal main 54d19db4b71d), exit 0, observed: actions = [dispatch_worker cause:OmniNode-ai/omnimarket:52c59723e1ad, 19 members, claude_opus, attempt 1], cause lease id 2254, no per-PR dispatch, degraded escalation_exhausted omnimarket#3321.

LC-F2 and LC-F3 (T1 model findings)

  • LC-F2 (mutant M_P11_late_exit_uncounted): an exit with no result after the 600 s spawn window and before the deadline spends an attempt, so a cause whose worker keeps dying parks after two attempts with one escalation. The node already did this (model question 2 below is answered yes, no code change needed); the tests lc_f2 pin it. RED proof is by mutation, since the behavior pre-exists: changing the attempt branch of _cause_lease_ended to the plan text (deadline only) fails all 3 lc_f2 tests; restored, they pass. Selector: -k lc_f2.
  • LC-F3 (mutant M_P14_sidecar_dedupe): the escalation dedupe reads the ledger row, the operator MSG carrying <cause key>@<parked_until>. New facts field cause_escalations (cause, at, dedupe_key; optional, empty by default). A park covered by a row younger than a park, with no later RELEASE row, adopts the row's time and key and emits no escalate_operator; otherwise it escalates as before. A crash between the MSG append and the state write therefore sends no second MSG. RED first: commit 6fee85c holds the tests, 5 of the 6 lc_f3 tests failing on the unknown facts field (the sixth, the no-row positive control, passes); commit 05d5b2c implements it, all pass. Selector: -k lc_f3. The controller half (read the ledger for the key before appending) is T4, OMN-20417.
  • Whole node suite at 05d5b2c on the Mac: 244 passed (S1-S20 file unchanged); ruff format and check clean; mypy --strict on the node clean. The earlier project version bump is dropped by the rebase: dev carries 0.4.290 after the release-open commit.

Park versus fallback (operator ruling 2026-10-04T10:11:35Z, ledger line 44477: fixed and landed, not parked)

Before: a cause that spent its budget parked for 12 hours and its members stayed suppressed, so no fixer ran for any of those red PRs for 12 hours. Now: a park alone releases its members to per-PR dispatch (rung 0, under the per-PR caps, pool and load pause). The cause itself still does not redispatch while parked, still escalates exactly once per park episode (dedupe and the LC-F3 ledger-row dedupe unchanged), and a RELEASE row or the 12 h expiry ends the park as before. A live cause lease, an open fix PR, or a park that an owner CLAIM covers still hold the members; a fixer HOLD still dispatches nothing; cause_not_shared still returns members to per-PR; a per-PR parked PR stays parked. Code: _holds_members in handler_pr_landing_decision.py (_active unchanged, so _try_dispatch_cause still refuses a parked cause). Tests: 12 test_fallback_* tests in test_landing_decision_causes.py, written RED first (10 failed, 275 passed on the unchanged handler), plus the s23, three-spawn-failure and lc_f2/lc_f3 assertions moved from "members stay suppressed" to the fallback. Mutating _holds_members to always return False fails 46 tests. The plan text "Members stay suppressed while parked" (knowledge-base-internal beta/plans/2026-10-03-red-pr-fixer-plan.md, Park and escalation) is superseded by this ruling.

Lab: host h202, lane market-3342-fallback, head 2299ddf, command uv run pytest tests/unit/nodes/node_pr_landing_decision_compute -q via onex_lab_run.py --via ssh (the bus path refused with exit 69), exit 0, observed 285 passed. Host h201 ruff format, ruff check and mypy --strict on the node exit 0; pre-commit on the three touched files exit 0. The tick-725 replay (cause:OmniNode-ai/omnimarket:52c59723e1ad, 19 members, no per-PR member dispatch while the lease is live) is among the 285.

Install and rebase

This is a decision-node change (node_pr_landing_decision_compute, contract 0.2.0). It installs through the same G-ACT re-pin of the landing controller's pinned decision node as omnimarket#3359 (OMN-20508), which landed first as 2332f6f on dev; this branch is rebased onto dev acd8a5b and resolved against it (the stale-refresh docstring item and the STALE_REFRESH_EXHAUSTED and CAUSE_EXHAUSTED degraded reasons coexist). One G-ACT re-pin installs both.

Model questions for OMN-20414

Each item is what this node does today, chosen from the plan text where the plan does not decide the interleaving; the model should confirm or overrule it.

  1. Does a live per-PR lease on a member run to its own deadline when the cause dispatches?
  2. Does an exit with no result after the 600 s spawn window but before the deadline currently count as an attempt?
  3. Does an absorbed cluster add its members not already in another cause to the host cause, and does a non-absorbed cluster form only if its members not already in a cause still meet the threshold?
  4. Does an escalated singleton need at least one readable annotation, and does an unread parked PR stay parked?
  5. Does a cause CLAIM also suppress per-PR dispatch for the members?
  6. Is an idle cause record that does not re-form in a tick dropped, so the attempt count restarts if the cluster re-forms later?
  7. Does external_blocker from a cause worker spend the attempt and does the cause re-dispatch while present, with no blocker-fingerprint suppression?
  8. After a fix merges, do members wait until each rerun registers and grades, and does a rerun that never registers hold them (fails closed)?
  9. Does cause_not_shared reset rung and park but keep per-PR blocked-outcome suppression?
  10. Does a floor breach lower the cluster threshold only, and is the 60-minute stall window unchanged?
  11. Is "reviewer pool" read as the hostile reviewer checks, set in the cluster_excluded_checks policy field?
  12. Under a fixer HOLD, is a spent cause not parked or escalated until the hold lifts?

Evidence-Ticket: OMN-20415
Evidence-Source: OCC#12778

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 14
Models succeeded: qwen3-review,gpt-oss-review


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

@jonahgabriel

Copy link
Copy Markdown
Contributor Author

Lab: head=05d5b2ce25af7000d4ec657c3d8b75c0cc5bfc73
Runner: onex_lab_run.py --via bus, lane rpf-3342-labproof, --ref jonah/omn-20415-landing-decision-shared-cause (head 05d5b2c). ONEX_DISPATCH_VENV was set after an initial exit 64; every placed run landed on host h200 (macOS) through the event bus.
Command prefix for each run: uv run pytest tests/unit/nodes/node_pr_landing_decision_compute -q
Run 1, host h200, -k "replay_tick722 or cause_permutation or cause_scenario or budget or fixer_hold or escalated_singleton or lc_f2 or lc_f3": exit 0, 41 passed, 203 deselected.
Run 2, host h200, -k replay_tick722: exit 0, 4 passed.
Run 3, host h200, -k cause_permutation: exit 0, 10 passed.
Run 4, host h200, -k cause_scenario: exit 0, 8 passed.
Run 5, host h200, -k fixer_hold: exit 0, 3 passed.
Run 6, host h200, whole node suite (no -k): exit 0, 244 passed (the whole suite includes the budget, escalated_singleton, lc_f2 and lc_f3 tests).
Separate -k runs for budget, escalated_singleton, lc_f2 and lc_f3 were each refused by the placer with exit 75 (over_capacity, h200 over the 0.75 load/core bar or its unit slot busy); they were not run on the launching host. Their tests are covered by run 1 and run 6.
Tick-725 replay: the committed fixture tests/unit/nodes/node_pr_landing_decision_compute/fixtures/replay_2026_10_03_omnimarket.json is exercised by the replay_tick722 tests (run 2, 4 passed), which assert one cause with key cause:OmniNode-ai/omnimarket:52c59723e1ad, 19 members including omnimarket#3321, and no per-PR dispatch for a member.
Source sha256 cross-check: landing-facts-snapshot.json 31610a0cd3c6052361126554bdb162bc9df33f0eb63da7b6a4faefbde83784ec and omnimarket-red-annotations.json 776bbf7a1624c24e883ae9c37bb5ad24240c734b976e31ddfbf5e85ea1b3d303 recomputed with sha256sum from .claude_scratch/red-pr-fixer/ match the values in the PR body, and the fixture file names the same two values.
CI on this head: not read from the PR watcher state, which was not available on the lane host (pr_state_local.py rc=3, no state file); CI state is UNREAD by this lane.
The PR stays draft; no code change was made by this lane.

Adds test_landing_decision_causes.py with 46 unit tests for node PR landing decision compute.
Includes fixtures/replay_2026_10_03_omnimarket.json containing the landing controller facts snapshot of tick 725.
The fixture is trimmed to 25 omnimarket PRs, 5 live leases and their records.
Author handles in annotation text are replaced by role names.
Test selectors cover replay_tick722, cause_permutation, cause_scenario S21 to S28, budget, fixer_hold, and escalated_singleton.
These tests fail against the current node because it lacks shared-cause rules.
The landing decision node groups red PRs in one repo that fail one check with one normalized failure signature into a shared cause.
One cause worker lease is issued per cause key and dispatched before per-PR workers.
Fleet-wide caps limit cause workers to 4, with 2 per repo and 6 workers of any kind per repo.
No per-PR worker is created for a member of a cause with a live lease, an open fix PR, or a park.
Two attempts are allowed per park episode, followed by a 12-hour park with one escalate_operator action.
A fixer HOLD revokes every live lease in its scope and dispatches nothing, while merges and reruns continue.
All new fields are optional, so existing S1 to S20 scenarios pass unchanged, and the contract version is 0.2.0.
@jonahgabriel
jonahgabriel force-pushed the jonah/omn-20415-landing-decision-shared-cause branch from 05d5b2c to c76bbdf Compare October 4, 2026 10:22
Parked shared causes no longer suppress member PRs. Members now
take the normal per-PR path (rung 0 ladder, per-PR caps, worker
pool, load pause) as a fallback so red PRs keep getting fixed
during the park.

The cause itself still does not redispatch while parked and still
escalates once per park episode. A RELEASE row or the 12 h expiry
still ends the park. A live cause lease, an open fix PR, or an
owner CLAIM on a parked cause still holds the members. A fixer
HOLD still dispatches nothing. Per-PR parked PRs stay parked.
cause_not_shared unchanged.

Implemented with a new helper _holds_members; _active is unchanged
for dispatch.

Tests: 12 new fallback tests; s23, three-spawn-failure, lc_f2 and
lc_f3 assertions updated from "members suppressed while parked" to
the fallback. Handler docstring and contract.yaml 0.2.0 header
updated; contract version stays 0.2.0.
…n 0.4.291 for the release-identity gate

Onex-Lane: market-3342-land
Onex-Session: 64e92e670889474bb15ab2c7d6ebc530
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 4, 2026
…nimarket#3342

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/37195884603
…e; S1 to S20 stay evidence

Onex-Lane: market-3342-land
Onex-Session: 64e92e670889474bb15ab2c7d6ebc530
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 4, 2026
)

* evidence(OMN-20415): author OCC companion for OmniNode-ai/omnimarket#3342

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 2299ddfaf1eb505f2e57e7c48991672b1ae0272f.

* evidence(OMN-20415): self-bind OCC#12778 + rebind contract_sha256

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#3342

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/37195884603

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <307849072+onexbot-occ-writer[bot]@users.noreply.github.com>
…caller test requires

Onex-Lane: market-3342-land
Onex-Session: 64e92e670889474bb15ab2c7d6ebc530
@jonahgabriel
jonahgabriel merged commit 4d7e6e6 into dev Oct 4, 2026
153 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-20415-landing-decision-shared-cause branch October 4, 2026 11:22
@jonahgabriel

Copy link
Copy Markdown
Contributor Author

Lab: head=9a5be7658e9d5f7eaef242e6c5e75760c249908b
Host h202 (via onex_lab_run.py --via ssh, lane market-3342-fallback), code head 2299ddf, command uv run pytest tests/unit/nodes/node_pr_landing_decision_compute -q, exit 0, observed 285 passed (RED on the unchanged handler: 10 failed, 275 passed). Head 9a5be76 adds only contracts/OMN-20415.yaml and the project version 0.4.291 on top of that code head; the focused selectors replay_tick722 (4), cause_permutation (10), cause_scenario (8), budget or fixer_hold or fallback (22) and escalated_singleton (3) pass on it locally, tests/ci/test_repo_evidence_caller_workflow.py passes (2), and CI Summary is green on this head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant