Skip to content

feat(OMN-14909): ship CI-01 upstream-failure-explosion cascade aggregator (count roots, not checks) - #2391

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-14909-ci01-cascade-reason-graph-aggregator
Jul 22, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-14909-ci01-cascade-reason-graph-aggregator

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Jul 21, 2026 •

Copy link
Copy Markdown
Collaborator

OMN-14909 — Ship the CI-01 upstream-failure-explosion fix (plan §3.C5)

Root-cause class CI-01 ("upstream failure explosion") from the 2026-07-16 CI
remediation plan; scheduled as C5 in docs/plans/2026-07-21-ci-capacity-recovery-plan.md.
Designed, previously not shipped.

Problem

On omnibase_infra#2370, a single defect (missing Evidence-Source: → occ-preflight / eligibility red) amplified into 36 red check-runs — 35 occ-preflight / eligibility (one per calling workflow, the C2 fan-out) + 1 CI Summary — plus 74
skipped
needs: occ-preflight dependents. ~37× amplification of one defect. Each of
the 35 occ reds is its own workflow run and therefore independently rerunnable,
which provokes the rerun reflex: the wall of red invites broad reruns, each
re-spending the 40-way matrix. This is a CI-cost driver, not only a diagnosis issue.

Fix — count ROOTS, not checks

New scripts/ci/ci_cascade_graph.py consumes the head SHA's full cross-workflow
check-run list (GET /commits/{sha}/check-runs) and collapses it into:

  • exactly one typed root cause — root election reuses the deterministic,
    unit-tested product_reason_graph.build_reason_graph classifier, so the
    OCC-independence property (a real product defect roots as PRODUCT_FAILED, not
    EVIDENCE_MISSING) is preserved;
  • every other red/skipped check → BLOCKED_UPSTREAM, content-addressed to the single
    root_receipt_id;
  • exactly one rerunnable unit (the root) — every dependent carries
    rerunnable=False. This is the anti-cosmetic guarantee: if a dependent were still
    independently rerunnable, the rerun reflex would survive.

Wired as an additive, report-only render in the CI Summary job: if: always()

  • continue-on-error: true, writes only to $GITHUB_STEP_SUMMARY, every command ends
    || true. It never changes the verdict — the existing poll step is the sole
    pass/fail authority.

Proof (plan §3.C5)

tests/ci/test_ci_cascade_graph_omn14909.py replays #2370's real captured blocked
head
(tests/ci/fixtures/omn14909_2370_cascade_checkruns.json, 122 check-runs):

  • root_count == 1, root kind EVIDENCE_MISSING;
  • 35 red + 74 skipped dependents, all BLOCKED_UPSTREAM, all bound to the single
    root receipt;
  • rerunnable_unit_count == 1 and every dependent rerunnable is False — no
    dependent independently rerunnable;
  • the 35 occ reds collapse to 1 ROOT + 34 BLOCKED_UPSTREAM; the red CI Summary is a
    dependent, not an independent failure.

(The plan's "1 root + 36 dependents" reflects its 37-red snapshot; the captured current
head is 36-red → 1 root + 35 red dependents + 74 skipped. Same structure.)

Gate not weakened

The CI Summary verdict logic (ci_summary_gate.py, the poll step) is byte-unchanged.
Seeded proof: an occ-preflight / eligibility red jobs list still yields
ci_summary_gate.py exit 1 (FAILURE). actionlint reports 0 new findings from
the added step.

dod_evidence

  • uv run pytest tests/ci/test_ci_cascade_graph_omn14909.py → 7 passed
  • regression: test_product_reason_graph_omn14707.py + test_ci_summary_gate.py → 58 passed
  • uv run mypy scripts/ci/ci_cascade_graph.py → clean; ruff check/format clean; pre-commit 46 passed / 0 failed
  • gate-not-weakened: seeded occ-red jobs.json → ci_summary_gate.py exit 1
  • -m CLI on workflow-shaped NDJSON → root_count=1, EVIDENCE_MISSING, rerunnable_units=1

Report-only; no gate weakened; no branch-protection writes. Single-repo canary; fan-out
to omnibase_core / omnimarket / omniclaude / onex_change_control (which carry the
same product-readiness-shadow surface) is a follow-up, not this pass.

Closes OMN-14909

Evidence-Ticket: OMN-14909
Evidence-Source: OCC#4569

…ator (count roots, not checks)

Plan section 3.C5 (docs/plans/2026-07-21-ci-capacity-recovery-plan.md); root-cause
class CI-01 from the 2026-07-16 CI remediation plan - designed, not shipped.

On omnibase_infra#2370 a single defect (missing Evidence-Source -> occ-preflight
eligibility red) amplifies into 36 red check-runs (35 occ-preflight/eligibility, one
per calling workflow, + 1 CI Summary) plus 74 skipped needs:occ-preflight dependents
- ~37x amplification. Because each occ red is its own workflow run, each is
independently rerunnable, which provokes the rerun reflex (a wall of red invites
broad reruns, each re-spending the 40-way matrix).

Ships scripts/ci/ci_cascade_graph.py, which consumes the head SHA's full
cross-workflow check-run list (GET /commits/{sha}/check-runs) and collapses it into:
  - exactly ONE typed root cause - root election reuses the deterministic,
    unit-tested product_reason_graph.build_reason_graph classifier, preserving the
    OCC-independence property (a real product defect roots as PRODUCT_FAILED, not
    EVIDENCE_MISSING);
  - every other red/skipped check marked BLOCKED_UPSTREAM, content-addressed to the
    single root_receipt_id; and
  - EXACTLY one rerunnable unit (the root) - every dependent carries
    rerunnable=False, the anti-cosmetic guarantee.

Wired as an additive, report-only render in the CI Summary job: if always() +
continue-on-error true, writes only to GITHUB_STEP_SUMMARY, ends every command with
|| true. It NEVER changes the verdict - the poll step remains the sole pass/fail
authority. Gate not weakened: a seeded occ-preflight red still yields ci_summary_gate
exit 1 (CI Summary = FAILURE).

Proof: tests/ci/test_ci_cascade_graph_omn14909.py replays #2370's real captured
blocked head (tests/ci/fixtures/omn14909_2370_cascade_checkruns.json) - 1 root
(EVIDENCE_MISSING), 35 red + 74 skipped dependents all BLOCKED_UPSTREAM, 1 rerunnable
unit, no dependent independently rerunnable. Plus OCC-independence, determinism,
green-head base case, and report-only-exit-0 CLI tests.

dod_evidence:
- uv run pytest tests/ci/test_ci_cascade_graph_omn14909.py -> 7 passed
- regression test_product_reason_graph_omn14707.py + test_ci_summary_gate.py -> 58 passed
- uv run mypy scripts/ci/ci_cascade_graph.py -> clean; ruff clean; pre-commit 46 passed / 0 failed
- gate-not-weakened: seeded occ-preflight-red jobs.json -> ci_summary_gate.py exit 1 (FAILURE)
- actionlint: 0 new findings from the added CI Summary step

Report-only; no gate weakened; no branch-protection writes; single-repo (fan-out to
core/omnimarket/omniclaude/onex_change_control is a follow-up).

Closes OMN-14909
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c3ec6436-aca8-432b-9d10-54b1faf74f7f

📥 Commits

Reviewing files that changed from the base of the PR and between f02b78d and ff86ee7.

📒 Files selected for processing (4)
  • .github/workflows/ci.yml
  • scripts/ci/ci_cascade_graph.py
  • tests/ci/fixtures/omn14909_2370_cascade_checkruns.json
  • tests/ci/test_ci_cascade_graph_omn14909.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jonah/omn-14909-ci01-cascade-reason-graph-aggregator

Comment @coderabbitai help to get the list of available commands.

@jonahgabriel
jonahgabriel merged commit 41e0661 into dev Jul 22, 2026
196 of 232 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-14909-ci01-cascade-reason-graph-aggregator branch July 22, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant