Repository navigation
feat(OMN-14909): ship CI-01 upstream-failure-explosion cascade aggregator (count roots, not checks) - #2391
Conversation
…ator (count roots, not checks)
Plan section 3.C5 (docs/plans/2026-07-21-ci-capacity-recovery-plan.md); root-cause
class CI-01 from the 2026-07-16 CI remediation plan - designed, not shipped.
On omnibase_infra#2370 a single defect (missing Evidence-Source -> occ-preflight
eligibility red) amplifies into 36 red check-runs (35 occ-preflight/eligibility, one
per calling workflow, + 1 CI Summary) plus 74 skipped needs:occ-preflight dependents
- ~37x amplification. Because each occ red is its own workflow run, each is
independently rerunnable, which provokes the rerun reflex (a wall of red invites
broad reruns, each re-spending the 40-way matrix).
Ships scripts/ci/ci_cascade_graph.py, which consumes the head SHA's full
cross-workflow check-run list (GET /commits/{sha}/check-runs) and collapses it into:
- exactly ONE typed root cause - root election reuses the deterministic,
unit-tested product_reason_graph.build_reason_graph classifier, preserving the
OCC-independence property (a real product defect roots as PRODUCT_FAILED, not
EVIDENCE_MISSING);
- every other red/skipped check marked BLOCKED_UPSTREAM, content-addressed to the
single root_receipt_id; and
- EXACTLY one rerunnable unit (the root) - every dependent carries
rerunnable=False, the anti-cosmetic guarantee.
Wired as an additive, report-only render in the CI Summary job: if always() +
continue-on-error true, writes only to GITHUB_STEP_SUMMARY, ends every command with
|| true. It NEVER changes the verdict - the poll step remains the sole pass/fail
authority. Gate not weakened: a seeded occ-preflight red still yields ci_summary_gate
exit 1 (CI Summary = FAILURE).
Proof: tests/ci/test_ci_cascade_graph_omn14909.py replays #2370's real captured
blocked head (tests/ci/fixtures/omn14909_2370_cascade_checkruns.json) - 1 root
(EVIDENCE_MISSING), 35 red + 74 skipped dependents all BLOCKED_UPSTREAM, 1 rerunnable
unit, no dependent independently rerunnable. Plus OCC-independence, determinism,
green-head base case, and report-only-exit-0 CLI tests.
dod_evidence:
- uv run pytest tests/ci/test_ci_cascade_graph_omn14909.py -> 7 passed
- regression test_product_reason_graph_omn14707.py + test_ci_summary_gate.py -> 58 passed
- uv run mypy scripts/ci/ci_cascade_graph.py -> clean; ruff clean; pre-commit 46 passed / 0 failed
- gate-not-weakened: seeded occ-preflight-red jobs.json -> ci_summary_gate.py exit 1 (FAILURE)
- actionlint: 0 new findings from the added CI Summary step
Report-only; no gate weakened; no branch-protection writes; single-repo (fan-out to
core/omnimarket/omniclaude/onex_change_control is a follow-up).
Closes OMN-14909
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
OMN-14909 — Ship the CI-01 upstream-failure-explosion fix (plan §3.C5)
Root-cause class CI-01 ("upstream failure explosion") from the 2026-07-16 CI
remediation plan; scheduled as C5 in
docs/plans/2026-07-21-ci-capacity-recovery-plan.md.Designed, previously not shipped.
Problem
On
omnibase_infra#2370, a single defect (missingEvidence-Source:→occ-preflight / eligibilityred) amplified into 36 red check-runs — 35occ-preflight / eligibility(one per calling workflow, the C2 fan-out) + 1CI Summary— plus 74skipped
needs: occ-preflightdependents. ~37× amplification of one defect. Each ofthe 35 occ reds is its own workflow run and therefore independently rerunnable,
which provokes the rerun reflex: the wall of red invites broad reruns, each
re-spending the 40-way matrix. This is a CI-cost driver, not only a diagnosis issue.
Fix — count ROOTS, not checks
New
scripts/ci/ci_cascade_graph.pyconsumes the head SHA's full cross-workflowcheck-run list (
GET /commits/{sha}/check-runs) and collapses it into:unit-tested
product_reason_graph.build_reason_graphclassifier, so theOCC-independence property (a real product defect roots as
PRODUCT_FAILED, notEVIDENCE_MISSING) is preserved;BLOCKED_UPSTREAM, content-addressed to the singleroot_receipt_id;rerunnable=False. This is the anti-cosmetic guarantee: if a dependent were stillindependently rerunnable, the rerun reflex would survive.
Wired as an additive, report-only render in the
CI Summaryjob:if: always()continue-on-error: true, writes only to$GITHUB_STEP_SUMMARY, every command ends|| true. It never changes the verdict — the existing poll step is the solepass/fail authority.
Proof (plan §3.C5)
tests/ci/test_ci_cascade_graph_omn14909.pyreplays#2370's real captured blockedhead (
tests/ci/fixtures/omn14909_2370_cascade_checkruns.json, 122 check-runs):root_count == 1, root kindEVIDENCE_MISSING;BLOCKED_UPSTREAM, all bound to the singleroot receipt;
rerunnable_unit_count == 1and every dependentrerunnable is False— nodependent independently rerunnable;
BLOCKED_UPSTREAM; the redCI Summaryis adependent, not an independent failure.
(The plan's "1 root + 36 dependents" reflects its 37-red snapshot; the captured current
head is 36-red → 1 root + 35 red dependents + 74 skipped. Same structure.)
Gate not weakened
The
CI Summaryverdict logic (ci_summary_gate.py, the poll step) is byte-unchanged.Seeded proof: an
occ-preflight / eligibilityred jobs list still yieldsci_summary_gate.pyexit 1 (FAILURE).actionlintreports 0 new findings fromthe added step.
dod_evidence
uv run pytest tests/ci/test_ci_cascade_graph_omn14909.py→ 7 passedtest_product_reason_graph_omn14707.py+test_ci_summary_gate.py→ 58 passeduv run mypy scripts/ci/ci_cascade_graph.py→ clean;ruff check/formatclean;pre-commit46 passed / 0 failedci_summary_gate.pyexit 1-mCLI on workflow-shaped NDJSON →root_count=1,EVIDENCE_MISSING,rerunnable_units=1Report-only; no gate weakened; no branch-protection writes. Single-repo canary; fan-out
to
omnibase_core/omnimarket/omniclaude/onex_change_control(which carry thesame
product-readiness-shadowsurface) is a follow-up, not this pass.Closes OMN-14909
Evidence-Ticket: OMN-14909
Evidence-Source: OCC#4569