Repository navigation
ci(OMN-13949): concurrency cancel-in-progress + job timeouts to kill superseded/zombie runs (SAT1) - #2209
Conversation
…superseded/zombie runs (SAT1)
Fleet-load reduction (Track 1, OMN-13932 §5.2). Superseded CI runs were
holding saturated self-hosted runners.
- Add top-level concurrency { group: workflow-<pr#|ref>, cancel-in-progress: true }
to PR-triggered check/test/gate workflows (24 workflows). merge_group entries
carry a unique github.ref so distinct queue entries never cross-cancel; the
required contexts still report on merge_group.
- Normalize 5 workflows that kept superseded PR runs alive (ci.yml,
compose-config-lint, env-parity, lane-census-staleness, runner-disk-preflight)
from cancel:false / merge_group-only to cancel-in-progress: true.
- Deploy/release/prod-mutation/tag workflows deliberately NOT given
cancel-in-progress: release.yml, release-python-reusable.yml,
build-and-push-*, auto-tag-on-merge, runtime-rebuild-trigger,
prod-promotion-lineage stay serialized (cancel:false) or untouched — never
cancel an in-flight prod build/tag/deploy.
- Add safety-net timeout-minutes to every job lacking one (37 jobs across
21 workflows; reusable 'uses:' jobs skipped) so a hung job cannot hold a
runner indefinitely.
Verified: tests/ci/test_ci_workflow_resilience.py + test_runner_disk_preflight.py
(51 passed); actionlint clean (only pre-existing shellcheck/secret findings);
pre-commit clean on changed files.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 6 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR updates over 40 GitHub Actions workflow files to add or modify ChangesWorkflow Concurrency and Timeout Configuration
Estimated code review effort: 2 (Simple) | ~12 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
1 participant
Summary
Track 1 fleet-load reduction (OMN-13932 §5.2 / SAT1). Superseded and zombie CI runs were occupying saturated self-hosted runners that could otherwise serve fresh work. This makes killing superseded runs the default and caps hung jobs.
Ticket: OMN-13949 (parent epic OMN-13932). Plan:
docs/plans/2026-07-05-runner-fleet-permanent-fix-plan.md§5.2.Changes
concurrency: { group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}, cancel-in-progress: true }to 24 PR-triggered check/test/gate workflows, and normalized 5 that previously kept superseded PR runs alive (ci.yml,compose-config-lint,env-parity,lane-census-staleness,runner-disk-preflight) fromcancel:false/ merge_group-only tocancel-in-progress: true.github.ref; onmerge_groupeach queue entry carries a uniquegithub.ref, so distinct entries never cross-cancel and required contexts still report. No required merge_group context is wedged.release.yml,release-python-reusable.yml,build-and-push-runtime.yml,build-and-push-migrate-image.ymlkeep serializedcancel-in-progress: false;auto-tag-on-merge,runtime-rebuild-trigger,prod-promotion-lineagereceived no concurrency block — never cancel an in-flight prod build/tag/deploy.timeout-minutesto every job that lacked one (37 jobs across 21 workflows; reusableuses:jobs skipped, which cannot carry the key) so a hung job cannot hold a runner indefinitely. Heavy self-hosted jobs (docker-build, ci test-parallel) already had explicit timeouts and were left as-is.dod_evidence
tests/ci/test_ci_workflow_resilience.py+tests/ci/test_runner_disk_preflight.py: 51 passed on the dev base.actionlint .github/workflows/*.yml: no new structural findings (only pre-existing shellcheck-style andcross_repo_patsecret-name notes).pre-commit runon all changed files: clean (includingonex-check-node-migration-sync).cancel-in-progress: true.Summary by CodeRabbit
Evidence-Source: OCC#3612
Evidence-Ticket: OMN-13949