Repository navigation
ci: auto-rerun nightly CI on failed/flaky jobs - #9174
Merged
Merged
Conversation
Add a workflow_run follow-up that re-runs the failed jobs of a "TheRock CI Nightly" run on fresh runners, up to 4 total attempts, mirroring the web-UI "Re-run failed jobs" button. Most nightly failures are infra flakes (no GPU, overloaded-machine timeout, network blip on artifact fetch) that clear on a healthy runner. Closes #9173 Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
github actions
project: none
tony-davis
marked this pull request as ready for review
July 8, 2026 00:37
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions follow-up workflow that listens for completed TheRock CI Nightly runs and automatically re-runs only the failed jobs (fresh runners) until a fixed attempt cap is reached, reducing manual intervention for infra flakes.
Changes:
- Introduces a new
workflow_run-triggered workflow to auto-rerun failed nightly jobs. - Applies guardrails via
conclusionfiltering and an attempt cap (run_attempt < 4). - Uses
gh run rerun --failedwithactions: writepermissions to mirror the UI behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
🎉 All checks passed! This PR is ready for review. |
tony-davis
enabled auto-merge (squash)
July 8, 2026 03:05
geomin12
reviewed
Jul 8, 2026
geomin12
left a comment
geomin12
left a comment
Collaborator
There was a problem hiding this comment.
let's do this as a trial run. if great, we can apply it to other workflows across ROCm
This was referenced Jul 21, 2026
tony-davis
added a commit
that referenced
this pull request
Jul 21, 2026
## Summary Extends the existing auto-retry watcher (added in #9174) to also cover `TheRock Multi-Arch Nightly CI`. It adds that workflow name to the `workflow_run` list in `.github/workflows/auto-retry-failed.yml`; the rerun job itself is generic (it re-runs the failed jobs of the triggering run), so no other change is needed. Most nightly failures are infrastructure flakes (no GPU assigned, an overloaded-machine timeout, a network blip during artifact fetch) that clear on a healthy runner. Closes #9173. ## Risk Assessment Risk 2. One-line addition to an isolated CI orchestration workflow. It does not touch any build or test lane, reusable workflow, or product code, and only acts on nightly runs that already concluded in `failure`. Worst case if it misbehaves is extra nightly runner usage, bounded by the `run_attempt < 4` cap; it cannot affect PR/presubmit CI. ## Related - Work tracking: #9173 (Closes) - Predecessor: #9174 (added `TheRock CI Nightly`) ## Device / Architecture Coverage Architecture-independent. CI orchestration only (it calls the GitHub API to re-run jobs); runs no device code and needs no per-arch verification. ## Testing Summary - YAML parses cleanly and the repo's applicable pre-commit hooks pass locally (trailing-whitespace, end-of-files, check-yaml, large-files; language hooks skipped as no relevant files). - Confirmed the watched run concludes as `failure` on a failed leg: `multi_arch_ci_summary` (`if: always()`) runs `build_tools/github_actions/workflow_summary.py`, which returns 1 when any needed job failed, so the existing `conclusion == 'failure'` gate applies unchanged. - Functional behavior can only be exercised on the default branch (`workflow_run` and the nightly schedule both use the default-branch copy). Plan: watch the first multi-arch nightly after merge and confirm a failed leg re-runs as a new attempt. ## Testing Checklist - [x] YAML parse - `python -c "import yaml; yaml.safe_load(...)"` - Status: Passed - [x] pre-commit (applicable hooks) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge multi-arch nightly observation - confirm auto-rerun fires and caps at 4 attempts - Status: Pending ## Flags / Guardrails Unchanged from #9174: retry cap `run_attempt < 4`; only fires on `conclusion == 'failure'`; `cancelled` (concurrency/superseded) and `startup_failure` excluded; requires `actions: write` on `GITHUB_TOKEN`. ## Adjacent Tests Considered No product or test code changes. The `RETRY_THIS_STEP` same-runner step retries remain complementary and unchanged. ## Technical Changes - `.github/workflows/auto-retry-failed.yml`: add `"TheRock Multi-Arch Nightly CI"` to the `workflow_run.workflows` list. Co-authored-by: Cursor <cursoragent@cursor.com>
Member
|
cc @amd-shiraz , as this overlaps with some other retry work happening on the runners themselves I'm scared to auto-retry workflows like this without some more careful review/consideration. This will to burn runner time and we should be investing in other approaches to actually de-flake build/test workflows. |
2 of 4 tasks
tony-davis
added a commit
that referenced
this pull request
Aug 13, 2026
## Summary Extends the existing auto-retry watcher (added in #9174, #9658) to also cover the per-PR `TheRock CI` workflow. Unlike the nightlies (4 total attempts), it gets only 1 auto-retry (2 total attempts) so a genuine regression still surfaces quickly and flaky infra doesn't multiply runner usage on every PR push. ## Risk Assessment Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded `failure`, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658), just applied to a second, lower-cap workflow name. ## Related - Work tracking: #10762 (Closes) - Related PRs: #9174, #9658 (nightly predecessors), #10752 (restricts `TheRock CI` to non-draft PRs -- complementary, reduces overall run volume) ## Device / Architecture Coverage Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification. ## Testing Summary - YAML parses cleanly (`python -c "import yaml; yaml.safe_load(open('.github/workflows/auto-retry-failed.yml'))"`). - `pre-commit run --files .github/workflows/auto-retry-failed.yml` passed locally (trailing-whitespace, end-of-file-fixer, check-yaml). - Functional behavior can only be exercised on the default branch, because `workflow_run` always uses the copy of this file on `develop`. Plan: watch the first `TheRock CI` failure after merge and confirm it gets exactly one auto-retry. ## Testing Checklist - [x] YAML parse - `python -c "import yaml; yaml.safe_load(...)"` - Status: Passed - [x] pre-commit (formatting/yaml) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge observation - confirm a failed `TheRock CI` run gets exactly 1 auto-retry - Status: Pending ## Flags / Guardrails Per-workflow attempt cap via `run_attempt`: `TheRock CI` gets `run_attempt < 2` (1 auto-retry, 2 total attempts); the two nightly workflows keep the existing `run_attempt < 4` (unchanged). Only fires on `conclusion == 'failure'`; `cancelled` (concurrency/superseded runs, or a run with individually cancelled jobs -- `rerun-failed-jobs` never touches those) and `startup_failure` are excluded. No new permissions; reuses the existing `actions: write` grant. ## Adjacent Tests Considered No product or test code changes. Checked that `TheRock CI`'s `cancel-in-progress: true` concurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the desired behavior (no point retrying stale code). ## Risk Acceptance / Waivers `W-BUILD`: CI orchestration only, no product behavior change; mirrors the already-proven nightly pattern from #9174/#9658. ## Technical Changes - Add `"TheRock CI"` to the `workflow_run.workflows` list in `.github/workflows/auto-retry-failed.yml`. - Change the attempt-cap `if:` condition to branch by workflow name: `run_attempt < 2` for `TheRock CI`, `run_attempt < 4` for everything else (nightlies, unchanged). - Update header/job comments to document the per-workflow cap and to note that `rerun-failed-jobs` already excludes individually cancelled jobs. Co-authored-by: Cursor <cursoragent@cursor.com>
2 of 4 tasks
davidd-amd
pushed a commit
that referenced
this pull request
Sep 28, 2026
## Summary `TheRock Multi-Arch CI` became the default per-PR presubmit in #11542, but it was never added to the auto-retry watcher's `workflows:` list, so the presubmit auto-retry added in #10763 silently stopped applying. Every failed Multi-Arch CI run today sits at attempt 1 until a human clicks "Re-run failed jobs." This adds it to the watch list with the same per-PR cap `TheRock CI` had: 1 auto-retry, 2 total attempts. ## Risk Assessment Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded `failure`, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658) and on `TheRock CI` (#10763), applied to one more workflow name. ## Related ISSUE ID: #12510 - Work tracking: Closes #12510 - Related PRs: #11542 (made Multi-Arch CI the default, which opened this gap), #9174, #9658, #10763 (the watcher's predecessors) ## Device / Architecture Coverage Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification. ## Testing Summary - Confirmed the gap is real: the last 15 failed `TheRock Multi-Arch CI` runs are all at `attempt=1`, while `TheRock Multi-Arch Nightly CI` failures reach `attempt=4` under the existing watcher. - Confirmed "re-run failed jobs" actually works on a Multi-Arch CI run, which is not obvious given its jobs come from reusable workflows in `ROCm/TheRock`. Run 35893574070 was manually re-run this way: attempt 2 re-executed only the failed test jobs (started 2026-09-24T00:22Z) while the setup jobs kept their attempt-1 timestamps (2026-09-23T17:52Z), and the re-run test jobs passed. - That run also confirms artifact addressing survives a retry: TheRock's build and test jobs key S3 artifacts on `github.run_id`, not `run_attempt`, so test jobs re-run roughly 6.5 hours later still found attempt 1's build output. - `actionlint` type-checks the `if:` expression. I verified the linter is actually reading it by breaking `contains` into `containz` on a scratch copy and watching it fail. - Functional behavior can only be exercised on the default branch, because `workflow_run` always uses the copy of this file on `develop`. Plan: watch the first Multi-Arch CI failure after merge and confirm it gets exactly one auto-retry. ## Testing Checklist - [x] actionlint - `actionlint .github/workflows/auto-retry-failed.yml` - Status: Passed - [x] pre-commit (formatting/yaml) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge observation - confirm a failed `TheRock Multi-Arch CI` run gets exactly 1 auto-retry - Status: Pending ## Flags / Guardrails Per-workflow attempt cap via `run_attempt`: the per-PR workflows (`TheRock CI`, `TheRock Multi-Arch CI`) get `run_attempt < 2` (1 auto-retry, 2 total attempts); the two nightlies keep `run_attempt < 4` (unchanged). Only fires on `conclusion == 'failure'`; `cancelled` (concurrency / superseded runs) and `startup_failure` are excluded. No new permissions; reuses the existing `actions: write` grant. ## Adjacent Tests Considered No product or test code changes. Checked that Multi-Arch CI's `cancel-in-progress: true` concurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the behavior we want. Also checked the two ASAN workflows (`TheRock Multi-Arch CI ASAN` and `TheRock Multi-Arch CI ASAN Nightly`), which are likewise unwatched; they are left out of this PR to keep the change on the default presubmit path, and are noted as follow-up in the issue. ## Risk Acceptance / Waivers `W-BUILD`: CI orchestration only, no product behavior change; mirrors the already-proven pattern from #9174, #9658, and #10763. ## Technical Changes - Add `"TheRock Multi-Arch CI"` to the `workflow_run.workflows` list in `.github/workflows/auto-retry-failed.yml`. - Replace the `== 'TheRock CI'` / `!= 'TheRock CI'` attempt-cap branch with two explicit `contains(fromJSON(...))` name lists, one per cap. The old fallback branch gave the nightly cap of 4 to any watched workflow that was not literally `TheRock CI`, so adding Multi-Arch CI without this rework would have granted presubmit 3 auto-retries. A watched workflow in neither list now gets no auto-retry at all, which fails safe on runner spend. - Update the header comment, which named `TheRock CI` as the only per-PR workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
workflow_runfollow-up workflow that automatically re-runs the failed jobs of aTheRock CI Nightlyrun on fresh runners, up to 4 total attempts. Most nightly failures are infrastructure flakes (no GPU assigned, an overloaded-machine timeout, a network blip during artifact fetch) that clear on a healthy runner, so this removes the manual "Re-run failed jobs" click. Closes #9173.Risk Assessment
Risk 2. New, self-contained CI orchestration workflow. It does not touch any existing build or test lane, reusable workflow, or product code, and it only acts on
TheRock CI Nightlyruns that already concluded infailure. The worst case if it misbehaves is extra nightly runner usage, bounded by therun_attempt < 4cap; it cannot affect PR/presubmit CI.Related
Device / Architecture Coverage
Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification.
Testing Summary
workflow_runand the nightly schedule both use the default-branch copy of the workflow. Plan: watch the first nightly after merge and confirm a failed leg re-runs and is recorded as a new attempt.Testing Checklist
python -c "import yaml; yaml.safe_load(...)"- Status: PassedFlags / Guardrails
Retry cap
run_attempt < 4(original run plus up to 3 auto-retries). Only fires whenconclusion == 'failure';cancelled(concurrency/superseded) andstartup_failureare excluded. Requiresactions: writeonGITHUB_TOKEN, scoped to this workflow.Adjacent Tests Considered
No product or test code changes. The existing
RETRY_THIS_STEPsame-runner step retries intest_component.ymlare complementary (they catch transient in-step flakes) and are left unchanged.Technical Changes
.github/workflows/auto-retry-failed.yml:workflow_runtrigger onTheRock CI Nightlycompletion; a single job that runsgh run rerun <id> --failedwhen the run failed and is still under the attempt cap.