Repository navigation
ci: also auto-retry the multi-arch nightly on failed/flaky jobs - #9658
Merged
Merged
Conversation
Add "TheRock Multi-Arch Nightly CI" to the auto-retry watcher's workflow_run list so its failed jobs are re-run on fresh runners, same as "TheRock CI Nightly". The rerun job is generic (it acts on the triggering run id), so no other change is needed. That run also concludes as `failure` on a failed leg (its multi_arch_ci_summary job runs workflow_summary.py, which exits non-zero), so the existing conclusion gate applies unchanged. Closes #9173 Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
github actions
project: none
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the existing auto-retry GitHub Actions watcher to also respond to completed failures of the multi-architecture nightly workflow, so that failed/flaky jobs are automatically re-run (up to the existing attempt cap) without manual intervention.
Changes:
- Adds
TheRock Multi-Arch Nightly CIto theon.workflow_run.workflowsallowlist in the auto-retry workflow, alongside the existing nightly workflow.
geomin12
approved these changes
Jul 21, 2026
2 of 4 tasks
tony-davis
added a commit
that referenced
this pull request
Aug 13, 2026
## Summary Extends the existing auto-retry watcher (added in #9174, #9658) to also cover the per-PR `TheRock CI` workflow. Unlike the nightlies (4 total attempts), it gets only 1 auto-retry (2 total attempts) so a genuine regression still surfaces quickly and flaky infra doesn't multiply runner usage on every PR push. ## Risk Assessment Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded `failure`, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658), just applied to a second, lower-cap workflow name. ## Related - Work tracking: #10762 (Closes) - Related PRs: #9174, #9658 (nightly predecessors), #10752 (restricts `TheRock CI` to non-draft PRs -- complementary, reduces overall run volume) ## Device / Architecture Coverage Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification. ## Testing Summary - YAML parses cleanly (`python -c "import yaml; yaml.safe_load(open('.github/workflows/auto-retry-failed.yml'))"`). - `pre-commit run --files .github/workflows/auto-retry-failed.yml` passed locally (trailing-whitespace, end-of-file-fixer, check-yaml). - Functional behavior can only be exercised on the default branch, because `workflow_run` always uses the copy of this file on `develop`. Plan: watch the first `TheRock CI` failure after merge and confirm it gets exactly one auto-retry. ## Testing Checklist - [x] YAML parse - `python -c "import yaml; yaml.safe_load(...)"` - Status: Passed - [x] pre-commit (formatting/yaml) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge observation - confirm a failed `TheRock CI` run gets exactly 1 auto-retry - Status: Pending ## Flags / Guardrails Per-workflow attempt cap via `run_attempt`: `TheRock CI` gets `run_attempt < 2` (1 auto-retry, 2 total attempts); the two nightly workflows keep the existing `run_attempt < 4` (unchanged). Only fires on `conclusion == 'failure'`; `cancelled` (concurrency/superseded runs, or a run with individually cancelled jobs -- `rerun-failed-jobs` never touches those) and `startup_failure` are excluded. No new permissions; reuses the existing `actions: write` grant. ## Adjacent Tests Considered No product or test code changes. Checked that `TheRock CI`'s `cancel-in-progress: true` concurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the desired behavior (no point retrying stale code). ## Risk Acceptance / Waivers `W-BUILD`: CI orchestration only, no product behavior change; mirrors the already-proven nightly pattern from #9174/#9658. ## Technical Changes - Add `"TheRock CI"` to the `workflow_run.workflows` list in `.github/workflows/auto-retry-failed.yml`. - Change the attempt-cap `if:` condition to branch by workflow name: `run_attempt < 2` for `TheRock CI`, `run_attempt < 4` for everything else (nightlies, unchanged). - Update header/job comments to document the per-workflow cap and to note that `rerun-failed-jobs` already excludes individually cancelled jobs. Co-authored-by: Cursor <cursoragent@cursor.com>
2 of 4 tasks
davidd-amd
pushed a commit
that referenced
this pull request
Sep 28, 2026
## Summary `TheRock Multi-Arch CI` became the default per-PR presubmit in #11542, but it was never added to the auto-retry watcher's `workflows:` list, so the presubmit auto-retry added in #10763 silently stopped applying. Every failed Multi-Arch CI run today sits at attempt 1 until a human clicks "Re-run failed jobs." This adds it to the watch list with the same per-PR cap `TheRock CI` had: 1 auto-retry, 2 total attempts. ## Risk Assessment Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded `failure`, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658) and on `TheRock CI` (#10763), applied to one more workflow name. ## Related ISSUE ID: #12510 - Work tracking: Closes #12510 - Related PRs: #11542 (made Multi-Arch CI the default, which opened this gap), #9174, #9658, #10763 (the watcher's predecessors) ## Device / Architecture Coverage Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification. ## Testing Summary - Confirmed the gap is real: the last 15 failed `TheRock Multi-Arch CI` runs are all at `attempt=1`, while `TheRock Multi-Arch Nightly CI` failures reach `attempt=4` under the existing watcher. - Confirmed "re-run failed jobs" actually works on a Multi-Arch CI run, which is not obvious given its jobs come from reusable workflows in `ROCm/TheRock`. Run 35893574070 was manually re-run this way: attempt 2 re-executed only the failed test jobs (started 2026-09-24T00:22Z) while the setup jobs kept their attempt-1 timestamps (2026-09-23T17:52Z), and the re-run test jobs passed. - That run also confirms artifact addressing survives a retry: TheRock's build and test jobs key S3 artifacts on `github.run_id`, not `run_attempt`, so test jobs re-run roughly 6.5 hours later still found attempt 1's build output. - `actionlint` type-checks the `if:` expression. I verified the linter is actually reading it by breaking `contains` into `containz` on a scratch copy and watching it fail. - Functional behavior can only be exercised on the default branch, because `workflow_run` always uses the copy of this file on `develop`. Plan: watch the first Multi-Arch CI failure after merge and confirm it gets exactly one auto-retry. ## Testing Checklist - [x] actionlint - `actionlint .github/workflows/auto-retry-failed.yml` - Status: Passed - [x] pre-commit (formatting/yaml) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge observation - confirm a failed `TheRock Multi-Arch CI` run gets exactly 1 auto-retry - Status: Pending ## Flags / Guardrails Per-workflow attempt cap via `run_attempt`: the per-PR workflows (`TheRock CI`, `TheRock Multi-Arch CI`) get `run_attempt < 2` (1 auto-retry, 2 total attempts); the two nightlies keep `run_attempt < 4` (unchanged). Only fires on `conclusion == 'failure'`; `cancelled` (concurrency / superseded runs) and `startup_failure` are excluded. No new permissions; reuses the existing `actions: write` grant. ## Adjacent Tests Considered No product or test code changes. Checked that Multi-Arch CI's `cancel-in-progress: true` concurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the behavior we want. Also checked the two ASAN workflows (`TheRock Multi-Arch CI ASAN` and `TheRock Multi-Arch CI ASAN Nightly`), which are likewise unwatched; they are left out of this PR to keep the change on the default presubmit path, and are noted as follow-up in the issue. ## Risk Acceptance / Waivers `W-BUILD`: CI orchestration only, no product behavior change; mirrors the already-proven pattern from #9174, #9658, and #10763. ## Technical Changes - Add `"TheRock Multi-Arch CI"` to the `workflow_run.workflows` list in `.github/workflows/auto-retry-failed.yml`. - Replace the `== 'TheRock CI'` / `!= 'TheRock CI'` attempt-cap branch with two explicit `contains(fromJSON(...))` name lists, one per cap. The old fallback branch gave the nightly cap of 4 to any watched workflow that was not literally `TheRock CI`, so adding Multi-Arch CI without this rework would have granted presubmit 3 auto-retries. A watched workflow in neither list now gets no auto-retry at all, which fails safe on runner spend. - Update the header comment, which named `TheRock CI` as the only per-PR workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the existing auto-retry watcher (added in #9174) to also cover
TheRock Multi-Arch Nightly CI. It adds that workflow name to theworkflow_runlist in.github/workflows/auto-retry-failed.yml; the rerun job itself is generic (it re-runs the failed jobs of the triggering run), so no other change is needed. Most nightly failures are infrastructure flakes (no GPU assigned, an overloaded-machine timeout, a network blip during artifact fetch) that clear on a healthy runner. Closes #9173.Risk Assessment
Risk 2. One-line addition to an isolated CI orchestration workflow. It does not touch any build or test lane, reusable workflow, or product code, and only acts on nightly runs that already concluded in
failure. Worst case if it misbehaves is extra nightly runner usage, bounded by therun_attempt < 4cap; it cannot affect PR/presubmit CI.Related
TheRock CI Nightly)Device / Architecture Coverage
Architecture-independent. CI orchestration only (it calls the GitHub API to re-run jobs); runs no device code and needs no per-arch verification.
Testing Summary
failureon a failed leg:multi_arch_ci_summary(if: always()) runsbuild_tools/github_actions/workflow_summary.py, which returns 1 when any needed job failed, so the existingconclusion == 'failure'gate applies unchanged.workflow_runand the nightly schedule both use the default-branch copy). Plan: watch the first multi-arch nightly after merge and confirm a failed leg re-runs as a new attempt.Testing Checklist
python -c "import yaml; yaml.safe_load(...)"- Status: Passedpre-commit run --files .github/workflows/auto-retry-failed.yml- Status: PassedFlags / Guardrails
Unchanged from #9174: retry cap
run_attempt < 4; only fires onconclusion == 'failure';cancelled(concurrency/superseded) andstartup_failureexcluded; requiresactions: writeonGITHUB_TOKEN.Adjacent Tests Considered
No product or test code changes. The
RETRY_THIS_STEPsame-runner step retries remain complementary and unchanged.Technical Changes
.github/workflows/auto-retry-failed.yml: add"TheRock Multi-Arch Nightly CI"to theworkflow_run.workflowslist.