Repository navigation
ci: also auto-retry per-PR TheRock CI on failed/flaky jobs - #10763
Merged
Merged
Conversation
Extends the existing auto-retry watcher (added in #9174, #9658) to also cover the per-PR `TheRock CI` workflow. Unlike the nightlies (4 total attempts), it gets only 1 auto-retry (2 total attempts) so a genuine regression still surfaces quickly and flaky infra doesn't multiply runner usage on every PR push. Cancelled runs/jobs remain excluded, unchanged from the existing behavior. Closes #10762 Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
github actions
project: none
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
tony-davis
marked this pull request as ready for review
August 13, 2026 20:49
tony-davis
requested review from
geomin12 and
talumbau
and
a lite review from Copilot
August 13, 2026 20:49
talumbau
approved these changes
Aug 13, 2026
talumbau
left a comment
talumbau
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Thanks @tony-davis !
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the existing CI “auto-rerun failed jobs” watcher workflow to also cover the per-PR TheRock CI workflow, while keeping retries bounded to avoid masking real regressions and to limit runner usage.
Changes:
- Add
"TheRock CI"to theworkflow_run.workflowswatch list. - Make the rerun attempt cap conditional:
< 2forTheRock CI,< 4for the nightly workflows. - Expand comments to document the per-workflow retry policy and clarify rerun behavior around cancelled jobs.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
geomin12
approved these changes
Aug 13, 2026
geomin12
left a comment
geomin12
left a comment
Collaborator
There was a problem hiding this comment.
good temporary measure until we onboard to prolense
let's give it a try and see how it does
This was referenced Sep 21, 2026
davidd-amd
pushed a commit
that referenced
this pull request
Sep 28, 2026
## Summary `TheRock Multi-Arch CI` became the default per-PR presubmit in #11542, but it was never added to the auto-retry watcher's `workflows:` list, so the presubmit auto-retry added in #10763 silently stopped applying. Every failed Multi-Arch CI run today sits at attempt 1 until a human clicks "Re-run failed jobs." This adds it to the watch list with the same per-PR cap `TheRock CI` had: 1 auto-retry, 2 total attempts. ## Risk Assessment Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded `failure`, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658) and on `TheRock CI` (#10763), applied to one more workflow name. ## Related ISSUE ID: #12510 - Work tracking: Closes #12510 - Related PRs: #11542 (made Multi-Arch CI the default, which opened this gap), #9174, #9658, #10763 (the watcher's predecessors) ## Device / Architecture Coverage Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification. ## Testing Summary - Confirmed the gap is real: the last 15 failed `TheRock Multi-Arch CI` runs are all at `attempt=1`, while `TheRock Multi-Arch Nightly CI` failures reach `attempt=4` under the existing watcher. - Confirmed "re-run failed jobs" actually works on a Multi-Arch CI run, which is not obvious given its jobs come from reusable workflows in `ROCm/TheRock`. Run 35893574070 was manually re-run this way: attempt 2 re-executed only the failed test jobs (started 2026-09-24T00:22Z) while the setup jobs kept their attempt-1 timestamps (2026-09-23T17:52Z), and the re-run test jobs passed. - That run also confirms artifact addressing survives a retry: TheRock's build and test jobs key S3 artifacts on `github.run_id`, not `run_attempt`, so test jobs re-run roughly 6.5 hours later still found attempt 1's build output. - `actionlint` type-checks the `if:` expression. I verified the linter is actually reading it by breaking `contains` into `containz` on a scratch copy and watching it fail. - Functional behavior can only be exercised on the default branch, because `workflow_run` always uses the copy of this file on `develop`. Plan: watch the first Multi-Arch CI failure after merge and confirm it gets exactly one auto-retry. ## Testing Checklist - [x] actionlint - `actionlint .github/workflows/auto-retry-failed.yml` - Status: Passed - [x] pre-commit (formatting/yaml) - `pre-commit run --files .github/workflows/auto-retry-failed.yml` - Status: Passed - [ ] PR CI - GitHub PR checks - Status: Pending - [ ] Post-merge observation - confirm a failed `TheRock Multi-Arch CI` run gets exactly 1 auto-retry - Status: Pending ## Flags / Guardrails Per-workflow attempt cap via `run_attempt`: the per-PR workflows (`TheRock CI`, `TheRock Multi-Arch CI`) get `run_attempt < 2` (1 auto-retry, 2 total attempts); the two nightlies keep `run_attempt < 4` (unchanged). Only fires on `conclusion == 'failure'`; `cancelled` (concurrency / superseded runs) and `startup_failure` are excluded. No new permissions; reuses the existing `actions: write` grant. ## Adjacent Tests Considered No product or test code changes. Checked that Multi-Arch CI's `cancel-in-progress: true` concurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the behavior we want. Also checked the two ASAN workflows (`TheRock Multi-Arch CI ASAN` and `TheRock Multi-Arch CI ASAN Nightly`), which are likewise unwatched; they are left out of this PR to keep the change on the default presubmit path, and are noted as follow-up in the issue. ## Risk Acceptance / Waivers `W-BUILD`: CI orchestration only, no product behavior change; mirrors the already-proven pattern from #9174, #9658, and #10763. ## Technical Changes - Add `"TheRock Multi-Arch CI"` to the `workflow_run.workflows` list in `.github/workflows/auto-retry-failed.yml`. - Replace the `== 'TheRock CI'` / `!= 'TheRock CI'` attempt-cap branch with two explicit `contains(fromJSON(...))` name lists, one per cap. The old fallback branch gave the nightly cap of 4 to any watched workflow that was not literally `TheRock CI`, so adding Multi-Arch CI without this rework would have granted presubmit 3 auto-retries. A watched workflow in neither list now gets no auto-retry at all, which fails safe on runner spend. - Update the header comment, which named `TheRock CI` as the only per-PR workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the existing auto-retry watcher (added in #9174, #9658) to also cover the per-PR
TheRock CIworkflow. Unlike the nightlies (4 total attempts), it gets only 1 auto-retry (2 total attempts) so a genuine regression still surfaces quickly and flaky infra doesn't multiply runner usage on every PR push.Risk Assessment
Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded
failure, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658), just applied to a second, lower-cap workflow name.Related
TheRock CIto non-draft PRs -- complementary, reduces overall run volume)Device / Architecture Coverage
Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification.
Testing Summary
python -c "import yaml; yaml.safe_load(open('.github/workflows/auto-retry-failed.yml'))").pre-commit run --files .github/workflows/auto-retry-failed.ymlpassed locally (trailing-whitespace, end-of-file-fixer, check-yaml).workflow_runalways uses the copy of this file ondevelop. Plan: watch the firstTheRock CIfailure after merge and confirm it gets exactly one auto-retry.Testing Checklist
python -c "import yaml; yaml.safe_load(...)"- Status: Passedpre-commit run --files .github/workflows/auto-retry-failed.yml- Status: PassedTheRock CIrun gets exactly 1 auto-retry - Status: PendingFlags / Guardrails
Per-workflow attempt cap via
run_attempt:TheRock CIgetsrun_attempt < 2(1 auto-retry, 2 total attempts); the two nightly workflows keep the existingrun_attempt < 4(unchanged). Only fires onconclusion == 'failure';cancelled(concurrency/superseded runs, or a run with individually cancelled jobs --rerun-failed-jobsnever touches those) andstartup_failureare excluded. No new permissions; reuses the existingactions: writegrant.Adjacent Tests Considered
No product or test code changes. Checked that
TheRock CI'scancel-in-progress: trueconcurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the desired behavior (no point retrying stale code).Risk Acceptance / Waivers
W-BUILD: CI orchestration only, no product behavior change; mirrors the already-proven nightly pattern from #9174/#9658.Technical Changes
"TheRock CI"to theworkflow_run.workflowslist in.github/workflows/auto-retry-failed.yml.if:condition to branch by workflow name:run_attempt < 2forTheRock CI,run_attempt < 4for everything else (nightlies, unchanged).rerun-failed-jobsalready excludes individually cancelled jobs.