Repository navigation
ci: also auto-retry TheRock Multi-Arch CI on failed/flaky jobs - #12511
Merged
Merged
Conversation
`TheRock Multi-Arch CI` became the default per-PR presubmit in #11542, but it was never added to the auto-retry watcher's `workflows:` list, so the presubmit auto-retry added in #10763 stopped applying. Every failed Multi-Arch CI run sits at attempt 1 until a human clicks "Re-run failed jobs." Add it to the watch list with the same per-PR cap as `TheRock CI`: 1 auto-retry, 2 total attempts. The nightlies keep their cap of 4. The attempt cap is now driven by two explicit workflow-name lists instead of `== 'TheRock CI'` / `!= 'TheRock CI'`, whose fallback branch would otherwise have given the new workflow the looser nightly cap. A watched workflow missing from both lists now gets no auto-retry, which fails safe on runner spend. Closes #12510 Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
github actions
project: none
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
No unresolved review issues remain.
Review effort: Lite
Findings: None
What changed in this PR
Updates CI auto-retry handling to cover TheRock Multi-Arch CI with the intended retry limits.
Changes:
- Adds Multi-Arch CI to watched workflows.
- Separates per-PR and nightly retry caps.
- Updates explanatory comments.
| File | Description |
|---|---|
.github/workflows/auto-retry-failed.yml |
Extends retry monitoring and applies correct retry caps. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
davidd-amd
approved these changes
Sep 28, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TheRock Multi-Arch CIbecame the default per-PR presubmit in #11542, but it was never added to the auto-retry watcher'sworkflows:list, so the presubmit auto-retry added in #10763 silently stopped applying. Every failed Multi-Arch CI run today sits at attempt 1 until a human clicks "Re-run failed jobs." This adds it to the watch list with the same per-PR capTheRock CIhad: 1 auto-retry, 2 total attempts.Risk Assessment
Risk 2. One-file change to an isolated CI orchestration workflow; no product code, reusable-workflow interface, or build/test lane is touched. It only acts on runs that already concluded
failure, and the per-workflow attempt cap bounds worst-case runner usage. This is the same mechanism already proven on the nightlies (#9174, #9658) and onTheRock CI(#10763), applied to one more workflow name.Related
ISSUE ID: #12510
Device / Architecture Coverage
Architecture-independent. This is CI orchestration only (it calls the GitHub API to re-run jobs); it runs no device code and needs no per-arch verification.
Testing Summary
TheRock Multi-Arch CIruns are all atattempt=1, whileTheRock Multi-Arch Nightly CIfailures reachattempt=4under the existing watcher.ROCm/TheRock. Run 35893574070 was manually re-run this way: attempt 2 re-executed only the failed test jobs (started 2026-09-24T00:22Z) while the setup jobs kept their attempt-1 timestamps (2026-09-23T17:52Z), and the re-run test jobs passed.github.run_id, notrun_attempt, so test jobs re-run roughly 6.5 hours later still found attempt 1's build output.actionlinttype-checks theif:expression. I verified the linter is actually reading it by breakingcontainsintocontainzon a scratch copy and watching it fail.workflow_runalways uses the copy of this file ondevelop. Plan: watch the first Multi-Arch CI failure after merge and confirm it gets exactly one auto-retry.Testing Checklist
actionlint .github/workflows/auto-retry-failed.yml- Status: Passedpre-commit run --files .github/workflows/auto-retry-failed.yml- Status: PassedTheRock Multi-Arch CIrun gets exactly 1 auto-retry - Status: PendingFlags / Guardrails
Per-workflow attempt cap via
run_attempt: the per-PR workflows (TheRock CI,TheRock Multi-Arch CI) getrun_attempt < 2(1 auto-retry, 2 total attempts); the two nightlies keeprun_attempt < 4(unchanged). Only fires onconclusion == 'failure';cancelled(concurrency / superseded runs) andstartup_failureare excluded. No new permissions; reuses the existingactions: writegrant.Adjacent Tests Considered
No product or test code changes. Checked that Multi-Arch CI's
cancel-in-progress: trueconcurrency interacts safely with a queued retry: a new push cancels a stale retry, which is the behavior we want. Also checked the two ASAN workflows (TheRock Multi-Arch CI ASANandTheRock Multi-Arch CI ASAN Nightly), which are likewise unwatched; they are left out of this PR to keep the change on the default presubmit path, and are noted as follow-up in the issue.Risk Acceptance / Waivers
W-BUILD: CI orchestration only, no product behavior change; mirrors the already-proven pattern from #9174, #9658, and #10763.Technical Changes
"TheRock Multi-Arch CI"to theworkflow_run.workflowslist in.github/workflows/auto-retry-failed.yml.== 'TheRock CI'/!= 'TheRock CI'attempt-cap branch with two explicitcontains(fromJSON(...))name lists, one per cap. The old fallback branch gave the nightly cap of 4 to any watched workflow that was not literallyTheRock CI, so adding Multi-Arch CI without this rework would have granted presubmit 3 auto-retries. A watched workflow in neither list now gets no auto-retry at all, which fails safe on runner spend.TheRock CIas the only per-PR workflow.