Skip to content

[AMD][CI] Run only the ROCm 7.2.4 image on the nightly schedule - #36217

Draft
michaelzhang-ai wants to merge 3 commits into
mainfrom
cursor/drop-rocm720-leg-from-amd-nightly-8403
Draft

michaelzhang-ai wants to merge 3 commits into
mainfrom
cursor/drop-rocm720-leg-from-amd-nightly-8403

Conversation

@michaelzhang-ai

@michaelzhang-ai michaelzhang-ai commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Motivation

#35603 put both rocm724 and rocm720 on the one 17:30 nightly. Four complete nights ran that way (2026-08-20 … 08-23, 47 job pairs a night) before #36171 changed the image-pull path. This PR makes the schedule run rocm724 only, on cost and redundancy grounds.

The 24-hour overruns on those four nights were not caused by the second leg. They were the shared local registry at 10.44.14.109:5000. On 08-22 a single MI30x job resolved its image at 00:27:35 and finished docker pull from the local registry at 03:33:50 — one successful pull, no fallback, 186 minutes. MI35x jobs never went down that path and held a 4.1 min median throughout, even though their job count doubled too, and the untouched ROCm 7.0 nightly on the same MI30x pools took the same 10x hit without changing its job count at all. With #36171 bypassing the local registry, tonight's run has both legs, an MI30x Setup docker median back at 14.9 min (max 18.8), and 86 of 92 jobs done in about 7.4 h. So both flavors do fit in a night now; the wall-clock argument for touching this workflow is gone.

What remains is cost and redundancy.

Of 188 job pairs over the four nights, 166 reached the same completed verdict, 17 disagreed and 5 involved a cancellation. Only five job-nights had rocm720 fail while rocm724 passed, spread over four distinct jobs, and none reproduced on another night:

job night
nightly-8-gpu-mi35x-deepseek-r1-hicache 08-20
nightly-8-gpu-mi35x-qwen35-triton-dcp 08-20
nightly-8-gpu-minimax-m27 08-22
nightly-accuracy-8-gpu-mi35x 08-22
nightly-4-gpu-mi35x-minimax-m25 08-23

No durable 7.2.0-specific regression appeared. The unique signal ran the other way: nightly-8-gpu-grok1-int4 failed on 724 three nights out of four while 720 stayed green all four, and grok2, minimax-m3 and glm5-mxfp4 each had a 724-only failure. That is expected, since rocm724 is Ubuntu 24.04 / Python 3.12 / torch 2.11 / triton 3.7 against 720's 22.04 / 3.10 / 2.9.1, and it argues for keeping 724 rather than both. Tonight repeats the pattern: 9 failures on 724, 8 on 720.

On cost, tonight's healthy run puts each leg at roughly 205–212 GPU-hours, so the second leg is about 210 GPU-hours a night.

Modifications

  • nightly-test-amd-rocm720.yml: the rocm_version matrix resolves to ["rocm724"] when no input is given, so the schedule and push triggers run one leg. all still expands to both.
  • Same file: the workflow_dispatch default moves from all to rocm724, with rocm720 and all still in the dropdown. The workflow_call default was already rocm724; its stale comment is corrected.
  • amd-ci-job-monitor.yml: rocm_flavors drops to rocm724, which that script is documented to keep in sync with the matrix.

Verified that every trigger resolves as intended (schedule / push["rocm724"], dispatch default → ["rocm724"], dispatch rocm720["rocm720"], dispatch or caller all → both), that no existing caller passes all (amd-aiter-scout, bot-bump-sglang-version and release-branch-cut all take the rocm724 default), and that check_workflow_job_names.py, actionlint and the full pre-commit set pass.

Open question for AMD CI owners

This is a coverage-vs-cost call, not a fix for an outage, so it is reasonable to reject it. rocm720 images are still published nightly by release-docker-amd-rocm720-nightly.yml and still built for stable releases by release-docker-amd.yml, so after this change they have no scheduled coverage. Three ways to go:

  1. merge this and accept that the published 7.2.0 images are untested;
  2. merge this plus a weekly 7.2.0 pass over a smoke subset (needs github.event.schedule in the concurrency group so the two crons do not cancel each other);
  3. drop this PR and keep both legs nightly, now that the registry fix has restored the ~7 h runtime, and revisit if the ~210 GPU-hours a night becomes a problem.

Checklist


CI States

Latest PR Test (Base): ❌ Run #32796016068
Latest PR Test (Extra): ❌ Run #32796015960
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

#35603 put both rocm724 and rocm720 on the 17:30 nightly. Over the four
complete nights that followed (2026-08-20..23, 47 job pairs a night) the
two legs agreed on 171 of 188 pairs, and only two jobs ever failed on
rocm720 with rocm724 green -- each once, neither reproducing on the other
three nights. The unique signal ran the other way: grok1-int4 failed on
rocm724 three nights out of four while rocm720 stayed green.

The second leg was not free. Every scheduled run since the merge was
cancelled at the 24h mark by the next night's schedule, nightly-4-gpu
produced a result in only two of eight leg-nights, and the untouched
ROCm 7.0 nightly sharing the MI30x pools went the same way.

Keep the schedule on rocm724, which is what the PR gate already defaults
to, and leave rocm720 one dispatch away: pick it from the dropdown, or
choose "all" for both.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Keep ROCm 7.2.0 coverage, but run it in a separate reusable-workflow
call twelve hours away from the primary ROCm 7.2.4 schedule. Include the
image flavor in the concurrency key so the two schedules cannot cancel
each other.

Teach the AMD job monitor and failure analyzer about the staggered
workflow so both flavors remain visible in CI reports.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
@cursor cursor Bot changed the title [AMD][CI] Run only the ROCm 7.2.4 image on the nightly schedule [AMD][CI] Stagger ROCm 7.2.0 and 7.2.4 nightly runs Aug 25, 2026
@michaelzhang-ai michaelzhang-ai changed the title [AMD][CI] Stagger ROCm 7.2.0 and 7.2.4 nightly runs [AMD][CI] Run only the ROCm 7.2.4 image on the nightly schedule Aug 25, 2026
@michaelzhang-ai
michaelzhang-ai marked this pull request as draft August 25, 2026 01:04
@Jiminator Jiminator closed this Sep 14, 2026
@Jiminator
Jiminator deleted the cursor/drop-rocm720-leg-from-amd-nightly-8403 branch September 14, 2026 04:44
@alexnails
alexnails restored the cursor/drop-rocm720-leg-from-amd-nightly-8403 branch September 14, 2026 05:40
@hnyls2002 hnyls2002 reopened this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants