[AMD][CI] Swap the AMD PR gate to ROCm 7.2 and demote ROCm 7.0 to a daily shadow - #34204
Merged
Merged
Conversation
Keep the existing workflow layout while aligning the ROCm 7.2 image and AITER rebuild path on Triton 3.7.
Preserve staged gate behavior and route operational tooling to ROCm 7.2 while enforcing AITER's pinned Triton stack.
Accept AMD's PEP 440 local-version suffix while still enforcing the pinned Triton revision.
Rely on the pinned installer and source checkout to avoid brittle failures from package local-version metadata.
Run all ROCm 7.2 jobs, including DeepSeek-V4, on the shared 12-hour cadence.
bingxche
force-pushed
the
amd_upgrade_pr_test
branch
from
August 10, 2026 15:33
10f1755 to
6fd0b11
Compare
Drop TRITON_REPO / TRITON_COMMIT: nothing reads them since the source build was replaced by AITER's pinned wheel. Skip pip's cache so the 430MB Triton wheel does not land in the image layer. Record why the install must follow the torch-ROCm metadata patch: the base Torch wheel pins triton==3.5.1, so installing Triton 3.7 before that patch makes any later pip install resolve torch and pull PyPI's CUDA triton, breaking the ROCm stack.
The 7.2 workflow had drifted from pr-test-amd.yml while it was a non-gating side workflow, so promoting it to the PR gate would have quietly changed how PRs are tested. Regenerate it from pr-test-amd.yml so every remaining difference is deliberate: the -rocm720 job ids, --rocm-version rocm720 on the container, rocm_version passed to the extra suite, and the two DeepSeek-V4 jobs. That restores runner_arch (mi300/mi325 dispatch, and with it the per-arch timeouts and job display names), picks up #34324's 8-GPU stage-C max-parallel 2, narrows the change filters back so a CI-script edit no longer lights up the diffusion suites, and drops both the extra fast-fail guards and a redundant amdsmi install. Demote pr-test-amd.yml to a rollback shadow that chases coverage instead of latency: it now fires once a day with nightly-test-amd.yml and runs every matrix shard at once, since nothing gates on it any more.
# Conflicts: # .github/workflows/pr-test-amd.yml
The image change and the gate swap are independent, and the image one needs a nightly build to prove it before anything depends on it. Split it into #34364 so this PR is only the gate swap and the two can land in either order.
bingxche
force-pushed
the
amd_upgrade_pr_test
branch
from
August 11, 2026 05:13
7a0d816 to
4668da4
Compare
The log claimed AITER_USE_SYSTEM_TRITON=0 on ROCm 7.2 while the build below always passes 1, which would send anyone debugging a Triton mismatch down the wrong path. The flag only decides whether to run AITER's Triton installer before the rebuild, so name and print it that way.
Every other GPU job in this workflow puts its runner label in the job name, so a flaky machine is identifiable straight from the checks list. The two dsv4 jobs pinned their pool directly in runs-on and showed nothing. Declare it in a single-entry matrix like the sibling mi35x jobs do, which both keeps the label in one place and gets it into the generated name.
The two dsv4 jobs were the only mi35x jobs here whose id said nothing about the arch, so they sorted away from their siblings in the checks list and in target_stage_select. Give them the -amd-mi35x-rocm720 tail the other mi35x jobs in this workflow use.
On a scheduled run the job swallowed the test's exit code and exited 0, so a failing accuracy test showed up green in the checks list and only in the step summary. Report the captured code like the sibling flash job does. run_suite's --continue-on-error, which the flash job passes, only stops it from breaking out of the file loop early; it still returns non-zero if any file failed. So neither job hides a failure now.
They only ran on schedule, so DeepSeek-V4 regressions landed and were found a day later. The four most recent scheduled runs took 17-19 minutes each, which is cheap enough to cover on every PR, so gate them like the other 8-GPU MI35x job: stage-B has to finish first, and the same change filters apply. Timeouts drop from 300/480 to 60 minutes, in line with the measured runtime, and the flash job's per-file budget comes down to match its job timeout rather than sitting above it. Its continue-on-error now reads the check-changes output every other job uses, instead of deriving it separately.
release-branch-cut already moved its PR suite to ROCm 7.2; its nightly call was still on 7.0, so a release branch got validated against the stack we are moving off. Both nightly workflows take the same workflow_call inputs, so this is a one-line swap. Name the two ROCm 7.0 workflows after their ROCm version, matching the 7.2 siblings that already carry it. Without that, the sidebar shows an unqualified "PR Test (AMD)" next to "PR Test ROCm 7.2 (AMD)" and it is not obvious which stack the unqualified one covers.
The PR gate and its reports moved to ROCm 7.2 but the nightly side did not, so the version-bump bot still validated a bump against the stack we are moving off, and the monitor still filed 7.0 results under the plain "Nightly" heading while 7.2 sat in a side report. Swap the nightly pair the same way the PR pair already was: the default report reads the 7.2 nightly, and the 7.0 one becomes an explicitly named shadow. The failure analysis follows, since it is meant to summarise whatever gates PRs. Left alone: amd-aiter-scout calls both stacks on purpose, and the snapshot dump enumerates all five workflows rather than picking a default.
bingxche
marked this pull request as ready for review
August 11, 2026 08:30
bingxche
requested review from
Fridge003,
HaiShaw,
Kangyan-Zhou,
bingxche and
ispobock
as code owners
August 11, 2026 08:30
Moving them into stage-C changed their gating but not their names, so they were the only jobs sequenced behind wait-for-stage-b without a tier prefix. Every other staged job carries one regardless of which section it sits in - the disaggregation job is named stage-b- for the same reason. No "test" segment: for the other stage-C jobs that string doubles as the run_suite --suite name, and these two run the nightly dsv4 suites instead.
bingxche
changed the title
bingxche
changed the title
HaiShaw
approved these changes
Aug 12, 2026
This was referenced Aug 12, 2026
This was referenced Aug 14, 2026
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
…aily shadow (sgl-project#34204) Co-authored-by: yctseng0211 <yctseng@smci355-ccs-aus-m12-17.cs-aus.dcgpu> Co-authored-by: Chen <bingxche@amd.com>
5 tasks
yctseng0211
added a commit
that referenced
this pull request
Aug 28, 2026
These four blocks came in with the gfx1250 bring-up line, not with the ROCm 10 work, and main itself dropped the equivalent code in #34204. With the gfx1250-rocm7_14 flavor gone they have no remaining consumer, and on a ROCm 10 image every one of them misreads the environment. The HIP case is the sharpest: its arms are named after ROCm generations but it switches on torch.version.hip, and the two only agreed by coincidence on the apt ROCm 7.x images. A ROCm 10.0.0 GA image reports HIP 7.15.26333, so it matched no arm and the installer exited 1 before doing anything. CT_EXTRA and the AITER build-method fork both key off $ROCM_HOME/.info/version, which the apt ROCm bases create and the pip SDK does not. On rocm1000 that resolves to "unknown", which would have selected the compressed-tensors pin that requires torch<2.11 against a torch 2.11 image. What stays: gfx1250 in the image-arch pattern, gfx1250 in the AITER stage lookup, and the MORI_COMMIT extraction guard.
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
…aily shadow (sgl-project#34204) Co-authored-by: yctseng0211 <yctseng@smci355-ccs-aus-m12-17.cs-aus.dcgpu> Co-authored-by: Chen <bingxche@amd.com>
5 tasks
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
…aily shadow (sgl-project#34204) Co-authored-by: yctseng0211 <yctseng@smci355-ccs-aus-m12-17.cs-aus.dcgpu> Co-authored-by: Chen <bingxche@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ROCm 7.2 is where AMD is going, but the PR gate, the nightly, release cuts and the CI reports all still default to ROCm 7.0. Make 7.2 the default everywhere and keep 7.0 as a daily rollback shadow.
Modifications
pr-test-amd-rocm720.ymlbecomes the PR gate. It had drifted while it was a non-gating side workflow, so it is regenerated frompr-test-amd.yml: normalising the-rocm720job ids,--rocm-version rocm720, the extra suite'srocm_versionand the two DeepSeek-V4 jobs leaves a 9-line delta. That restoresrunner_arch(and with it the per-arch timeouts and job display names), picks up [AMD][CI] Run MI300 8-GPU stage-C shards two at a time #34324's stage-Cmax-parallel: 2, and narrows the change filters back so a CI-script edit no longer lights up the diffusion suites.pr-test-amd.ymlbecomes a daily ROCm 7.0 shadow. Nopull_requesttrigger, fires alongsidenightly-test-amd.yml, and everymax-parallelcap removed — nothing gates on it, so it chases coverage over latency.exit 0-ing on scheduled runs, which hid failures.release-branch-cutandbot-bump-sglang-versioncall the 7.2 nightly;amd-ci-job-monitorandci_failures_analysisreport 7.2 under the default heading with 7.0 as a named shadow; both ROCm 7.0 workflows are renamed to say which stack they cover.amd-aiter-scoutstill calls both on purpose.linux-mi35x-gpu-8, taking its peak concurrency from 3 to 5.CI States
Latest PR Test (Base): ❌ Run #31477440566
Latest PR Test (Extra): ❌ Run #31477440310