[AMD][CI] Merge the four MI35x DeepSeek-V3.2 nightly jobs into two to save runtime - #36736
Merged
Merged
Conversation
MI35x V3.2 coverage was split across four jobs -- accuracy and perf, basic and
MTP -- while the MI30x V3.2 coverage does the same work in two jobs that each
run accuracy then perf. Both steps in a pair serve the same DeepSeek-V3.2
weights, so the split bought a second container setup and a second cold weight
load per config and nothing else.
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 + nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720
-> nightly-8-gpu-mi35x-deepseek-v32-rocm720
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 + nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720
-> nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720
Suites, launch flags, step timeouts and per-file timeouts are carried over
verbatim; only the job packing changes. Each perf step carries
`if: ${{ !cancelled() }}` so it still runs when accuracy fails, which is what
the separate jobs gave us. Perf stays blocking, since check-all-jobs listed
both perf jobs before this change.
Measured against the four split jobs from the 8/24 and 8/26 scheduled runs:
134.5 -> 114.0 min per image flavor, 17.93 -> 15.20 GPU-h, and 8 scheduled
MI35x 8-GPU job runs per night become 4.
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
5 tasks
michaelzhang-ai
marked this pull request as ready for review
August 27, 2026 22:23
michaelzhang-ai
requested review from
Fridge003,
HaiShaw,
Kangyan-Zhou,
bingxche,
ispobock and
merrymercy
as code owners
August 27, 2026 22:23
Drop `if: ${{ !cancelled() }}` from both merged jobs' perf steps. Perf numbers
from a configuration that just missed its GSM8K floor are not worth ~27 min of
an 8-GPU MI35x node, so let the default step behaviour skip perf once accuracy
has failed.
This is the one behavioural difference from the four split jobs, where accuracy
and perf ran independently. Both steps still fail the job, so a perf regression
on a passing configuration is caught as before.
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Add `continue-on-error: true` to both merged jobs' perf steps, so a perf regression reports in the step summary without failing the nightly. The accuracy step stays blocking, and that is what check-all-jobs gates on. The two merged jobs now have exactly the gating shape of the MI30x V3.2 pair in this same workflow: accuracy blocking, perf non-blocking, and no `if:` guard so perf is skipped once accuracy has failed. It also matches what nightly-test-amd.yml states for the four jobs it still has split, where the perf jobs are excluded from check-all-jobs as "perf failures don't block CI". This does change ROCm 7.2 behaviour: the standalone perf jobs were listed in check-all-jobs needs, so a perf failure used to fail the nightly here. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
HaiShaw
approved these changes
Aug 27, 2026
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 31, 2026
… save runtime (sgl-project#36736) Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: quitenode <quitenode@users.noreply.github.com>
nzr-niu
pushed a commit
to nzr-niu/sglang
that referenced
this pull request
Sep 1, 2026
… save runtime (sgl-project#36736) Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: quitenode <quitenode@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
MI35x DeepSeek-V3.2 coverage is split across four jobs — accuracy and perf, basic and MTP — while the MI30x V3.2 jobs in the same workflow do the same work in two, each running accuracy then perf. Both steps in a pair serve the same weights, so the split buys a second container setup and a second cold weight load per config and nothing else.
Modifications
New names follow the MI30x convention with an
mi35xinfix. Suites, launch flags, step timeouts and--timeout-per-fileare carried over verbatim;job_selectandcheck-all-jobsare updated to the new names. 45 jobs -> 43. No test file or suite touched.The two merged jobs end up with exactly the gating shape of the MI30x V3.2 pair in this same workflow — accuracy blocking, perf non-blocking, no
if:guard — so MI35x and MI30x V3.2 now behave identically.Accuracy Tests
No test, suite, flag, threshold or timeout changes.
Both merged jobs passed in run 32907097246 (
rocm720,continue_on_error=false), each running its accuracy step and then its perf step. The dispatch resolved both new job names with no oldaccuracy-/perf-name present, checking the renames against real Actions resolution. That run predates #36396, but these two job bodies are byte-identical to what it executed.pre-commitpasses, includingcheck for duplicate workflow job namesandvalidate registered test CI registries. Every job body diffed againstmain: only the two merged jobs andcheck-all-jobsdiffer, so #36396's additions and removals are preserved.Speed Tests and Profiling
Four split jobs from the 8/24 and 8/26 scheduled runs versus the two merged jobs from run 32907097246. All
rocm720,linux-mi35x-gpu-8, GPU-hours = wall time x 8.Across both image flavors that is ~5.5 GPU-h/night, and 8 scheduled MI35x 8-GPU job runs become 4 — likely the bigger win given queue depths on that pool. Of the 20.5 min, the 14.0 min of eliminated setup is mechanical and reliable; the 6.4 min of test time is one sample against two and within the baseline's own 4.9 min spread, so the defensible floor is ~3.7 GPU-h/night from setup alone.
On a night where accuracy fails, skipping perf reclaims a further ~27 min of node time per affected config, which the split jobs spent benchmarking a broken configuration.
Checklist
CI States
Latest PR Test (Base): ✅ Run #33122934517
Latest PR Test (Extra): ❌ Run #33122934627
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.