Skip to content

[AMD][CI] Merge the four MI35x DeepSeek-V3.2 nightly jobs into two to save runtime - #36736

Merged
HaiShaw merged 3 commits into
mainfrom
cursor/amd-nightly-merge-mi35x-v32-jobs-ef39
Aug 27, 2026
Merged

HaiShaw merged 3 commits into
mainfrom
cursor/amd-nightly-merge-mi35x-v32-jobs-ef39

Conversation

@michaelzhang-ai

@michaelzhang-ai michaelzhang-ai commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Motivation

MI35x DeepSeek-V3.2 coverage is split across four jobs — accuracy and perf, basic and MTP — while the MI30x V3.2 jobs in the same workflow do the same work in two, each running accuracy then perf. Both steps in a pair serve the same weights, so the split buys a second container setup and a second cold weight load per config and nothing else.

Modifications

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720      + nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720
  -> nightly-8-gpu-mi35x-deepseek-v32-rocm720
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720  + nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720
  -> nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720

New names follow the MI30x convention with an mi35x infix. Suites, launch flags, step timeouts and --timeout-per-file are carried over verbatim; job_select and check-all-jobs are updated to the new names. 45 jobs -> 43. No test file or suite touched.

The two merged jobs end up with exactly the gating shape of the MI30x V3.2 pair in this same workflow — accuracy blocking, perf non-blocking, no if: guard — so MI35x and MI30x V3.2 now behave identically.

  • Perf is skipped when accuracy fails, by default step behaviour. The split jobs ran the two independently, so a failing night still produced a perf table; benchmarking a configuration that just missed its GSM8K floor is not worth ~27 min of an 8-GPU node.

Accuracy Tests

No test, suite, flag, threshold or timeout changes.

Both merged jobs passed in run 32907097246 (rocm720, continue_on_error=false), each running its accuracy step and then its perf step. The dispatch resolved both new job names with no old accuracy-/perf- name present, checking the renames against real Actions resolution. That run predates #36396, but these two job bodies are byte-identical to what it executed.

pre-commit passes, including check for duplicate workflow job names and validate registered test CI registries. Every job body diffed against main: only the two merged jobs and check-all-jobs differ, so #36396's additions and removals are preserved.

Speed Tests and Profiling

Four split jobs from the 8/24 and 8/26 scheduled runs versus the two merged jobs from run 32907097246. All rocm720, linux-mi35x-gpu-8, GPU-hours = wall time x 8.

jobs total setup+install test GPU-h
split (mean of 2 nights) 4 134.5 min 26.3 108.0 17.93
merged 2 114.0 min 12.3 101.6 15.20
delta -2 -20.5 min (-15%) -14.0 -6.4 -2.73

Across both image flavors that is ~5.5 GPU-h/night, and 8 scheduled MI35x 8-GPU job runs become 4 — likely the bigger win given queue depths on that pool. Of the 20.5 min, the 14.0 min of eliminated setup is mechanical and reliable; the 6.4 min of test time is one sample against two and within the baseline's own 4.9 min spread, so the defensible floor is ~3.7 GPU-h/night from setup alone.

On a night where accuracy fails, skipping perf reclaims a further ~27 min of node time per affected config, which the split jobs spent benchmarking a broken configuration.

Checklist


CI States

Latest PR Test (Base): ✅ Run #33122934517
Latest PR Test (Extra): ❌ Run #33122934627
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

MI35x V3.2 coverage was split across four jobs -- accuracy and perf, basic and
MTP -- while the MI30x V3.2 coverage does the same work in two jobs that each
run accuracy then perf. Both steps in a pair serve the same DeepSeek-V3.2
weights, so the split bought a second container setup and a second cold weight
load per config and nothing else.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 + nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720
  -> nightly-8-gpu-mi35x-deepseek-v32-rocm720
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 + nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720
  -> nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720

Suites, launch flags, step timeouts and per-file timeouts are carried over
verbatim; only the job packing changes. Each perf step carries
`if: ${{ !cancelled() }}` so it still runs when accuracy fails, which is what
the separate jobs gave us. Perf stays blocking, since check-all-jobs listed
both perf jobs before this change.

Measured against the four split jobs from the 8/24 and 8/26 scheduled runs:
134.5 -> 114.0 min per image flavor, 17.93 -> 15.20 GPU-h, and 8 scheduled
MI35x 8-GPU job runs per night become 4.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Drop `if: ${{ !cancelled() }}` from both merged jobs' perf steps. Perf numbers
from a configuration that just missed its GSM8K floor are not worth ~27 min of
an 8-GPU MI35x node, so let the default step behaviour skip perf once accuracy
has failed.

This is the one behavioural difference from the four split jobs, where accuracy
and perf ran independently. Both steps still fail the job, so a perf regression
on a passing configuration is caught as before.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
@michaelzhang-ai michaelzhang-ai changed the title [AMD][CI] Merge the four MI35x DeepSeek-V3.2 nightly jobs into two [AMD][CI] Merge the four MI35x DeepSeek-V3.2 nightly jobs into two to save runtime Aug 27, 2026
Add `continue-on-error: true` to both merged jobs' perf steps, so a perf
regression reports in the step summary without failing the nightly. The
accuracy step stays blocking, and that is what check-all-jobs gates on.

The two merged jobs now have exactly the gating shape of the MI30x V3.2 pair
in this same workflow: accuracy blocking, perf non-blocking, and no `if:` guard
so perf is skipped once accuracy has failed. It also matches what
nightly-test-amd.yml states for the four jobs it still has split, where the
perf jobs are excluded from check-all-jobs as "perf failures don't block CI".

This does change ROCm 7.2 behaviour: the standalone perf jobs were listed in
check-all-jobs needs, so a perf failure used to fail the nightly here.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
@HaiShaw
HaiShaw merged commit e283c9f into main Aug 27, 2026
96 of 100 checks passed
@HaiShaw
HaiShaw deleted the cursor/amd-nightly-merge-mi35x-v32-jobs-ef39 branch August 27, 2026 22:40
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 31, 2026
… save runtime (sgl-project#36736)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
nzr-niu pushed a commit to nzr-niu/sglang that referenced this pull request Sep 1, 2026
… save runtime (sgl-project#36736)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants