[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph - #53155
Merged
Merged
Conversation
Rohan138
requested review from
ProExpertProg,
WoosukKwon,
houseroad,
mgoin,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
August 20, 2026 18:59
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Collaborator
Author
AndreasKaratzas
approved these changes
Aug 24, 2026
Member
|
/ci run |
|
✅ @Rohan138, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #85387 for commit |
Contributor
|
Should revisit the perf numbers here after #53712, and maybe also let DSV4 go back on MRV2 |
Member
|
/ci run |
Member
|
/amd-ci run |
|
✅ Triggered Buildkite CI #85730 for commit |
|
✅ Triggered Buildkite AMD CI #12389 for commit |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
… cudagraph vLLM vllm-project#52861 routed the DSA architectures onto the non-compiled V2 model runner / breakable-cudagraph path, but GlmMoeDsaForCausalLM was left out of the ROCm carve-out, flipping GLM-5.2 to MRV2 + breakable cudagraphs and regressing batch-1 decode TPOT by ~30-37% on gfx950 (FP8 and MXFP4). Add GlmMoeDsaForCausalLM to ROCM_DEFAULT_MRV1_ARCHITECTURES so GLM-5.2 stays on the compiled MRV1 path, and default breakable cudagraphs off entirely on ROCm (they regress performance today); VLLM_USE_BREAKABLE_CUDAGRAPH=1 still forces them on. Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rohan138
force-pushed
the
rocm-glm52-mrv1-exclude
branch
from
August 28, 2026 19:33
ee00634 to
d8602dd
Compare
Collaborator
Author
|
/ci run |
Collaborator
Author
|
/amd-ci retry |
|
✅ Triggered Buildkite CI #86044 for commit |
|
✅ No failed, timed-out, or expired jobs need retrying: https://buildkite.com/vllm/amd-ci/builds/12427 |
Collaborator
Author
|
/ci retry |
Collaborator
Author
|
/amd-ci retry |
|
✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/86044 |
|
✅ No failed, timed-out, or expired jobs need retrying: https://buildkite.com/vllm/amd-ci/builds/12478 |
Member
|
/ci run |
|
✅ Triggered Buildkite CI #86417 for commit |
Collaborator
Author
|
/ci run |
|
✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/86417 |
Collaborator
Author
|
/amd-ci run |
|
✅ Triggered Buildkite AMD CI #12493 for commit |
am-cohere
pushed a commit
to am-cohere/vllm
that referenced
this pull request
Sep 1, 2026
…llm-project#53155) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
mylibrar
pushed a commit
to tanyuqian/vllm
that referenced
this pull request
Sep 3, 2026
…llm-project#53155) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
sheralskumar
pushed a commit
to sheralskumar/vllm
that referenced
this pull request
Sep 8, 2026
…llm-project#53155) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
D-G-Dimitrov
pushed a commit
to D-G-Dimitrov/vllm
that referenced
this pull request
Sep 9, 2026
…llm-project#53155) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit ce2e343)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph
Purpose
#52861 routed the DSA models to the V2 model runner (MRV2) and breakable CUDA
graphs. On ROCm it excluded
DeepseekV32ForCausalLM/DeepseekV4ForCausalLMfrom MRV2 (the
TODO(rocm)notes these are "unsupported by MRV2 or slower withMRV2 on AMD GPUs") but missed
GlmMoeDsaForCausalLM. GLM-5.2 (both FP8 andMXFP4) thus became the only DSA model defaulting to MRV2 + breakable cudagraph on
ROCm, which regresses decode latency on gfx950 (MI355X) by ~30–37% TPOT at
batch 1 in nightly perf-eval.
This PR, ROCm-only:
GlmMoeDsaForCausalLMto the MRV2 exclusion so GLM-5.2 stays on thecompiled MRV1 path like the other DSA models;
currently regresses performance. Opt-in via
VLLM_USE_BREAKABLE_CUDAGRAPH=1is unchanged.
No effect on non-ROCm platforms.
Not a duplicate
gh pr list --repo vllm-project/vllm --state open --search "52861 in:body"and asearch for the GLM/ROCm model-runner area returned no open PR addressing this;
the only related open PR (#41834) is NVIDIA SM12x DeepSeek-V4 enablement.
Perf results
GLM-5.2-MXFP4, gfx950 (MI355X), TP8,
--load-format dummy, in/out 1024,vllm bench serve --dataset-name random --max-concurrency 1 --num-prompts 10 --ignore-eos(warmup pass discarded), median TPOT:VLLM_USE_BREAKABLE_CUDAGRAPH=0only (still MRV2)Decomposition: MRV2 accounts for ~3.0 ms of the ~4.3 ms regression and breakable
cudagraph for ~1.2 ms, so both changes are needed to fully recover; disabling
breakable alone (env workaround) leaves ~23% on the table.
Testing
Using V2 Model Runner/Breakable CUDA graph enabledstartup logs are absent with this PR.tests/test_config.pycovers the ROCm MRV2 / breakable defaults; thesubmitter should run
pytest tests/test_config.py -k "rocm or dsa or breakable"in a built env (not runnable in the perf container used here).
AI assistance
This change was prepared with AI assistance (Claude) and reviewed by the
submitter.