Repository navigation
[AMD][MI355X] DSv4.1 Flash: enable the vLLM mono decode kernels (image TBD) / [AMD][MI355X] DSv4.1 Flash：启用 vLLM mono decode 内核（镜像待定） - #3690
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
填写 perf-changelog 中 #3690 的 PR 链接。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Native DeepSeek-V4.1 a4w4 support from vllm-project/vllm#58819. | ||
| VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4: '1' |
There was a problem hiding this comment.
🔴 Setting VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 (agentic.yaml:66) drops MoE activation precision from a8w4 to a4w4, and the recipe's own changelog says this reaches the "shared DSpark MoE layers" used by the draft (perf-changelog.yaml:9283). The old comment at line 42 confirms the prior baseline used CK a8w4; this submission-added env var lowers activations to 4-bit for those same layers when DSpark verifies, which CONTRIBUTING.md's Draft-model precision rule forbids: "Online or offline quantization of draft ... activations ... below the shipped precision, whether through a flag, environment variable" (CONTRIBUTING.md:77-80), and the baseline for comparison is the same image with no draft-related settings from the submission (CONTRIBUTING.md:40-42). …
Why this was flagged
…Matching the #3571 GSM8K baseline does not exempt this per CONTRIBUTING.md's own text, since draft-precision changes shift accuracy loss to acceptance rate, which GSM8K does not measure. Fix: either confine the a4w4 opt-in so DSpark's shared MoE path stays at the shipped baseline precision (a8w4) or obtain an explicit CODEOWNER draft-precision waiver per CONTRIBUTING.md before enabling it process-wide.
Trigger: the role env sets VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 (agentic.yaml:66) on a DSpark recipe where speculative-config uses method dspark (agentic.yaml:56) and DSpark verification reuses the target's MoE weights (changelog says 'shared DSpark MoE layers', perf-changelog.yaml:9283). Before this diff, moe-backend: aiter resolved to the CK a8w4 kernel (old comment, now replaced at agentic.yaml:42); after this diff the same layers run a4w4, i.e. activations drop from 8-bit to 4-bit for both target and draft tokens.
Verification: normal; acknowledged in diff. This is a genuine Draft-model precision policy concern that the base branch does not have.
Decisive facts from the diff itself:
- agentic.yaml:66 adds
VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4: '1'to the roleenvof a DSpark (speculative-decoding) recipe (speculative-config: '{"method":"dspark",...}', agentic.yaml:56). - The old comment at line 42 ("The plain name lets the CK a8w4 MoE kernel win vLLM's priority list") establishes the pre-change out-of-the-box default was CK a8w4 (8-bit activations).
There was a problem hiding this comment.
@chunfangamd can we get a waiver on that? Or I can simply remove the a4w4 flag here.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38028752236 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38028752236 |
|
Closing this PR as a4w4 doesn't show significant perf gain. and a lower precision of draft model requires extra waiver. |
1 similar comment
|
Closing this PR as a4w4 doesn't show significant perf gain. and a lower precision of draft model requires extra waiver. |
Fangzhou-Ai
removed
the
full-sweep-enabled
There was a problem hiding this comment.
Code review found no new issues
No new issues were found in this update; 1 finding from earlier reviews is still open above.
This review covers commit 144ab45, which is no longer the latest commit on this pull request; later commits are not covered by it.
perf-changelog.yaml is rebuilt as origin/main byte for byte, then the #3690 and #3735 entries are appended at the tail. This keeps the GLM-5.2 #3616 entry's config-keys header, which the #3690 merge commit had folded into the preceding pr-link scalar. perf-changelog.yaml 按字节重建为 origin/main 的内容，再在末尾追加 #3690 与 #3735 两条记录。这样可保留 GLM-5.2 #3616 条目的 config-keys 头，该头 曾被 #3690 的合并提交并入前一条的 pr-link 标量。 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace --max-num-batched-tokens 16384 with 8192 on the thirteen TP2/TP4 points that still carried it; TP2 c64 was already 8192. The chunk is now uniform across the arm, matching the B300 arm and the B200 TP4 arm from concurrency 8 up. A prefill chunk scheduled into the same engine step as a decode batch stalls every decoding sequence for the length of that chunk, so the stall scales with the chunk. On the two TP4 ep1 arms, the p90-minus-p50 ITL spread is 10.81 ms at c64 on MI355X at 16384 against 1.74 ms on B200 at 8192. Re-append the #3690 changelog entry at the tail. Its merge commit had folded the config-keys header of the GLM-5.2 #3616 entry into the preceding pr-link scalar, which re-keyed #3616 and left #3690's own entry unparseable. 将仍为 16384 的十三个 TP2/TP4 数据点的 --max-num-batched-tokens 改为 8192； TP2 c64 原本已是 8192。该分块现在在整个臂上统一，与 B300 臂以及并发 8 及以上 的 B200 TP4 臂一致。 与 decode 批次调度进同一个引擎步的 prefill 分块，会让所有正在解码的序列停顿 该分块的时长，因此停顿时间随分块大小增长。在两个 TP4 ep1 臂上，c64 的 ITL p90 减 p50 差值在 16384 的 MI355X 上为 10.81 ms，而在 8192 的 B200 上为 1.74 ms。 同时在文件末尾重新追加 #3690 的 changelog 记录。其合并提交曾把 GLM-5.2 #3616 条目的 config-keys 头并入前一条的 pr-link 标量，导致 #3616 被错误归键， 且 #3690 自身的记录无法被解析。 Co-authored-by: Cursor <cursoragent@cursor.com>
Merges the updated #3690 branch, so this branch again differs from it by exactly one thing. Narrows that difference: --max-num-batched-tokens 8192 now applies only to override_tp4_c32, override_tp4_c64 and override_tp2_c32 (override_tp2_c64 was already 8192); concurrency 1-16 returns to 16384. The 14-point A/B in run 37375212446 against baseline run 37080504692 found gains only at c32 and above (TP2 c32 ITL p90 -14.4%, TP4 c64 -27.8%) and repeated TTFT regressions below it (TP4 c4 +29.5%, TP2 c4 +18.8%), matching the profiled chunked-prefill step counts. perf-changelog.yaml conflicted at its append-only tail; resolved to main's bytes verbatim plus the #3690 and #3735 entries. 合并更新后的 #3690 分支，使本分支与其重新只差一处，并收窄这处差异： --max-num-batched-tokens 8192 现在仅作用于 override_tp4_c32、 override_tp4_c64 与 override_tp2_c32（override_tp2_c64 本就是 8192）， 并发 1-16 回到 16384。运行 37375212446 对基线运行 37080504692 的 14 点 A/B 显示收益只出现在并发 32 及以上（TP2 c32 的 ITL p90 -14.4%，TP4 c64 -27.8%），低于该并发则 TTFT 反复恶化（TP4 c4 +29.5%，TP2 c4 +18.8%）， 与 profile 中分块 prefill 步的计数一致。 perf-changelog.yaml 在仅可追加的尾部产生冲突，解析为逐字节保留 main 的 内容并追加 #3690 与 #3735 两个条目。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
a4w4 dropped from this branch; the ladder now sweeps a8w4Run 37424787697 on this image failed all nine TP=2 points and passed all nine TP=4 points. Root cause, reproduced locally on MI355X and bisected with a TP2/TP4 A/B on one build:
DSv4.1-Flash shards TP=4 only escapes because its What changed here
Fix statusvllm-project/vllm#60273 rounds the a4w4 intermediate to 256. It works (TP=2 boots, GSM8K 0.9629 flexible-extract over all 1319 questions) but costs +11% MoE weights at TP=2, +33% at TP=4, and moves both rows off every tuned AITER row, so it is not the fix this recipe should wait for. The defect is AITER-side and the cheaper fix is validated locally: AI assistance was used for the root-cause investigation and this change. |
Fangzhou-Ai
changed the title
Repin the MI355X DeepSeek-V4.1-Flash DSpark recipe to nightly-rocm100-21d93d0d and retune the ladder around it. - Leave VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4 unset. The a4w4 opt-in faults during graph capture at TP=2; all nine TP=2 points failed and all nine TP=4 points passed in run 37424787697. vllm-project/vllm#60273 tracks the fix, so the experts stay on the generic AITER CK a8w4 selector. - Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 so tensor-parallel all-reduces above the quick-reduce size threshold use the INT4 codec. - Hold both TP rows at concurrency 64. Run 37510565585 measured TP=2 c128 at 150,683 tok/s/chip against 151,896 at c64 with prompt tokens within 0.5%, so the extra concurrency bought real prefill rather than scored tokens. - Set VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD=1024. Upstream caps the shared-expert overlap at 256 tokens, which predates speculative decoding: five DSpark drafts make a decode step CONC x 6 tokens, so c64 submits 384 and the overlap never engages above c42. 1024 covers the ladder and matches the ATOM default, and stays below the prefill chunk so memory profiling never takes the overlap. - Restore the upstream 16384 prefill chunk at TP2 c64, making the chunk uniform across all 14 points. That point had capped it at 8192 to buy KV room, but the room was taken by a profiling bug rather than by the chunk: the ROCm paged MXFP4 indexer sizes its decode logits workspace as (rows, max-model-len), and with no ambient vLLM config in scope the row count fell back to the batched-token count, so at this recipe's 1M context the reservation scaled linearly with the chunk. vllm-project/vllm#58014 bounds rows by max-num-seqs x (1 + 5 drafts), and the repinned image carries that fix. Rebased onto main: the configuration-procedure guides this branch had edited were removed by #3789, so those edits are dropped. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
d0a98ff to
8a96da0
Compare
- Set VLLM_ROCM_MONO_DECODE=1 (vllm-project/vllm#60397): two persistent launches a layer for decode steps of up to 48 rows; larger steps take the regular path. - Image TBD until the first ROCm 10.0 nightly with #60397 is published. - Drop the a4w4 notes and shorten the comments and the changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fangzhou-Ai
changed the title
Fangzhou-Ai
removed
the
full-sweep-enabled
Keep the upstream default for simplicity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the TBD placeholder with nightly-rocm100-7d0b4e57, the first ROCm 10.0 nightly built after vllm-project/vllm#60397 merged, so the VLLM_ROCM_MONO_DECODE=1 setting on this recipe has kernels to run. 将 TBD 占位符替换为 nightly-rocm100-7d0b4e57，这是 vllm-project/vllm#60397 合并后构建的首个 ROCm 10.0 nightly，使本 recipe 中的 VLLM_ROCM_MONO_DECODE=1 设置有对应的 kernel 可用。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
This branch has not been deployed
3 participants
Enables the DeepSeek-V4.1 mono decode kernels from vllm-project/vllm#60397 in the MI355X DeepSeek-V4.1-Flash vLLM DSpark AgentX recipe (
dsv41flash-fp4-mi355x-vllm-agentic-dspark), to measure their end-to-end gain on the full ladder.Changes
VLLM_ROCM_MONO_DECODE=1. The kernels run a decode step of up to 48 rows as two persistent launches a layer, at TP2 and TP4. With five DSpark drafts that is concurrency 1-8; larger steps take the regular path, so c16-c64 should be unchanged.TBD. It will be the first ROCm 10.0 nightly that contains vllm#60397; the sweep runs once that image is published and pinned here.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4(vllm-project/recipes#1061);For reviewers
中文
在 MI355X DeepSeek-V4.1-Flash vLLM DSpark AgentX 配置（
dsv41flash-fp4-mi355x-vllm-agentic-dspark）中启用 vllm-project/vllm#60397 的 DeepSeek-V4.1 mono decode 内核，以在完整并发梯度上测量其端到端收益。改动
VLLM_ROCM_MONO_DECODE=1：在 TP2 和 TP4 下，每层用两个常驻内核处理不超过 48 行的 decode 步。DSpark 有 5 个草稿 token，对应并发 1-8；更大的步走原有路径，因此 c16-c64 预期不变。TBD：将使用第一个包含 vllm#60397 的 ROCm 10.0 nightly，镜像发布并固定后再运行 sweep。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4（vllm-project/recipes#1061）；评审注意
AI model disclosure
Prepared with Claude Opus 5.5 (
claude-opus-5-5) in Claude Code: the recipe, master-config and changelog edits, the merge withmain, and this description. Earlier revisions of this PR were prepared with Claude Opus 5.5 through a Cursor agent.