[AMD][DSV4] Enable FHMoE on MI355X vLLM: 8K/1K STP, 8K/1K MTP, AgentX / [AMD][DSV4] 在 MI355X vLLM 上启用 FHMoE：8K/1K STP、8K/1K MTP、AgentX - #2851
Conversation
Fuse the DeepSeek-V4 native-FP8 shared expert into the MXFP4 routed-expert AITER kernel by moving the 8K/1K STP, 8K/1K MTP, and AgentX entries onto the first ROCm nightly containing vllm-project/vllm#53161 (vllm@de69e821). VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 was already requested by the AgentX script, but every earlier nightly rejected this mixed FP4+FP8 checkpoint in vLLM's eligibility check and self-disabled the fusion. Co-authored-by: Cursor Agent <agent@cursor.com> Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Repin all three DSv4 MI355X vLLM entries to nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 (Docker Hub pushed 2026-09-07T05:26:48Z), the first ROCm nightly that ships vllm-project/vllm#53161 (vllm-project/vllm@de69e821), which fuses the DeepSeek-V4 native-FP8 shared expert into the MXFP4 routed-expert AITER kernel (FHMoE). The tag commit is 7 commits ahead of de69e821 with nothing behind it." | ||
| - "Export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 from the 8K/1K STP and MTP scripts; the AgentX script already set it. On every nightly before de69e821 the request was inert: vLLM's eligibility check rejected this mixed FP4+FP8 checkpoint and the fusion self-disabled at startup. #53161 is what makes the flag take effect." | ||
| - "Fusion eligibility is gated on gfx950, tensor parallelism 8, data parallelism 1, no expert parallelism and no EPLB, --moe-backend aiter, BF16 model dtype, and the DeepSeek-V4 block-128 FP8 shared-expert quantization config. All three entries satisfy it; the AgentX DP-attention arm does not and keeps the unfused path." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING |
There was a problem hiding this comment.
🔴 The appended entry uses pull/PENDING as its pr-link, but validate_perf_changelog.py only accepts the literal XXX placeholder (or pull/XXX) or an exact pull/<pr_number> match — PENDING matches neither, so CI validation fails once this PR gets a number. Fix: use pull/XXX as the placeholder (matching CONTRIBUTING.md's documented convention) so it passes validate_added_pr_link and gets corrected to the real number on merge.
Extended reasoning...
utils/validate_perf_changelog.py:144-160 validate_added_pr_link() checks the new entry's pr-link against PR_LINK_PLACEHOLDERS = {"XXX", ".../pull/XXX"} and CANONICAL_PR_LINK (\d+ only); .../pull/PENDING is in neither set, so the function raises ChangelogValidationError. The PR description claims python3 utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD passed locally, but that run was presumably against a base without a PR number assigned (pr_number is None path checks CANONICAL_PR_LINK only, which also fails for PENDING) — either way this value fails validation, blocking CI on this PR or a downstream PR-numbered rerun until corrected to the real PR number or XXX.
Verification: normal. The appended perf-changelog entry ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING (perf-changelog.yaml, last line of the diff hunk). The documented placeholder convention is .../pull/XXX (CONTRIBUTING.md:81), and the validator only accepts PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"} or a canonical…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34087124574 |
vLLM clamps torch threads itself, but only after weight loading completes: "Reducing Torch threads from 16 to 1 for serving" is logged after "Model loading took", so the 805 GiB checkpoint is read with 16 torch threads per worker across 8 workers. Setting it externally caps threads from process start, which is what that log line recommends. Weight loading currently costs 96-99 min per job on this runner, about half of each ~3h job. It does not affect any measured number, since loading completes before warmup and the profiling phase. Also drop the now-stale note that the FHMoE flag needs a build with vllm-project/vllm#53161; the pinned nightly ships it. Co-authored-by: Cursor Agent <cursor@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…fhmoe-nightly Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # configs/amd-master.yaml # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34109201379 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34109535814 |
Resolve conflicts from #2792 and #2841. Both bumped DSv4 MI355X vLLM images to nightlies that predate the FHMoE commit vllm-project/vllm@de69e821 (7c5dc571 by 266 commits, e962733e by 28), which would self-disable the fusion, so all three entries stay on nightly-d9105ea8. Keep the VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 knob that #2792 added to the 8K/1K STP and MTP scripts. perf-changelog.yaml stays append-only with the #2851 entries at the tail. 将 main 合并进 DSv4 MI355X FHMoE 分支，解决 #2792 与 #2841 带来的冲突。 这两个 PR 都把 DSv4 MI355X vLLM 镜像升级到了早于 FHMoE 提交 vllm-project/vllm@de69e821 的 nightly（7c5dc571 落后 266 个提交，e962733e 落后 28 个），会导致该融合自动关闭，因此三个条目继续固定在 nightly-d9105ea8。 保留 #2792 为 8K/1K STP 与 MTP 脚本新增的 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 开关。perf-changelog.yaml 保持仅追加，#2851 的条目位于文件末尾。 Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Fangzhou-Ai
changed the title
Companion recipe PRThe upstream recipe change that goes with this repin is vllm-project/recipes#932 — [ROCm] Correct DeepSeek-V4-Pro shared-expert fusion note. It updates It is still a draft because the image pinned in that section ( 配套的 recipe PR与本次镜像重新固定配套的上游 recipe 改动是 vllm-project/recipes#932 —— [ROCm] Correct DeepSeek-V4-Pro shared-expert fusion note。 该 PR 修改 它目前仍是 draft，因为该章节中固定的镜像（ |
|
@chunfangamd |
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: the linked upstream recipe PR vllm-project/recipes#932 is still OPEN, not merged. The standard requires the upstream recipe change to be merged (or already published) before this PR merges, and the currently published DeepSeek-V4-Pro recipe still documents shared-expert fusion as self-disabling on this checkpoint, so published docs do not yet cover this PR's FHMoE behavior. Merge #932, then refresh the sign-off. ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
|
✅✅✅ Verdict: PASS ✅✅✅Supersedes the earlier rejection on this SHA: the sole blocker, vllm-project/recipes#932, is now MERGED (2026-09-08T06:07:30Z). ✅ Check 0 (CODEOWNER): PASS — |
|
hi @chunfangamd are we good to merge now? |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 26794c9. Configure here.
lgtm |
The #2851 block was spliced between the '- config-keys:' header and the body of the #2866 qwen3.5-fp8-b200-sglang entry, so the two mappings collapsed and the file no longer parsed. check-changelog failed with 'expected <document start>, but found <block mapping start>' and skipped every sweep job. Restore the base file byte-for-byte and append the entry at the end, as validate_perf_changelog.py's append-only rule requires.
Review notes before we tick off the full sweep / 在验收 full sweep 之前的审阅记录The diff at Four things came up that affect the sweep sign-off rather than the code. 1.
|
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34287188950 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34287188950 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34287188950 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34287188950 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34287188950 |
|
/reuse-sweep-run 34287188950 |
|
/stage-results 34287188950 |
|
@chunfangamd staged run 34287188950: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-08~r34287188950 This run remains available across future |
|
/reuse-sweep-run 34287188950 |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34287188950
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34287188950
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
vllm-project/recipes#932
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: per MODELS.md, 2026-09-08 was the last day for the Single-turn 8k1k scenario on ✅ Check 0 (CODEOWNER): PASS — |
4 participants
What
Turns on the DeepSeek-V4 heterogeneous fused MoE (FHMoE) path for all three DSv4 MI355X vLLM entries:
dsv4-fp4-mi355x-vllm(8K/1K STP)dsv4-fp4-mi355x-vllm-mtp(8K/1K MTP)dsv4-fp4-mi355x-vllm-agentic-mtp(AgentX)Two pieces are required, and neither works without the other:
nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36(Docker Hub pushed 2026-09-07T05:26:48Z), the first ROCm nightly containing vllm-project/vllm#53161 (de69e821). Verified via the compare API: the tag commit is 7 ahead ofde69e821, 0 behind.VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1from the two 8K/1K scripts. The AgentX script already set it.This PR also exports
OMP_NUM_THREADS=1from all three scripts. vLLM clamps torch threads itself, but only after the weights are loaded, so the checkpoint is otherwise read with 16 torch threads per worker across 8 workers. Setting it externally caps threads from process start. It does not affect any measured number: loading completes before warmup and profiling.Why the flag alone was never enough
On every nightly before
de69e821, the export was inert. vLLM's eligibility check rejected this mixed FP4+FP8 checkpoint and the fusion self-disabled at startup, which is what thevllm-project/recipesDSv4-Pro caveat describes. #53161 adds the heterogeneous path that makes the flag take effect.Image pin versus the automated bumps
Two image bumps landed on
mainwhile this PR was open, and both would silently disable FHMoE:de69e821e962733e7c5dc571Both predate the fusion commit, so the merge here keeps all three entries on
d9105ea8. TheVLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4knob that #2792 added to the 8K/1K scripts is preserved.perf-changelog.yamlremains append-only, with the #2792 entry in place and the #2851 entries at the tail.Eligibility
vLLM gates the fused path on gfx950, TP8, DP1, no expert parallelism, no EPLB,
--moe-backend aiter, BF16 dtype, and the DSv4 block-128 FP8 shared-expert quant config. All three entries qualify. The AgentX DP-attention arm (TP1/DP8) does not and keeps the unfused path, so it is unaffected.Rejection is logged at debug level, so reviewers should confirm from the server log that the fused path was selected rather than inferring it from the flag. The line to look for is
Using AITER DeepSeek V4 heterogeneous MoE for CSV-covered MoE input rows. Note that AITER's tuned-config coverage is probed per token count, so the fused kernel is selected only for coveredM; the 8K/1K lanes sweep concurrency 4-512 and it is worth checking both ends of that range rather than startup alone.Testing
Local validation:
A full sweep completed on this branch: run 34109535814, 50 jobs, 0 failures, covering the 8K/1K STP and MTP lanes at concurrency 4-512 and AgentX at concurrency 1-64 plus the DP-attention arm. The server log from that run confirms the fused path was selected.
中文说明
为三个 DSv4 MI355X vLLM 条目启用 DeepSeek-V4 异构融合 MoE（FHMoE）路径：
dsv4-fp4-mi355x-vllm（8K/1K STP）dsv4-fp4-mi355x-vllm-mtp（8K/1K MTP）dsv4-fp4-mi355x-vllm-agentic-mtp（AgentX）需要同时满足两项，缺一不可：
nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36（Docker Hub 推送时间 2026-09-07T05:26:48Z），这是第一个包含 vllm-project/vllm#53161（de69e821）的 ROCm nightly。已通过 compare API 核实：该 tag 提交领先de69e8217 个提交，落后 0 个。VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1。AgentX 脚本此前已经设置。本 PR 还在三个脚本中导出
OMP_NUM_THREADS=1。vLLM 自身会限制 torch 线程数，但只在权重加载完成之后才生效，因此检查点会以每个 worker 16 个 torch 线程、共 8 个 worker 的方式被读取。在外部设置该变量可以从进程启动起就限制线程数。这不会影响任何测量结果：加载在预热和性能采集之前就已完成。为什么仅有开关是不够的
在
de69e821之前的所有 nightly 上，这个导出都是无效的。vLLM 的适用性检查会拒绝这个 FP4 与 FP8 混合的检查点，融合在启动时自动关闭，这也正是vllm-project/recipes中 DSv4-Pro 说明所描述的情况。#53161 引入了异构路径，使该开关真正生效。镜像固定与自动化升级的取舍
本 PR 开启期间，
main上合入了两个镜像升级，两者都会静默关闭 FHMoE：de69e821的位置e962733e7c5dc571两者都早于融合提交，因此这里的合并让三个条目继续保持在
d9105ea8。#2792 为 8K/1K 脚本新增的VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4开关予以保留。perf-changelog.yaml保持仅追加：#2792 的条目位置不变，#2851 的条目位于文件末尾。生效条件
vLLM 对融合路径的判定条件为：gfx950、TP8、DP1、不启用专家并行、不启用 EPLB、
--moe-backend aiter、BF16 数据类型，以及 DSv4 的 block-128 FP8 共享专家量化配置。三个条目均满足。AgentX 的 DP-attention 分支（TP1/DP8）不满足，将继续使用非融合路径，因此不受影响。拒绝信息记录在 debug 级别，因此审阅者应当从 server log 确认确实选择了融合路径，而不是仅凭开关推断。需要查找的日志行是
Using AITER DeepSeek V4 heterogeneous MoE for CSV-covered MoE input rows。需要注意，AITER 的调优配置覆盖范围是按 token 数逐个探测的，只有被覆盖的M才会选中融合内核；8K/1K 分支的并发范围是 4-512，因此值得同时检查该范围的两端，而不只是启动阶段。中文验证
本地校验：
本分支已完成一次完整 sweep：run 34109535814，共 50 个任务，0 个失败，覆盖 8K/1K STP 与 MTP 分支的并发 4-512、AgentX 的并发 1-64 以及 DP-attention 分支。该次运行的 server log 确认已选中融合路径。
Note
Medium Risk
Repinning changes the MoE kernel path and benchmark numbers for three high-visibility DSv4 MI355X vLLM entries; mistakes in image eligibility or fusion gating would silently alter serving behavior.
Overview
Enables DeepSeek-V4 heterogeneous fused MoE (FHMoE) for all three MI355X vLLM configs (
dsv4-fp4-mi355x-vllm,-mtp, and-agentic-mtp) by repinning them tovllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36, the first ROCm nightly that includes vllm-project/vllm#53161. ExistingVLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1exports become effective on that image; comments in the 8K/1K launch scripts andconfigs/amd-master.yamlnow describe the fusion instead of noting self-disable on older builds.The three serving scripts also export
OMP_NUM_THREADS=1so weight loading on cold WEKA reads does not run with oversubscribed torch threads before vLLM’s post-load clamp.perf-changelog.yamldocuments the repin, FHMoE behavior, eligibility (TP8/DP1 arms), and the thread cap.Reviewed by Cursor Bugbot for commit 088279b. Bugbot is set up for automated code reviews on this repo. Configure here.