[AMD][dsv4] Add MTP variant of dsv4-fp4-mi355x-sglang-disagg - #2305
Conversation
Mirrors the base disagg search space and topologies (same 20260701 image, mori KV transfer + MoE a2a, sglang-router --pd-disaggregation); enables DeepSeek-V4 built-in MTP on decode via spec-decoding: mtp, following the dsr1-fp8-mi355x-sglang-disagg-mtp precedent (DECODE_MTP_SIZE=2 on the pure-TP8 arm, =1 on the DEP8 arms).
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29980711776 |
The new MTP config set DECODE_MTP_SIZE=1/2 + spec-decoding: mtp, but the DeepSeek-V4-Pro-DI recipe (the models.yaml key non-agentic disagg resolves to) had no mtp_flags. build_server_config therefore emitted --speculative-num-steps/--speculative-num-draft-tokens with no --speculative-algorithm, so sglang never enabled speculative decoding and the MTP arms ran identical plain-autoregressive decode to the spec-none base run. Add mtp_flags (NEXTN, eagle-topk 1) to DeepSeek-V4-Pro-DI; it is consumed only when DECODE_MTP_SIZE>0, so it is a no-op for the spec-none base scenario.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992478501 |
…code at init) The prior mtp_flags used --speculative-algorithm NEXTN (the DeepSeek-R1/V3 keyword), which crashes the dsv4 decode server at startup. DeepSeek-V4 on sglang loads its built-in MTP via the EAGLE spec path with eagle-topk 1 (in-checkpoint draft, no draft-model-path), matching the in-repo dsv4 sglang recipes under srt-slurm-recipes/sglang/deepseek-v4/. Switch the keyword and document why it must stay EAGLE.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29994577055 |
Depth 2 on the pure-TP8/EP1 arm was lossy on the 20260701 build (greedy gsm8k ~0.85, below the 0.91 dsv4 floor); depth 1 is lossless (~0.96, matching the spec-none base). Set DECODE_MTP_SIZE=1 on that arm to match the DEP8 arms, and record the finding in the changelog and config comment.
Resolve perf-changelog conflict (keep dsr1-fp8-b300-sglang-mtp entry from main). Fix disagg MTP prefill/decode symmetry: build_server_config appended the speculative flags to the decode server only, so prefill never allocated the nextn (MTP draft) KV layer -- prefill registered 3 PD state components, decode 4. SGLang PD-disaggregation requires matching speculative config on both roles; apply the flags to both. Bump the scenario image to lmsysorg/sglang-rocm:v0.5.15-rocm720-mi35x-20260713, which hard-fails the mismatch instead of silently corrupting greedy EAGLE verification.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30186038896 |
Restore draft depth 2 on the pure-TP8/EP1 decode arm (was 1). Depth-2 delivers ~15-20% higher output throughput at low/mid concurrency on this bandwidth-bound topology; the both-sided-flags fix must hold greedy EAGLE accuracy at depth 2 for it to stay. DEP8 arms unchanged (depth 1).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30189328462 |
|
/reuse-sweep-run |
1 participant
No description provided.