perf(dsv4): update B200 SGLang AgentX HiCache MTP DEP8 configuration / 更新 B200 SGLang AgentX HiCache MTP DEP8 配置 - #2656
Conversation
更新 B200 SGLang AgentX HiCache MTP DEP8 配置。
在性能变更日志中补充 PR #2656 链接。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --moe-a2a-backend deepep | ||
| --deepep-config "$DEEPEP_CONFIG" | ||
| ) | ||
| CHUNKED_PREFILL_SIZE=32768 | ||
| # This is the global budget across all eight DP-attention ranks. | ||
| CHUNKED_PREFILL_SIZE=65536 | ||
| SWA_FULL_TOKENS_RATIO=0.02 | ||
| else | ||
| PARALLEL_ARGS+=( | ||
| --moe-runner-backend flashinfer_mxfp4 |
There was a problem hiding this comment.
🔴 MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) is only added in the DP_ATTENTION=true branch (line 109) — the TP-only else branch never touches MODEL_ARGS, so the tp:8/kv-offloading:none and tp:8/kv-offloading:dram search-space rows launch without the FP4 indexer while the DP-attn+EP rows get it. The sibling recipe benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh applies this flag on both branches (lines 95, 111), and benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh:292 sets the equivalent use_fp4_indexer_cache unconditionally, confirming it's an indexer-level optimization independent of DP-attn/TP-only, not something that should be scoped to the DP-attn branch — fix by adding it to the else branch (or hoisting it above the if) so both paths get it.
Extended reasoning...
The bug: In benchmarks/single_node/agentic/dsv4_fp4_b200_sglang_mtp.sh, this PR moves MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) from its own initialization block into the middle of the function, but only inside the if [ "$DP_ATTENTION" = "true" ]; then branch (line 109/128-136 in the final file). MODEL_ARGS=() is initialized empty right before the if/else at line 104, and the else (TP-only) branch at lines 137-141 only appends --moe-runner-backend flashinfer_mxfp4 and --disable-flashinfer-autotune to PARALLEL_ARGS — it never touches MODEL_ARGS. So for any launch where DP_ATTENTION != true, MODEL_ARGS stays empty and --enable-deepseek-v4-fp4-indexer is never passed to sglang.launch_server.\n\nWhere this bites: configs/nvidia-master.yaml's dsv4-fp4-b200-sglang-agentic-hicache-mtp search-space has four rows; two of them (tp: 8, kv-offloading: none and tp: 8, kv-offloading: dram) have no dp-attn: true, meaning DP_ATTENTION resolves to false/unset for those launches and they take the TP-only else branch. Those two rows will run this whole DEP8 sweep without the FP4 indexer, while the other two rows (tp: 8, ep: 8, dp-attn: true, ...) get it. Since this recipe's entire purpose is to produce a comparable TP-only vs DP-attn perf picture for the DEP8 config, an indexer optimization present on one side and silently absent on the other will skew that comparison — a TP-only regression (or an apparent DP-attn win) could actually just be the missing flag, not a real architectural difference.\n\nWhy nothing else catches this: there's no validation tying MODEL_ARGS contents to DP_ATTENTION, no lint/test asserting flag parity across branches, and the flag is silently omitted rather than erroring — the server launches fine either way, it's just running a materially different (unoptimized) configuration on the TP-only rows.\n\nPrecedent proving this is a path-independent flag, not a DP-attn-specific one: benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh sets --enable-deepseek-v4-fp4-indexer in both its DP-attn branch (line 95) and its TP-only else branch (line 111) — I verified this directly by reading the file just now. Notably, on that B300 recipe the indexer flag is the one thing kept identical across both paths, while the DeepGEMM MoE FP4 env vars (SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS, USE_MXF4_KIND, etc.) remain DP-attn-only — i.e. the sibling recipe already draws exactly the distinction this PR should draw: MoE/DP-attn tuning stays scoped to DP-attn, but the indexer flag applies everywhere. Separately, benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh:292 sets the equivalent use_fp4_indexer_cache: true unconditionally in the shared --attention-config, outside any DP_ATTENTION conditional. Both sibling recipes agree the indexer is orthogonal to the DP-attn/TP-only choice.\n\nStep-by-step proof: (1) MODEL_ARGS=() at line 104. (2) For the tp: 8, kv-offloading: none row in configs/nvidia-master.yaml, no dp-attn key is set, so DP_ATTENTION is false at runtime. (3) Execution takes the else branch at line 137, which appends only to PARALLEL_ARGS. (4) MODEL_ARGS remains (). (5) At line ~207, "${MODEL_ARGS[@]}" expands to nothing in SGLANG_CMD. (6) sglang.launch_server starts without --enable-deepseek-v4-fp4-indexer for this row, while the dp-attn: true rows in the same sweep do get it — an inconsistent, unintended asymmetry.\n\nFix: move MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) out of the if branch — either place it right after MODEL_ARGS=() (applying unconditionally, matching the vLLM sibling) or add the same line to the else branch (matching the B300 sibling's per-branch structure) so both TP-only and DP-attn rows receive the flag.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32168174681 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32168174681 |
|
/stage-results 32168174681 |
|
@functionstackx staged run 32168174681: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-18~r32168174681 This run remains available across future |
|
/reuse-sweep-run |
# Conflicts: # perf-changelog.yaml
|
/reuse-sweep-run |
2 participants
Description
Updates the
dsv4-fp4-b200-sglang-agentic-hicache-mtpDEP8 configuration.Uses image
lmsysorg/sglang:v0.5.17-cu130.描述
更新
dsv4-fp4-b200-sglang-agentic-hicache-mtpDEP8 配置。使用镜像
lmsysorg/sglang:v0.5.17-cu130。Related Issue
Fixes #
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.