Skip to content

perf(dsv4): update B200 SGLang AgentX HiCache MTP DEP8 configuration / 更新 B200 SGLang AgentX HiCache MTP DEP8 配置 #2656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
functionstackx merged 4 commits into from
Aug 19, 2026
Merged

perf(dsv4): update B200 SGLang AgentX HiCache MTP DEP8 configuration / 更新 B200 SGLang AgentX HiCache MTP DEP8 配置 #2656

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b0f58bc
perf(dsv4): update B200 SGLang AgentX HiCache MTP DEP8 configuration
hshrivastava-droid Aug 18, 2026
2085965
chore(changelog): link PR #2656
hshrivastava-droid Aug 18, 2026
ed75a92
Merge remote-tracking branch 'origin/main' into pr2656-work
functionstackx Aug 19, 2026
3e58b2c
chore: refresh PR #2656 for sweep reuse [skip-sweep]
functionstackx Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions benchmarks/single_node/agentic/dsv4_fp4_b200_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,14 +101,19 @@ fi

PARALLEL_ARGS=(--tp "$TP")
METRICS_ARGS=(--enable-metrics --enable-cache-report)
MODEL_ARGS=()
CHUNKED_PREFILL_SIZE=8192
SWA_FULL_TOKENS_RATIO=0.1
if [ "$DP_ATTENTION" = "true" ]; then
DEEPEP_CONFIG='{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'
MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer)
export SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1
export SGLANG_OPT_FIX_HASH_MEGA_MOE=1
export SGLANG_OPT_USE_FAST_MASK_EP=1
export SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8192
export SGLANG_OPT_FIX_NEXTN_MEGA_MOE=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0
PARALLEL_ARGS+=(
Expand All @@ -123,15 +128,16 @@ if [ "$DP_ATTENTION" = "true" ]; then
--moe-a2a-backend deepep
--deepep-config "$DEEPEP_CONFIG"
)
CHUNKED_PREFILL_SIZE=32768
# This is the global budget across all eight DP-attention ranks.
CHUNKED_PREFILL_SIZE=65536
SWA_FULL_TOKENS_RATIO=0.02
else
PARALLEL_ARGS+=(
--moe-runner-backend flashinfer_mxfp4
Comment on lines 128 to 136

@claude claude Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) is only added in the DP_ATTENTION=true branch (line 109) — the TP-only else branch never touches MODEL_ARGS, so the tp:8/kv-offloading:none and tp:8/kv-offloading:dram search-space rows launch without the FP4 indexer while the DP-attn+EP rows get it. The sibling recipe benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh applies this flag on both branches (lines 95, 111), and benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh:292 sets the equivalent use_fp4_indexer_cache unconditionally, confirming it's an indexer-level optimization independent of DP-attn/TP-only, not something that should be scoped to the DP-attn branch — fix by adding it to the else branch (or hoisting it above the if) so both paths get it.

Extended reasoning...

The bug: In benchmarks/single_node/agentic/dsv4_fp4_b200_sglang_mtp.sh, this PR moves MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) from its own initialization block into the middle of the function, but only inside the if [ "$DP_ATTENTION" = "true" ]; then branch (line 109/128-136 in the final file). MODEL_ARGS=() is initialized empty right before the if/else at line 104, and the else (TP-only) branch at lines 137-141 only appends --moe-runner-backend flashinfer_mxfp4 and --disable-flashinfer-autotune to PARALLEL_ARGS — it never touches MODEL_ARGS. So for any launch where DP_ATTENTION != true, MODEL_ARGS stays empty and --enable-deepseek-v4-fp4-indexer is never passed to sglang.launch_server.\n\nWhere this bites: configs/nvidia-master.yaml's dsv4-fp4-b200-sglang-agentic-hicache-mtp search-space has four rows; two of them (tp: 8, kv-offloading: none and tp: 8, kv-offloading: dram) have no dp-attn: true, meaning DP_ATTENTION resolves to false/unset for those launches and they take the TP-only else branch. Those two rows will run this whole DEP8 sweep without the FP4 indexer, while the other two rows (tp: 8, ep: 8, dp-attn: true, ...) get it. Since this recipe's entire purpose is to produce a comparable TP-only vs DP-attn perf picture for the DEP8 config, an indexer optimization present on one side and silently absent on the other will skew that comparison — a TP-only regression (or an apparent DP-attn win) could actually just be the missing flag, not a real architectural difference.\n\nWhy nothing else catches this: there's no validation tying MODEL_ARGS contents to DP_ATTENTION, no lint/test asserting flag parity across branches, and the flag is silently omitted rather than erroring — the server launches fine either way, it's just running a materially different (unoptimized) configuration on the TP-only rows.\n\nPrecedent proving this is a path-independent flag, not a DP-attn-specific one: benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh sets --enable-deepseek-v4-fp4-indexer in both its DP-attn branch (line 95) and its TP-only else branch (line 111) — I verified this directly by reading the file just now. Notably, on that B300 recipe the indexer flag is the one thing kept identical across both paths, while the DeepGEMM MoE FP4 env vars (SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS, USE_MXF4_KIND, etc.) remain DP-attn-only — i.e. the sibling recipe already draws exactly the distinction this PR should draw: MoE/DP-attn tuning stays scoped to DP-attn, but the indexer flag applies everywhere. Separately, benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh:292 sets the equivalent use_fp4_indexer_cache: true unconditionally in the shared --attention-config, outside any DP_ATTENTION conditional. Both sibling recipes agree the indexer is orthogonal to the DP-attn/TP-only choice.\n\nStep-by-step proof: (1) MODEL_ARGS=() at line 104. (2) For the tp: 8, kv-offloading: none row in configs/nvidia-master.yaml, no dp-attn key is set, so DP_ATTENTION is false at runtime. (3) Execution takes the else branch at line 137, which appends only to PARALLEL_ARGS. (4) MODEL_ARGS remains (). (5) At line ~207, "${MODEL_ARGS[@]}" expands to nothing in SGLANG_CMD. (6) sglang.launch_server starts without --enable-deepseek-v4-fp4-indexer for this row, while the dp-attn: true rows in the same sweep do get it — an inconsistent, unintended asymmetry.\n\nFix: move MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer) out of the if branch — either place it right after MODEL_ARGS=() (applying unconditionally, matching the vLLM sibling) or add the same line to the else branch (matching the B300 sibling's per-branch structure) so both TP-only and DP-attn rows receive the flag.

--disable-flashinfer-autotune
)
fi

MODEL_ARGS=()
# The B200-specialized image deadlocks immediately after weight loading when
# forced through the B300 compressed-attention/page-size overrides.
# DeepGEMM's DSv4 indexer needs a multi-GiB temporary allocation at long
Expand Down Expand Up @@ -186,7 +192,7 @@ SGLANG_CMD=(
--trust-remote-code
"${PARALLEL_ARGS[@]}"
--mem-fraction-static "$MEM_FRACTION_STATIC"
--swa-full-tokens-ratio 0.1
--swa-full-tokens-ratio "$SWA_FULL_TOKENS_RATIO"
--max-running-requests "$MAX_RUNNING_REQUESTS"
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
--chunked-prefill-size "$CHUNKED_PREFILL_SIZE"
Expand Down
6 changes: 3 additions & 3 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -941,9 +941,9 @@ dsv4-fp4-b200-sglang-agentic-hicache-mtp:
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 3, 4, 5] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [8, 10, 16, 32, 40, 44] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [16, 24, 32, 38, 44, 48, 50, 52], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [16, 32, 38, 44, 50, 56, 64, 66, 68], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [8, 10, 16, 32] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [16], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [16, 40, 48], router: { name: sglang-router, version: "0.3.2" } }

dsv4-fp4-b200-vllm:
image: vllm/vllm-openai:v0.25.0
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6153,3 +6153,11 @@
- "Add MiniMax-M3 NVFP4 B300 single-node TensorRT-LLM AgentX with EAGLE3-GQA speculative decoding (3 draft tokens)."
- "Use the nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc23.post1 image."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2658

- config-keys:
- dsv4-fp4-b200-sglang-agentic-hicache-mtp
scenario-type:
- agentic-coding
description:
- "Update the B200 SGLang AgentX HiCache MTP DEP8 configuration."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2656