Skip to content

Revert "[AgentX] DeepSeek-V4 B300 SGLang update" (#2701) / 回滚 "[AgentX] DeepSeek-V4 B300 SGLang update"（#2701） #2736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Oseltamivir wants to merge 2 commits into from
Closed

Revert "[AgentX] DeepSeek-V4 B300 SGLang update" (#2701) / 回滚 "[AgentX] DeepSeek-V4 B300 SGLang update"（#2701） #2736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b03415f
Revert "[AgentX] DeepSeek-V4 B300 SGLang update (#2701)"
Oseltamivir Aug 26, 2026
d6afb3a
chore(changelog): set canonical pr-link for the #2701 revert entry
Oseltamivir Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 17 additions & 87 deletions benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,17 +65,14 @@ export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1
CACHE_ARGS=()
WARMUP_ARGS=()
if require_agentic_kv_offload_backend hicache; then
# DeepSeek V4 HiCache rejects --hicache-size and controls capacity only
# through a host/device token ratio, so TOTAL_CPU_DRAM_GB cannot apply
# directly. Host capacity scales with BOTH the ratio and device KV, so it
# also grows with mem-fraction-static -- the two knobs multiply. Measured:
# TP8 ratio=2 at mem-fraction 0.835 gives 999 GB. ratio=4 at mem-fraction
# 0.93 overshoots: it left only 5.84 GB free on a 2,964 GB node and the
# V4 paged pool failed to allocate. ratio=3 keeps the tier near 2 TB with
# room for the paged pool, page cache, AIPerf and the router, while still
# well above the old half-node rule that pinned TP8 to ratio=2.
# DeepSeek V4 HiCache currently rejects --hicache-size and supports
# capacity control only through a host/device token-capacity ratio.
# DSv4 exposes capacity as a host/device token ratio rather than bytes.
# Measurements put TP8 ratio=2 near 950 GB and TP4 ratio=8 near 1 TB,
# both below their configured capacities. The old TP4 ratio=16
# used roughly 2 TB and violated the half-node allocation rule.
if [ "$TP" -ge 8 ]; then
DEFAULT_HICACHE_RATIO=3
DEFAULT_HICACHE_RATIO=2
else
DEFAULT_HICACHE_RATIO=8
fi
Expand Down Expand Up @@ -119,53 +116,21 @@ if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS+=(
--dp "$TP"
--tokenizer-worker-num "$TP"
--enable-prefill-delayer
--prefill-decode-interval 20
--enable-dp-attention
--enable-dp-attention-local-control-broadcast
--incremental-streaming-output
--stream-interval 20
--dist-init-addr "127.0.0.1:$((PORT + 2000))"
--ep-size "$EP_SIZE"
--moe-a2a-backend megamoe
--enable-deepseek-v4-fp4-indexer
--moe-runner-backend flashinfer_mxfp4
--disable-flashinfer-autotune
)
# DEP4 shards the model over half the node, so per-rank weights roughly
# double and the weights-only floor rises above 0.9 (the engine reports a
# minimum viable 0.9013 and refuses to start). Keep upstream's 0.95 there.
# DEP8 has room for the lower value, which leaves mega-MoE workspace
# headroom.
if [ "$TP" -ge 8 ]; then
# Mega-MoE's transient workspace lives OUTSIDE the static allocation and
# needs a single ~7 GB contiguous block, so headroom must grow with
# concurrency. Measured at conc 256: 0.835 (~42 GB free) runs; 0.93
# (~16 GB free) and 0.95 (~11 GB free) both die with a CUDA OOM on one
# DP rank, which then hangs the whole engine in the MLP-sync collective.
MEM_FRACTION_STATIC=0.93
if [ "$CONC" -ge 512 ]; then
MEM_FRACTION_STATIC=0.86
elif [ "$CONC" -ge 384 ]; then
MEM_FRACTION_STATIC=0.88
elif [ "$CONC" -ge 256 ]; then
MEM_FRACTION_STATIC=0.9
fi
else
# DEP4 is squeezed from both sides: weights occupy ~90% of each GPU when
# the model is sharded over half the node, so the engine refuses to start
# below ~0.902 (no KV left), while megamoe still needs its ~7 GB
# workspace above the static budget. 0.95 leaves only ~11 GB free -- the
# same margin that OOM'd a rank at DEP8 conc 256 -- so use 0.93, which
# gives the ~16 GB that DEP8 conc 128 runs with at the same per-rank load
# (max-running-requests/dp = 32 in both cases).
MEM_FRACTION_STATIC=0.93
MEM_FRACTION_STATIC=0.95
if [ "$CONC" -ge 512 ]; then
# Leave room for FlashInfer's transient MoE workspace at the DEP8 tail.
MEM_FRACTION_STATIC=0.94
fi
# --chunked-prefill-size is a GLOBAL budget: server_args.py divides it by
# dp_size, and dp_size is TP here. Scale it so every DEP shape gets the
# per-rank 8192 that was tuned, rather than 16384/rank at DEP4 -- which
# exceeds MegaMoE's per-rank token cap (a startup ValueError) and measured
# slower at DEP8 when tried directly.
CHUNKED_PREFILL_SIZE=$((8192 * TP))
CHUNKED_PREFILL_SIZE=16384
else
PARALLEL_ARGS+=(
--moe-runner-backend flashinfer_mxfp4
Expand All @@ -182,24 +147,9 @@ MODEL_ARGS=(
# AgentX concurrency counts live session trees, not individual requests.
# Allow subagent fan-out to exceed CONC without clipping request bursts.
MAX_RUNNING_REQUESTS=$((2 * CONC))
# Subagent fan-out means live requests exceed CONC (see MAX_RUNNING_REQUESTS
# above), so sizing decode graphs at CONC would drop every larger batch to
# eager decode. Capture past the fan-out; the runtime clamps this down to the
# request pool size anyway.
CUDA_GRAPH_MAX_BS=$((CONC * 4))
CUDA_GRAPH_MAX_BS=$CONC
[ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64

# --cuda-graph-max-bs is an alias whose dest is cuda_graph_max_bs_decode, so the
# two forms below are the same knob and must not both be passed.
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS")
SWA_FULL_TOKENS_RATIO=0.1
if [ "$DP_ATTENTION" = "true" ]; then
# Decode graphs must cover the padded MTP batch across all DP ranks, which
# exceeds CONC; capping at 64 would fall back to eager decode.
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544)
SWA_FULL_TOKENS_RATIO=0.075
fi

export PYTHONNOUSERSITE=1
export TORCH_CUDA_ARCH_LIST=10.0
# Agentic warmup dispatches hundreds of large prompts at once. SGLang's
Expand All @@ -218,17 +168,6 @@ export SGLANG_OPT_USE_JIT_NORM=1
export SGLANG_OPT_USE_JIT_INDEXER_METADATA=1
export SGLANG_OPT_USE_TOPK_V2=1
export SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1
if [ "$DP_ATTENTION" = "true" ]; then
# MegaMoE's FP4/MXF4 activation path is opt-in -- both flags default False,
# so --moe-a2a-backend megamoe alone runs a different kernel than the one
# measured. DG_USE_FP4_ACTS / DG_USE_MXF4_KIND are forwarded to DeepGEMM
# automatically from these two.
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1
# Must cover the per-rank prefill budget (8192) or startup raises; the
# extra 128 is headroom over the exact-fit boundary.
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320
fi
if [ "${EVAL_ONLY}" != "true" ]; then
export SGLANG_SIMULATE_ACC_LEN=2.49
export SGLANG_SIMULATE_ACC_METHOD=match-expected
Expand All @@ -252,9 +191,9 @@ SGLANG_CMD=(
--trust-remote-code
"${PARALLEL_ARGS[@]}"
--mem-fraction-static "$MEM_FRACTION_STATIC"
--swa-full-tokens-ratio "$SWA_FULL_TOKENS_RATIO"
--swa-full-tokens-ratio 0.1
--max-running-requests "$MAX_RUNNING_REQUESTS"
"${CUDA_GRAPH_ARGS[@]}"
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
--allow-auto-truncate
--chunked-prefill-size "$CHUNKED_PREFILL_SIZE"
--tool-call-parser deepseekv4
Expand Down Expand Up @@ -313,16 +252,7 @@ if [ "$USE_SGLANG_ROUTER" = "true" ]; then
--connect-timeout-secs 900 \
--request-timeout-secs 14400 \
--disable-health-check \
`# A single transient router->engine send failure would otherwise` \
`# surface as a 500, and AgentX aborts the whole run when a root` \
`# warmup request fails ("ProfileAborted"). Measured at conc 512:` \
`# 22 such transients in one 3600s run, spread over all 8 DP` \
`# workers, every one of them recovered by the retry; with retries` \
`# disabled a single one killed a 2h15m arm.` \
--retry-max-retries 8 \
--retry-initial-backoff-ms 500 \
--retry-max-backoff-ms 10000 \
--retry-backoff-multiplier 2 > "$ROUTER_LOG" 2>&1 &
--disable-retries > "$ROUTER_LOG" 2>&1 &
ROUTER_PID=$!
echo "Router PID: $ROUTER_PID"
wait_for_ready \
Expand Down
20 changes: 10 additions & 10 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1155,7 +1155,7 @@ dsv4-fp4-b300-sglang:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 4096, conc-end: 4096 }

dsv4-fp4-b300-sglang-agentic-hicache-mtp:
image: lmsysorg/sglang-staging:dev-cu13-pr-35880
image: lmsysorg/sglang:v0.5.17-cu130
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b300-nv
Expand All @@ -1164,21 +1164,21 @@ dsv4-fp4-b300-sglang-agentic-hicache-mtp:
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.95
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [32, 64, 128, 256, 384, 512, 576], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 20, 24, 32] }
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32, 40, 48, 52, 56, 60, 64, 72] }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [8, 16, 24, 32, 40, 64], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [32, 40, 48, 56, 64, 72, 80, 88, 96, 128], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [52, 72, 100, 128, 144, 196, 512], router: { name: sglang-router, version: "0.3.2" } }

# DeepSeek-V4-Pro on B300 with EAGLE/MTP speculative decoding. Recipe is
# selected inside benchmarks/single_node/dsv4_fp4_b300_sglang_mtp.sh by
# DP_ATTENTION:
# dp-attn: false -> TP-only + flashinfer_mxfp4 + chunked-prefill 8192
# + mem-fraction 0.88 + swa-full-tokens-ratio 0.1
# dp-attn: true -> DP-attn + megamoe + fp4 indexer
# + chunked-prefill 65536 + mem-fraction 0.90
# + swa-full-tokens-ratio 0.075
# + prefill-decode-interval 20
# Both paths share EAGLE (3,1,4) and max-running-requests 2*CONC.
# + EAGLE (3,1,4) + mem-fraction 0.90
# dp-attn: true -> DP-attn + flashinfer_mxfp4 + chunked-prefill 32768
# + EAGLE (1,1,2) + mem-fraction 0.92 + max-running 256
Comment on lines 1178 to +1181

@claude claude Bot Aug 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New comment block documenting benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh no longer matches the script it describes. It claims dp-attn:false uses mem-fraction 0.90 (script default is 0.88), dp-attn:true uses chunked-prefill 32768 (script sets 16384), EAGLE (1,1,2) for dp-attn:true (script always uses --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, unconditionally), mem-fraction 0.92 for dp-attn:true (script sets 0.95, or 0.94 when CONC>=512), and a fixed max-running-requests of 256 (script computes MAX_RUNNING_REQUESTS=2*CONC, not a fixed value).

Extended reasoning...

An engineer re-landing the reverted #35880 tuning (as perf-changelog.yaml instructs) or debugging this recipe reads this comment to understand current serving parameters, tunes against the wrong baseline values (e.g. assumes 32768 chunked-prefill or a fixed 256 max-running-requests), and either misconfigures the re-land or wastes time reconciling the mismatch with the actual script.

Verification: nit — documentation-only mismatch introduced by this diff; no runtime effect. The rewritten comment in configs/nvidia-master.yaml (dp-attn:false -> mem-fraction 0.90; dp-attn:true -> chunked-prefill 32768 + EAGLE (1,1,2) + mem-fraction 0.92 + max-running 256) contradicts the script benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh: MEM_FRACTION_STATIC=0.88 default (dp-attn:false), `MEM_

dsv4-fp4-b300-sglang-mtp:
image: lmsysorg/sglang:nightly-dev-cu13-20260610-f332e526
model: deepseek-ai/DeepSeek-V4-Pro
Expand Down
10 changes: 10 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6467,3 +6467,13 @@
- "Reduce the concurrency-1536 disaggregated topology from six to five DEP8 prefill workers while retaining one DEP16 decode worker."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2644


- config-keys:
- dsv4-fp4-b300-sglang-agentic-hicache-mtp
scenario-type:
- agentic-coding
description:
- "Revert #2701. Restore the image to lmsysorg/sglang:v0.5.17-cu130, the 42-point search space (TP4 and DEP4 rows included), dram-utilization 0.80, and the pre-#2701 serving flags in the recipe script."
- "#2701 served every point on lmsysorg/sglang-staging:dev-cu13-pr-35880, a staging image carrying the unmerged draft sgl-project/sglang#35880, so its results do not correspond to any released SGLang build. Its ingested run (32695861783) has been purged from the dashboard."
- "Re-land the tuning once #35880 merges upstream and ships in a released tag."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2736