Skip to content

Add DSV4 B200 disaggregated Dynamo SGLang MTP configuration / 新增 DSV4 B200 分离式 Dynamo SGLang MTP 配置 #2554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adibarra merged 3 commits into from
Aug 12, 2026
Merged

Add DSV4 B200 disaggregated Dynamo SGLang MTP configuration / 新增 DSV4 B200 分离式 Dynamo SGLang MTP 配置 #2554

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0555089
feat(config): add DSV4 B200 nscale UCX canary
RohitNagraj Aug 10, 2026
8c15a0a
feat(config): expand DSV4 B200 Dynamo-SGLang MTP matrix
RohitNagraj Aug 11, 2026
6d9249c
chore: merge main into dsv4 branch
adibarra Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions ...e/srt-slurm-recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p2d-dep8-dep8-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: "disagg-b200-8k1k-mid-curve-1p2d-dep8-dep8-mtp"

frontend:
type: dynamo
enable_multiple_frontends: true
num_additional_frontends: 8

dynamo:
hash: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e"
install: true

model:
path: "deepseek-v4-pro"
container: "lmsysorg/sglang:nightly-dev-cu13-20260710-cfc66e05"
precision: "fp4"

sbatch_directives:
cpus-per-task: "144"
mem: "0"
Comment on lines +17 to +19

@claude claude Bot Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: cpus-per-task: "144" looks copied from GB200/B300 recipes and may be unsatisfiable on nscale B200 nodes
Why it matters: 144 matches GB200/GB300 Grace nodes (2×72 cores) — see e.g. disagg-gb200-mid-curve-1p1d-dep8-dep16-mtp.yaml, which this recipe otherwise mirrors. The B200 nscale nodes are x86 (the sibling dsv4-fp4-b200-dynamo-vllm-mtp recipes use the vllm-x86_64 image), and none of the existing B200 nscale recipes set cpus-per-task (they use sbatch_directives: segment: "1" instead, e.g. benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-dep8-mtp.yaml:15). If the nscale nodes have fewer than 144 cores available per task, sbatch rejects the job with "Requested node configuration is not available" and the canary never starts.
Fix: Confirm the nscale B200 node core count supports --cpus-per-task=144, or drop the directive (and consider whether segment: "1" from the existing nscale recipes is needed) so Slurm uses its defaults.


resources:
gpu_type: "b200"
gpus_per_node: 8
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 8
decode_nodes: 2
decode_workers: 2
gpus_per_decode: 8

health_check:
max_attempts: 240

backend:
type: sglang

prefill_environment:
PYTHONUNBUFFERED: "1"
SGLANG_RADIX_FORCE_MISS: "1"
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
SGLANG_DEFAULT_THINKING: "1"
SGLANG_DSV4_REASONING_EFFORT: "max"
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN: "1"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "9216"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: "1"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: "1"
SGLANG_OPT_FIX_MEGA_MOE_MEMORY: "1"

NCCL_CUMEM_ENABLE: "1"
# Keep RC for inter-node transport, but stage CUDA buffers through host memory.
UCX_TLS: "rc,cuda_ipc,cuda_copy,self"
UCX_IB_GPU_DIRECT_RDMA: "no"
UCX_PROTO_EMULATION_ENABLE: "y"

@claude claude Bot Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: UCX_PROTO_EMULATION_ENABLE does not appear to be a real UCX configuration variable
Why it matters: Mainline UCX exposes UCX_PROTO_ENABLE / UCX_PROTO_INFO, and proto-v2 "emulation" protocols are selected automatically when a transport lacks a capability — I can find no PROTO_EMULATION_ENABLE knob in the UCX config table. Unknown UCX_* vars are silently ignored (at most a "unused env var" warning), so the run will proceed, but the canary's stated third knob ("enable host-staged protocol emulation") would be a no-op and conclusions drawn from this arm could be misattributed. Host staging is still forced here by UCX_IB_GPU_DIRECT_RDMA=no + cuda_copy, so the experiment isn't broken — it just may not test what the PR description claims. (Same variable appears in the decode block at line 77.)
Fix: Verify the variable exists in the container's UCX build (ucx_info -c | grep -i proto), or rely on UCX_PROTO_INFO=y output (already enabled) to confirm which protocol was actually selected, and drop the variable if it's not recognized. Note: I could not verify against UCX source/docs from this sandbox (network access blocked), so please double-check before drawing conclusions from the run.

UCX_PROTO_INFO: "y"
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"

decode_environment:
PYTHONUNBUFFERED: "1"
SGLANG_RADIX_FORCE_MISS: "1"
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
SGLANG_DEFAULT_THINKING: "1"
SGLANG_DSV4_REASONING_EFFORT: "max"
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN: "1"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "2048"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: "1"
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: "1"
SGLANG_OPT_FIX_MEGA_MOE_MEMORY: "1"

NCCL_CUMEM_ENABLE: "1"
# Keep RC for inter-node transport, but stage CUDA buffers through host memory.
UCX_TLS: "rc,cuda_ipc,cuda_copy,self"
UCX_IB_GPU_DIRECT_RDMA: "no"
UCX_PROTO_EMULATION_ENABLE: "y"
UCX_PROTO_INFO: "y"
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"
SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: "0"

sglang_config:
prefill:
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
model-path: "/model/"
trust-remote-code: true
tool-call-parser: deepseekv4
disaggregation-mode: "prefill"
disaggregation-transfer-backend: nixl

tensor-parallel-size: 8
data-parallel-size: 8
expert-parallel-size: 8

enable-dp-attention: true
enable-dp-lm-head: true

moe-a2a-backend: "megamoe"
deepep-config: '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'

mem-fraction-static: 0.8
max-running-requests: 512
cuda-graph-max-bs: 512
chunked-prefill-size: 65536
stream-interval: 60

decode:
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
model-path: "/model/"
trust-remote-code: true
tool-call-parser: deepseekv4
disaggregation-mode: "decode"
disaggregation-transfer-backend: nixl

tensor-parallel-size: 8
data-parallel-size: 8
expert-parallel-size: 8

enable-dp-attention: true
enable-dp-lm-head: true

moe-a2a-backend: "megamoe"
deepep-config: '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'

speculative-algo: "EAGLE"
speculative-num-steps: 3
speculative-eagle-topk: 1
speculative-num-draft-tokens: 4

mem-fraction-static: 0.85
max-running-requests: 1024
cuda-graph-max-bs: 512
swa-full-tokens-ratio: 0.15
context-length: 16384
stream-interval: 60

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
random_range_ratio: 0.8
concurrencies: "256"
req_rate: "inf"
use_chat_template: true
custom_tokenizer: "sa_bench_tokenizers.sglang_deepseek_v4.SGLangDeepseekV4Tokenizer"
31 changes: 31 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5280,6 +5280,37 @@ dsv4-fp4-gb300-dynamo-vllm:
ep: 16
dp-attn: true

dsv4-fp4-b200-dynamo-sglang-mtp:
image: lmsysorg/sglang:nightly-dev-cu13-20260710-cfc66e05
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b200-nscale
precision: fp4
framework: dynamo-sglang
router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" }
kv-p2p-transfer: nixl
multinode: true
disagg: true
scenarios:
fixed-seq-len:
- isl: 8192
osl: 1024
search-space:
- spec-decoding: "mtp"
conc-list: [256]
prefill:
num-worker: 1
tp: 8
ep: 8
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p2d-dep8-dep8-mtp.yaml"
decode:
num-worker: 2
tp: 8
ep: 8
dp-attn: true

dsv4-fp4-b200-dynamo-vllm-mtp:
image: vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9
model: deepseek-ai/DeepSeek-V4-Pro
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5770,3 +5770,11 @@
- "models.yaml: add mtp_flags (--speculative-algorithm EAGLE --speculative-eagle-topk 1) to DeepSeek-V4-Pro-AgentX; num-steps/num-draft-tokens are derived from DECODE_MTP_SIZE in server_sglang.sh. DeepSeek-V4's MTP head is a native NextN/EAGLE draft shipped with the model, so no --speculative-draft-model-path is needed."
- "Two search-space arms: TP8/EP1 no-DP at conc-list [2,4,8,16,32], and TP8/EP8/DPA (ep=8, dp-attn=true) at conc-list [64,96,128], both on image lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260719."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2309

- config-keys:
- dsv4-fp4-b200-dynamo-sglang-mtp
description:
- "Add a focused DeepSeek-V4-Pro FP4 Dynamo-SGLang MTP configuration on the B200 nscale runner."
- "Keep UCX RC transport while disabling GPU-direct RDMA registration and enabling host-staged protocol emulation for CUDA buffers."
- "Image: lmsysorg/sglang:nightly-dev-cu13-20260710-cfc66e05"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2554
26 changes: 19 additions & 7 deletions runners/launch_b200-nscale-slurm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
# them would force every dgxc path to become an override hook and would let a
# dgxc-only change silently alter nscale runs.
#
# Scope: multi-node Dynamo-vLLM DeepSeek-V4-Pro and Kimi K2.6 FP4 runs on
# the b200-nscale/b200-new runner labels. Anything else exits non-zero.
# Scope: multi-node Dynamo-vLLM DeepSeek-V4-Pro and Kimi K2.6 FP4 runs, plus
# DeepSeek-V4-Pro FP4 Dynamo-SGLang MTP, on the b200-nscale/b200-new runner
# labels. Anything else exits non-zero.

SLURM_PARTITION="batch_1"
SLURM_ACCOUNT="benchmark"
Expand Down Expand Up @@ -49,8 +50,9 @@ else
exit 1
fi

if [[ $FRAMEWORK != "dynamo-vllm" ]]; then
echo "Unsupported framework for b200-nscale: $FRAMEWORK (only dynamo-vllm)" >&2
if [[ $FRAMEWORK != "dynamo-vllm" ]] &&
[[ $MODEL_PREFIX != "dsv4" || $PRECISION != "fp4" || $FRAMEWORK != "dynamo-sglang" || $SPEC_DECODING != "mtp" ]]; then
echo "Unsupported framework/configuration for b200-nscale: $MODEL_PREFIX/$PRECISION/$FRAMEWORK/$SPEC_DECODING" >&2
exit 1
fi

Expand All @@ -59,7 +61,15 @@ export SERVED_MODEL_NAME=$MODEL
echo "Cloning srt-slurm repository..."
SRT_REPO_DIR="srt-slurm"
rm -rf "$SRT_REPO_DIR"
if [[ $MODEL_PREFIX == "dsv4" ]]; then
if [[ $MODEL_PREFIX == "dsv4" && $FRAMEWORK == "dynamo-sglang" ]]; then
git clone --branch main --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" || exit 1
cd "$SRT_REPO_DIR" || exit 1
# Pin srt-slurm so this focused run changes only the runner and UCX
# transport behavior.
git checkout 04e87fcc505d6d851451781a5499ca19a02ec2b4 || exit 1
mkdir -p recipes/sglang/deepseek-v4
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4" recipes/sglang/deepseek-v4
elif [[ $MODEL_PREFIX == "dsv4" ]]; then
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" || exit 1
cd "$SRT_REPO_DIR" || exit 1
git checkout aflowers/vllm-gb200-v0.20.0 || exit 1
Expand Down Expand Up @@ -165,6 +175,7 @@ model_paths:
# Container aliases
containers:
dynamo-vllm: "${SQUASH_FILE}"
dynamo-sglang: "${SQUASH_FILE}"
"${IMAGE}": "${SQUASH_FILE}"
nginx-sqsh: "${NGINX_SQUASH_FILE}"
use_exclusive_sbatch_directive: true
Expand Down Expand Up @@ -201,8 +212,9 @@ sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "$CONFIG_PATH"
inject_synthetic_acceptance "$CONFIG_PATH" "$FRAMEWORK" || exit 1

SRTCTL_PREFLIGHT_ARGS=()
# Kimi K2.6 weights are staged on the Slurm compute nodes, not the login node.
if [[ $MODEL_PREFIX == "kimik2.6" ]]; then
# These weights are staged on the Slurm compute nodes, not the login node.
if [[ $MODEL_PREFIX == "kimik2.6" ]] ||
[[ $MODEL_PREFIX == "dsv4" && $FRAMEWORK == "dynamo-sglang" ]]; then
SRTCTL_PREFLIGHT_ARGS+=(--no-preflight)
fi

Expand Down
Loading