Skip to content

[AgentX] vLLM Deepseek-V4 B200/B300 agg #2202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cquil11 wants to merge 11 commits into from
Closed

[AgentX] vLLM Deepseek-V4 B200/B300 agg #2202

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7c5e43d
dsv4-b200-agg: DEP8 + TP8; dsv4-b300-agg: DEP4 + TP4
ivanium Jul 14, 2026
cbb2642
docs(agentic): set perf-changelog pr-link to #2188
ivanium Jul 14, 2026
8ae6406
refactor(agentic): simplify vLLM recipe config
cquil11 Jul 14, 2026
db7e5a7
refactor(agentic): simplify optional variables
cquil11 Jul 14, 2026
9bfd6f1
Merge branch 'main' into agent/copy-pr-2188
cquil11 Jul 14, 2026
a14052f
Update perf-changelog.yaml
cquil11 Jul 14, 2026
ccf2fc3
chore(config): update DSV4 vLLM nightly
cquil11 Jul 14, 2026
7dfc4b8
fix(agentic): dsv4-b300 TP uses FlashInfer all-reduce, not custom
ivanium Jul 14, 2026
9e51817
Require merged upstream recipe PRs (#2210)
adibarra Jul 14, 2026
b70303b
fix(changelog): restore stray whitespace deletions in perf-changelog
ivanium Jul 14, 2026
60acff5
chore: update vllm-simple version
ivanium Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 51 additions & 25 deletions benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@
# Highest aggregate throughput at large CONC.
#
# Image is configured in nvidia-master.yaml. block_size=256,
# kv-cache-dtype=fp8, FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph
# capture with custom_ops=all (per the vLLM blog recipe at
# https://vllm.ai/blog/deepseek-v4).
# kv-cache-dtype=fp8, FLASHINFER_MLA_SPARSE_DSV4 attention with the FP4 indexer
# cache, FULL_DECODE_ONLY cudagraph capture, and (in EP tiers) mega-MoE backend.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
#
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake.
# Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host
# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple or mooncake.

source "$(dirname "$0")/../../benchmark_lib.sh"

Expand Down Expand Up @@ -65,12 +65,6 @@
resolve_trace_source
install_agentic_deps

# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings,
# which fails DSV4 indexer compilation with mlir_global_dtors(..., data).
# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved.
agentic_pip_install --quiet --force-reinstall --no-deps \
'nvidia-cutlass-dsl-libs-cu13==4.5.2'

# vllm-project/router expands the one HTTP backend into one logical worker per
# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's
# X-Correlation-ID is stable for every turn of a conversation; alias it to the
Expand All @@ -94,6 +88,10 @@
# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's
# store mask. 32k matches the trace-replay tuning validated for this workload.
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_USE_RUST_FRONTEND=1
export VLLM_DSV4_MEGA_FP8_COMBINE=1
export VLLM_RPC_TIMEOUT=600000

# ---- Server config ----------------------------------------------------------
SERVER_LOG="$RESULT_DIR/server.log"
Expand All @@ -106,8 +104,22 @@
MOONCAKE_MASTER_PID=""

OFFLOAD_ARGS=()

if require_agentic_kv_offload_backend mooncake; then
case "${KV_OFFLOAD_BACKEND:-}" in
"")
require_agentic_kv_offload_none
;;
vllm-simple)
require_agentic_kv_offload_backend vllm-simple
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
# Identical prefixes must hash to identical block keys across DP ranks.
export PYTHONHASHSEED=42
OFFLOAD_ARGS=(
--kv-transfer-config
"{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":${CPU_BYTES_PER_RANK},\"lazy_offload\":false,\"enable_cross_layers_blocks\":\"true\"}}"
)
;;

Check failure on line 120 in benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh

View check run for this annotation

Claude / Claude Code Review

B200 vllm-simple KV offload config: wrong extra_config key and known-broken lazy_offload value 

The new B200 vllm-simple KV-offload config (line 118) hardcodes `lazy_offload:false`, which per this repo's own note in `kimik2.5_int4_h200.sh:47-49` is known to hit a `popleft_n` AssertionError at low/mid CONC specifically for DSv4 + SimpleCPUOffloadConnector — exactly the combo and CONC range (8-72) swept here. It also uses key `cpu_bytes_to_use_per_rank` instead of the connector's documented `cpu_bytes_to_use` key used everywhere else (kimik2.5 and the B300 sibling in this same PR), which lik
Comment on lines +115 to +136

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new B200 vllm-simple KV-offload config (line 118) hardcodes lazy_offload:false, which per this repo's own note in kimik2.5_int4_h200.sh:47-49 is known to hit a popleft_n AssertionError at low/mid CONC specifically for DSv4 + SimpleCPUOffloadConnector — exactly the combo and CONC range (8-72) swept here. It also uses key cpu_bytes_to_use_per_rank instead of the connector's documented cpu_bytes_to_use key used everywhere else (kimik2.5 and the B300 sibling in this same PR), which likely means the intended CPU_BYTES_PER_RANK budget is silently ignored. Both affect every B200 vllm-simple tier in nvidia-master.yaml.

Extended reasoning...

Issue 1 — lazy_offload:false (high confidence, blocking):

dsv4_fp4_b200_vllm.sh:118 sets kv_connector_extra_config.lazy_offload=false (eager mode) for SimpleCPUOffloadConnector on DeepSeek-V4. The repo's own documented knowledge in kimik2.5_int4_h200.sh:47-49 states plainly: "JSON form (rather than --kv_offloading_backend native shortcut) so we can pass lazy_offload=true. Eager mode hits a popleft_n AssertionError at low/mid CONC on DSv4 + SimpleCPUOffloadConnector." That note is model-specific (DSv4) and connector-specific (SimpleCPUOffloadConnector) — exactly the combination this PR newly wires up for the first time on B200.

nvidia-master.yaml sweeps this backend at conc-list: [8, 12, 16] (pure-TP8 tier) and conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72] (DEP8 tier) — squarely the low/mid CONC range where the assertion was documented. Corroborating that this is an oversight rather than a deliberate change: the B300 sibling script added in this same PR wires up the identical connector but omits lazy_offload entirely (taking the connector's default), so B200's explicit false is inconsistent with both its own sibling and the pre-existing, proven-safe kimik2.5 pattern. There's no evidence in the PR that the new nightly image (vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-515d6e9) fixes this path — the kimik2.5 script, unaffected by this PR, still passes lazy_offload=true to work around it. If the assertion still fires, every B200 vllm-simple server launch crashes at startup/low-CONC.

Issue 2 — cpu_bytes_to_use_per_rank key (lower confidence, flagged for author confirmation):

The same line uses key cpu_bytes_to_use_per_rank, whereas every other use of this connector in the repo — kimik2.5_int4_h200.sh:51 and the B300 sibling added in this same PR (dsv4_fp4_b300_vllm.sh:104) — uses cpu_bytes_to_use. A verifier refuted this as a bug, arguing the two keys could be deliberately different because B200's pure-TP8 vllm-simple tier is a single engine with world_size=TP=8, and per the documented connector behavior (kimik2.5_int4_h200.sh:42-45: "SimpleCPUOffloadConnector internally divides cpu_bytes_to_use by world_size"), passing the already-divided CPU_BYTES_PER_RANK under the auto-dividing cpu_bytes_to_use key would yield TOTAL/64 instead of TOTAL/8 — a real problem the author may have been trying to avoid by using a different, non-dividing key name.

That concern about double-division is legitimate for the pure-TP8 tier, but it doesn't establish that cpu_bytes_to_use_per_rank is an actual recognized connector option — no reference to it exists anywhere else in this repo, and the connector's documented interface (per kimik2.5's own comment) only knows about cpu_bytes_to_use. The correct fix for the pure-TP8 case, following the established kimik2.5 pattern, would have been to pass the un-divided aggregate TOTAL_CPU_DRAM_GB under cpu_bytes_to_use (letting the connector's internal division by world_size=8 produce the right per-rank share), not to invent a new key name. Separately, this same code path is also used for the B200 DEP8 tier, where DP_ATTENTION=true makes each engine's attention world_size=1 (since --tensor-parallel-size 1 --data-parallel-size 8) — there, passing the pre-divided value under the real cpu_bytes_to_use key (matching B300's DEP4 pattern) would be correct with no double-division risk at all. Since kv_connector_extra_config is a permissive dict typically consumed via .get(key, default), the most likely outcome of the unrecognized key is that the connector silently falls back to its default CPU budget rather than the intended TOTAL_CPU_DRAM_GB-derived budget, for both B200 vllm-simple tiers.

Given the refutation raises a plausible alternative explanation that can't be fully ruled out without inspecting the pinned nightly vLLM build's actual connector source, I'm including this as a secondary, lower-confidence item for the author to confirm/fix alongside the lazy_offload issue — which is the well-documented, high-confidence part of this finding and is sufficient on its own to block the B200 vllm-simple tiers.

Proof sketch for lazy_offload: (1) DSv4 B200 vllm-simple tier starts with KV_OFFLOAD_BACKEND=vllm-simple, CONC=8 (lowest swept value). (2) Server launches with --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector",...,"lazy_offload":false,...}'. (3) Per the repo's documented DSv4 + SimpleCPUOffloadConnector behavior, eager mode hits a popleft_n AssertionError at low/mid CONC. (4) The server process crashes during warmup/serving at low CONC, failing that sweep tier — exactly the failure mode kimik2.5 explicitly engineers around by setting lazy_offload=true.

mooncake)
require_agentic_kv_offload_backend mooncake
# Embedded mode contributes one segment per GPU rank to a shared
# distributed store, so pre-divide the aggregate host-memory budget.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))
Expand All @@ -127,16 +139,14 @@
"global_segment_size": "${PER_RANK_GB}GB",
"local_buffer_size": "4GB",
"protocol": "rdma",
"device_name": "mlx5_0",
"device_name": "",
"enable_offload": false
}
EOF
export MOONCAKE_CONFIG_PATH
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
# Identical prefixes must hash to identical store keys across DP ranks.
export PYTHONHASHSEED=0
# B200 GPU memory registration works through DMA-BUF, but the compute
# nodes do not expose nvidia_peermem. Force Mooncake's DMA-BUF
# GPUDirect RDMA path instead of its legacy ibv_reg_mr path.
export WITH_NVIDIA_PEERMEM=0
export MC_SLICE_SIZE=1048576
export MC_WORKERS_PER_CTX=4
Expand Down Expand Up @@ -167,16 +177,27 @@
--kv-transfer-config
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
)
fi
;;
*)
echo "Error: unsupported B200 KV_OFFLOAD_BACKEND='${KV_OFFLOAD_BACKEND:-}'" >&2
exit 1
;;
esac

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
fi

EP_ARGS=()
FAST_MOE_ARGS=()
if [ "$EP_SIZE" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
FAST_MOE_ARGS=(
--moe-backend deep_gemm_amxf4_mega_moe
--enable-ep-weight-filter
--prefill-schedule-interval 16
)
fi

# AgentX concurrency counts live session trees, not individual requests.
Expand All @@ -197,18 +218,23 @@
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
--attention_config.use_fp4_indexer_cache=True
--max-model-len 1048576
--gpu-memory-utilization 0.92
--numa-bind
--enable-cumem-allocator
--no-enable-flashinfer-autotune
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--enable-prefix-caching
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
--no-disable-hybrid-kv-cache-manager
--disable-uvicorn-access-log
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
--max-num-seqs "$MAX_NUM_SEQS"
--max-cudagraph-capture-size "$MAX_NUM_SEQS"
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
"${FAST_MOE_ARGS[@]}"
"${OFFLOAD_ARGS[@]}"
)
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
Expand Down
Loading
Loading