SemiAnalysisAI · stewtong · Aug 8, 2026 · Aug 8, 2026
@@ -138,6 +138,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init
 | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios |
 |---|---|---|---|---|
 | Qwen3.8 2.4T | `qwen3.8` | TBD | Agentic coding | |
+| DeepSeek-V4-Flash-0731 | `dsv4f` | TBD | Single-turn 8k1k | |
 | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark only) | Agentic coding non-DSpark arm (deprecated from day 0) |
 | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (MTP-only pending — the non-MTP arm still runs; see the Deprecation Notice) | |
 | MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) |

@@ -138,6 +138,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和
 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 |
 |---|---|---|---|---|
 | Qwen3.8 2.4T | `qwen3.8` | 待定 | 智能体编码 | |
+| DeepSeek-V4-Flash-0731 | `dsv4f` | 待定 | 单轮 8k1k | |
 | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（仅 DSpark） | 智能体编码非 DSpark 分支（自第 0 天起弃用） |
 | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（「仅 MTP」待执行 —— 非 MTP 分支仍在运行，见弃用公告） | |
 | MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 智能体编码 | 单轮 1k1k、单轮 8k1k（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） |

diff --git a/benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh b/benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh
@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+
+# DeepSeek-V4-Flash-0731 B200 single-node vLLM recipe, validated on an
+# 8x B200 SXM6 node (driver 580.126.09 / 580.173.02) with
+# vllm/vllm-openai:v0.25.0 (digest
+# sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97,
+# vLLM build commit dd10e03f95f94edbea1975c67ace3a35ec9a8a40).
+#
+# Two serving arms, both with expert parallel: vLLM v0.25.0's MegaMoE
+# backend hard-requires EP and raises NotImplementedError before weight
+# load if it is missing.
+#   TP arm (DP_ATTENTION=false): tensor-parallel 8 + EP8. This is the
+#     single-stream / latency arm; at 1M context the TP8 vs DP8
+#     break-even is c=3 (vllm-project/vllm#51454).
+#   DP arm (DP_ATTENTION=true):  data-parallel 8 (TP1 per replica) + EP8.
+#     This is the concurrency and long-context arm: 7.7x more total KV
+#     capacity than TP8 and a same-node 3.44x TTFT p50 advantage at
+#     1M c=8 (honest range across metrics and runs: 3.2-3.7x).
+# Prefix caching is left at the engine default (enabled): the report
+# shows it intact on Flash, with a 22x warm-TTFT speedup on re-sent 1M
+# documents, and the random-id benchmark has no shared prefixes either
+# way.
+
+source "$(dirname "$0")/../../benchmark_lib.sh"
+
+check_env_vars \
+    MODEL \
+    TP \
+    DP_ATTENTION \
+    CONC \
+    ISL \
+    OSL \
+    MAX_MODEL_LEN \
+    RANDOM_RANGE_RATIO \
+    RESULT_FILENAME
+
+if [[ -n "$SLURM_JOB_ID" ]]; then
+    echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
+fi
+
+nvidia-smi
+
+# The Flash checkpoint is ~155 GiB. Download from HF only when handed a
+# bare repo id (b200-nb / b200-cw runners); clusters with pre-staged
+# weights hand a local path via the launcher instead. The serving runs
+# behind this config key were validated against this exact snapshot.
+MODEL_REVISION=${MODEL_REVISION:-7872f01b1d1fe23eabc4c98b48bffcef5a386062}
+REVISION_ARGS=()
+if [[ "$MODEL" != /* ]]; then
+    hf download "$MODEL" --revision "$MODEL_REVISION"
+    REVISION_ARGS=(--revision "$MODEL_REVISION")
+fi
+
+SERVER_LOG=/workspace/server.log
+
+# Engine startup covers MegaMoE init plus fp4 indexer cache allocation;
+# allow an hour so cold weight loads do not hit the default readiness
+# window.
+export VLLM_ENGINE_READY_TIMEOUT_S=3600
+
+PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
+if [ "${DP_ATTENTION}" = "true" ]; then
+    PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
+fi
+
+EP_ARGS=()
+if [ "${EP_SIZE:-1}" -gt 1 ]; then
+    EP_ARGS=(--enable-expert-parallel)
+fi
+
+if [ "${EVAL_ONLY}" = "true" ]; then
+    EVAL_MAX_MODEL_LEN=$(compute_eval_context_length "$MODEL" "$MAX_MODEL_LEN")
+    export EVAL_MAX_MODEL_LEN
+    SERVE_MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN"
+else
+    SERVE_MAX_MODEL_LEN="$MAX_MODEL_LEN"
+fi
+
+# Start GPU monitoring (power, temperature, clocks every second)
+start_gpu_monitor
+
+set -x
+vllm serve "$MODEL" --host 0.0.0.0 --port "$PORT" \
+    --trust-remote-code \
+    "${REVISION_ARGS[@]}" \
+    --kv-cache-dtype fp8 \
+    --block-size 256 \
+    --moe-backend deep_gemm_mega_moe \
+    --attention_config.use_fp4_indexer_cache=True \
+    --tokenizer-mode deepseek_v4 \
+    --tool-call-parser deepseek_v4 \
+    --enable-auto-tool-choice \
+    --reasoning-parser deepseek_v4 \
+    "${PARALLEL_ARGS[@]}" \
+    "${EP_ARGS[@]}" \
+    --max-model-len "$SERVE_MAX_MODEL_LEN" \
+    --max-num-batched-tokens 8192 \
+    --max-num-seqs 256 \
+    --gpu-memory-utilization 0.95 > "$SERVER_LOG" 2>&1 &
+
+SERVER_PID=$!
+
+# Wait for server to be ready
+wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
+
+pip install -q datasets pandas
+
+run_benchmark_serving \
+    --model "$MODEL" \
+    --port "$PORT" \
+    --backend vllm \
+    --input-len "$ISL" \
+    --output-len "$OSL" \
+    --random-range-ratio "$RANDOM_RANGE_RATIO" \
+    --num-prompts "$((CONC * 10))" \
+    --max-concurrency "$CONC" \
+    --result-filename "$RESULT_FILENAME" \
+    --result-dir /workspace/ \
+    --trust-remote-code
+
+# After throughput, run evaluation only if RUN_EVAL is true
+if [ "${RUN_EVAL}" = "true" ]; then
+    run_eval --framework lm-eval --port "$PORT"
+    append_lm_eval_summary
+fi
+
+# Stop GPU monitoring
+stop_gpu_monitor
+set +x
@@ -1017,6 +1017,33 @@ dsv4-fp4-b200-vllm-mtp:
       - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp }
       - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256, spec-decoding: mtp }
 
+# DeepSeek-V4-Flash-0731: first Flash-on-B200 key. Both arms validated on
+# Nebius 8x B200 SXM with vllm/vllm-openai:v0.25.0 (vllm-project/vllm#51454).
+# Model snapshot 7872f01b1d1fe23eabc4c98b48bffcef5a386062 is pinned inside
+# benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh (MODEL_REVISION).
+# TP8+EP8 is the single-stream/latency arm (break-even vs DP8 at c=3 in the
+# report's 1M ladder); DP8 (attention DP + EP8) is the concurrency arm. Both
+# arms carry ep: 8 because vLLM v0.25.0's MegaMoE backend refuses engine init
+# without expert parallel (NotImplementedError before weight load).
+dsv4f-fp8-b200-vllm:
+  image: vllm/vllm-openai:v0.25.0
+  model: deepseek-ai/DeepSeek-V4-Flash-0731
+  model-prefix: dsv4f
+  runner: cluster:b200-nb
+  precision: fp8
+  framework: vllm
+  multinode: false
+  scenarios:
+    fixed-seq-len:
+    - isl: 8192
+      osl: 1024
+      search-space:
+      # TP8 + EP8, single-stream to mid concurrency
+      - { tp: 8, ep: 8, conc-start: 1, conc-end: 64 }
+      # DP8 (attention DP) + EP8, concurrency/throughput arm; capped by
+      # --max-num-seqs 256 in the script
+      - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256 }
+
 # NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/DeepSeek/DeepSeek-R1
 # does not have a B300-specific recipe, so this config reuses the existing DSR1 FP4
 # B200 SGLang recipe as-is until B300-specific tuning is available.

diff --git a/configs/runners.yaml b/configs/runners.yaml
@@ -243,6 +243,9 @@ labels:
     - b200-dgxc_07
     - b200-dgxc_08
     - b200-dgxc_09
+  cluster:b200-nb:
+    - b200-nb_0
+    - b200-nb_1
   cluster:b300-cw:
     - b300-cw_0
     - b300-cw_01

diff --git a/perf-changelog.yaml b/perf-changelog.yaml
@@ -5697,3 +5697,11 @@
     - "Use the official lmsysorg/sglang:v0.5.16-rocm720-mi30x image, enable SGLang prompt-cache reporting, and pass the backend Prometheus metrics endpoint explicitly to AIPerf"
     - "Allow up to 30 minutes for healthy AgentX responses admitted near the end of the measurement window to drain before AIPerf finalizes profile metric coverage"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2528
+
+- config-keys:
+    - dsv4f-fp8-b200-vllm
+  description:
+    - "Add DeepSeek-V4-Flash-0731 FP8 B200 vLLM single-node 8k1k: the first Flash-on-B200 config key, on the new cluster:b200-nb runner label (Nebius fleet), routed to benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh. Image vllm/vllm-openai:v0.25.0 (digest sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97, vLLM build dd10e03f95f94edbea1975c67ace3a35ec9a8a40); model snapshot 7872f01b1d1fe23eabc4c98b48bffcef5a386062 pinned inside the script via MODEL_REVISION."
+    - "Recipe flags are the TP8/DP8 recipe validated on 8x B200 and reported upstream in vllm-project/vllm#51454: --kv-cache-dtype fp8, --block-size 256, --moe-backend deep_gemm_mega_moe, --attention_config.use_fp4_indexer_cache=True, deepseek_v4 tokenizer / tool-call / reasoning parsers, --max-num-batched-tokens 8192, --max-num-seqs 256, --gpu-memory-utilization 0.95. Both arms run expert parallel because vLLM v0.25.0's MegaMoE backend refuses engine init without EP (NotImplementedError before weight load). TP8 covers conc 1-64 (single-stream arm; DP8 break-even at c=3 in the report's 1M ladder), DP8 (attention DP + EP8) covers conc 64-256, capped by --max-num-seqs. Prefix caching is left at the engine default: the report shows it intact on Flash (22x warm-TTFT speedup on a re-sent 1M document)."
+    - "MODELS.md gains the DeepSeek-V4-Flash-0731 row (prefix dsv4f); runners.yaml gains cluster:b200-nb (b200-nb_0/1)."
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2535