diff --git a/MODELS.md b/MODELS.md index 85924875ad..3632837b06 100644 --- a/MODELS.md +++ b/MODELS.md @@ -138,6 +138,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios | |---|---|---|---|---| | Qwen3.8 2.4T | `qwen3.8` | TBD | Agentic coding | | +| DeepSeek-V4-Flash-0731 | `dsv4f` | TBD | Single-turn 8k1k | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark only) | Agentic coding non-DSpark arm (deprecated from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (MTP-only pending — the non-MTP arm still runs; see the Deprecation Notice) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | diff --git a/MODELS_zh.md b/MODELS_zh.md index 92886ace74..437feb093f 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -138,6 +138,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者,并响应 AI 实验室和 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 | |---|---|---|---|---| | Qwen3.8 2.4T | `qwen3.8` | 待定 | 智能体编码 | | +| DeepSeek-V4-Flash-0731 | `dsv4f` | 待定 | 单轮 8k1k | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码(仅 DSpark) | 智能体编码非 DSpark 分支(自第 0 天起弃用) | | GLM-5.2 | `glm5.2` | 2026-07-18([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | 智能体编码(「仅 MTP」待执行 —— 非 MTP 分支仍在运行,见弃用公告) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | 智能体编码 | 单轮 1k1k、单轮 8k1k(2026-08-04 移除,[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | diff --git a/benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh b/benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh new file mode 100644 index 0000000000..887c3a7392 --- /dev/null +++ b/benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash + +# DeepSeek-V4-Flash-0731 B200 single-node vLLM recipe, validated on an +# 8x B200 SXM6 node (driver 580.126.09 / 580.173.02) with +# vllm/vllm-openai:v0.25.0 (digest +# sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97, +# vLLM build commit dd10e03f95f94edbea1975c67ace3a35ec9a8a40). +# +# Two serving arms, both with expert parallel: vLLM v0.25.0's MegaMoE +# backend hard-requires EP and raises NotImplementedError before weight +# load if it is missing. +# TP arm (DP_ATTENTION=false): tensor-parallel 8 + EP8. This is the +# single-stream / latency arm; at 1M context the TP8 vs DP8 +# break-even is c=3 (vllm-project/vllm#51454). +# DP arm (DP_ATTENTION=true): data-parallel 8 (TP1 per replica) + EP8. +# This is the concurrency and long-context arm: 7.7x more total KV +# capacity than TP8 and a same-node 3.44x TTFT p50 advantage at +# 1M c=8 (honest range across metrics and runs: 3.2-3.7x). +# Prefix caching is left at the engine default (enabled): the report +# shows it intact on Flash, with a 22x warm-TTFT speedup on re-sent 1M +# documents, and the random-id benchmark has no shared prefixes either +# way. + +source "$(dirname "$0")/../../benchmark_lib.sh" + +check_env_vars \ + MODEL \ + TP \ + DP_ATTENTION \ + CONC \ + ISL \ + OSL \ + MAX_MODEL_LEN \ + RANDOM_RANGE_RATIO \ + RESULT_FILENAME + +if [[ -n "$SLURM_JOB_ID" ]]; then + echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" +fi + +nvidia-smi + +# The Flash checkpoint is ~155 GiB. Download from HF only when handed a +# bare repo id (b200-nb / b200-cw runners); clusters with pre-staged +# weights hand a local path via the launcher instead. The serving runs +# behind this config key were validated against this exact snapshot. +MODEL_REVISION=${MODEL_REVISION:-7872f01b1d1fe23eabc4c98b48bffcef5a386062} +REVISION_ARGS=() +if [[ "$MODEL" != /* ]]; then + hf download "$MODEL" --revision "$MODEL_REVISION" + REVISION_ARGS=(--revision "$MODEL_REVISION") +fi + +SERVER_LOG=/workspace/server.log + +# Engine startup covers MegaMoE init plus fp4 indexer cache allocation; +# allow an hour so cold weight loads do not hit the default readiness +# window. +export VLLM_ENGINE_READY_TIMEOUT_S=3600 + +PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1) +if [ "${DP_ATTENTION}" = "true" ]; then + PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP") +fi + +EP_ARGS=() +if [ "${EP_SIZE:-1}" -gt 1 ]; then + EP_ARGS=(--enable-expert-parallel) +fi + +if [ "${EVAL_ONLY}" = "true" ]; then + EVAL_MAX_MODEL_LEN=$(compute_eval_context_length "$MODEL" "$MAX_MODEL_LEN") + export EVAL_MAX_MODEL_LEN + SERVE_MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN" +else + SERVE_MAX_MODEL_LEN="$MAX_MODEL_LEN" +fi + +# Start GPU monitoring (power, temperature, clocks every second) +start_gpu_monitor + +set -x +vllm serve "$MODEL" --host 0.0.0.0 --port "$PORT" \ + --trust-remote-code \ + "${REVISION_ARGS[@]}" \ + --kv-cache-dtype fp8 \ + --block-size 256 \ + --moe-backend deep_gemm_mega_moe \ + --attention_config.use_fp4_indexer_cache=True \ + --tokenizer-mode deepseek_v4 \ + --tool-call-parser deepseek_v4 \ + --enable-auto-tool-choice \ + --reasoning-parser deepseek_v4 \ + "${PARALLEL_ARGS[@]}" \ + "${EP_ARGS[@]}" \ + --max-model-len "$SERVE_MAX_MODEL_LEN" \ + --max-num-batched-tokens 8192 \ + --max-num-seqs 256 \ + --gpu-memory-utilization 0.95 > "$SERVER_LOG" 2>&1 & + +SERVER_PID=$! + +# Wait for server to be ready +wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" + +pip install -q datasets pandas + +run_benchmark_serving \ + --model "$MODEL" \ + --port "$PORT" \ + --backend vllm \ + --input-len "$ISL" \ + --output-len "$OSL" \ + --random-range-ratio "$RANDOM_RANGE_RATIO" \ + --num-prompts "$((CONC * 10))" \ + --max-concurrency "$CONC" \ + --result-filename "$RESULT_FILENAME" \ + --result-dir /workspace/ \ + --trust-remote-code + +# After throughput, run evaluation only if RUN_EVAL is true +if [ "${RUN_EVAL}" = "true" ]; then + run_eval --framework lm-eval --port "$PORT" + append_lm_eval_summary +fi + +# Stop GPU monitoring +stop_gpu_monitor +set +x diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index 393babf91b..bc304d39be 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -1017,6 +1017,33 @@ dsv4-fp4-b200-vllm-mtp: - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256, spec-decoding: mtp } +# DeepSeek-V4-Flash-0731: first Flash-on-B200 key. Both arms validated on +# Nebius 8x B200 SXM with vllm/vllm-openai:v0.25.0 (vllm-project/vllm#51454). +# Model snapshot 7872f01b1d1fe23eabc4c98b48bffcef5a386062 is pinned inside +# benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh (MODEL_REVISION). +# TP8+EP8 is the single-stream/latency arm (break-even vs DP8 at c=3 in the +# report's 1M ladder); DP8 (attention DP + EP8) is the concurrency arm. Both +# arms carry ep: 8 because vLLM v0.25.0's MegaMoE backend refuses engine init +# without expert parallel (NotImplementedError before weight load). +dsv4f-fp8-b200-vllm: + image: vllm/vllm-openai:v0.25.0 + model: deepseek-ai/DeepSeek-V4-Flash-0731 + model-prefix: dsv4f + runner: cluster:b200-nb + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # TP8 + EP8, single-stream to mid concurrency + - { tp: 8, ep: 8, conc-start: 1, conc-end: 64 } + # DP8 (attention DP) + EP8, concurrency/throughput arm; capped by + # --max-num-seqs 256 in the script + - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256 } + # NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/DeepSeek/DeepSeek-R1 # does not have a B300-specific recipe, so this config reuses the existing DSR1 FP4 # B200 SGLang recipe as-is until B300-specific tuning is available. diff --git a/configs/runners.yaml b/configs/runners.yaml index 6f0b35e9c0..a581d77e85 100644 --- a/configs/runners.yaml +++ b/configs/runners.yaml @@ -243,6 +243,9 @@ labels: - b200-dgxc_07 - b200-dgxc_08 - b200-dgxc_09 + cluster:b200-nb: + - b200-nb_0 + - b200-nb_1 cluster:b300-cw: - b300-cw_0 - b300-cw_01 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 641d92ce58..65f619ec73 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5697,3 +5697,11 @@ - "Use the official lmsysorg/sglang:v0.5.16-rocm720-mi30x image, enable SGLang prompt-cache reporting, and pass the backend Prometheus metrics endpoint explicitly to AIPerf" - "Allow up to 30 minutes for healthy AgentX responses admitted near the end of the measurement window to drain before AIPerf finalizes profile metric coverage" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2528 + +- config-keys: + - dsv4f-fp8-b200-vllm + description: + - "Add DeepSeek-V4-Flash-0731 FP8 B200 vLLM single-node 8k1k: the first Flash-on-B200 config key, on the new cluster:b200-nb runner label (Nebius fleet), routed to benchmarks/single_node/fixed_seq_len/dsv4f_fp8_b200_vllm.sh. Image vllm/vllm-openai:v0.25.0 (digest sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97, vLLM build dd10e03f95f94edbea1975c67ace3a35ec9a8a40); model snapshot 7872f01b1d1fe23eabc4c98b48bffcef5a386062 pinned inside the script via MODEL_REVISION." + - "Recipe flags are the TP8/DP8 recipe validated on 8x B200 and reported upstream in vllm-project/vllm#51454: --kv-cache-dtype fp8, --block-size 256, --moe-backend deep_gemm_mega_moe, --attention_config.use_fp4_indexer_cache=True, deepseek_v4 tokenizer / tool-call / reasoning parsers, --max-num-batched-tokens 8192, --max-num-seqs 256, --gpu-memory-utilization 0.95. Both arms run expert parallel because vLLM v0.25.0's MegaMoE backend refuses engine init without EP (NotImplementedError before weight load). TP8 covers conc 1-64 (single-stream arm; DP8 break-even at c=3 in the report's 1M ladder), DP8 (attention DP + EP8) covers conc 64-256, capped by --max-num-seqs. Prefix caching is left at the engine default: the report shows it intact on Flash (22x warm-TTFT speedup on a re-sent 1M document)." + - "MODELS.md gains the DeepSeek-V4-Flash-0731 row (prefix dsv4f); runners.yaml gains cluster:b200-nb (b200-nb_0/1)." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2535