Skip to content

[Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
functionstackx wants to merge 11 commits into from
Closed

[Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b4fd077
feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic r…
functionstackx Jul 27, 2026
4dbbdc8
docs: link PR #2355 in changelog entry and MODELS rows
functionstackx Jul 27, 2026
e8d42a7
fix: pin agentic srt-slurm to NVIDIA v1.0.36, dynamo 1.2.1, conc-8 sm…
functionstackx Jul 27, 2026
c1e2a56
fix: drop OpenAI-frontend tool-choice flags from dynamo-vllm worker args
functionstackx Jul 27, 2026
ef35fd1
fix: try dynamo wheel 1.2.0.dev20260426 for Kimi-K3 frontend tokenizer
functionstackx Jul 27, 2026
be6c56e
fix: pin dynamo to day-zero Kimi-K3 commit, restore kimi_k3 parser flags
functionstackx Jul 27, 2026
c6917e6
fix: use dynamo namespaced --dyn-* kimi_k3 parser args on the worker
functionstackx Jul 27, 2026
a26853a
fix: disable aiperf conv-aware routing (session_control 400-rejected)
functionstackx Jul 27, 2026
f61eafb
fix: patch kimi-k3 image mamba_hybrid index_fill_ dtype via setup_script
functionstackx Jul 27, 2026
fff449c
fix: drop gpu-memory-utilization to 0.90 (flashinfer MoE workspace OOM)
functionstackx Jul 27, 2026
73f7e63
fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM)
functionstackx Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
name: "kimik3-vllm-agg-b200-tp8pp2-agentic"

# Kimi-K3 MXFP4 B200 AGGREGATED TP8 x PP2 agentic recipe (2 nodes / 16 GPUs).
# The native MXFP4 checkpoint (2.8T total params, ~1.4TB of weights) does not
# fit one 8xB200 node, so TP8 shards attention/dense (/8) and PP2 splits the
# 93 layers (/2) across 16 GPUs. Plain TP (NOT TEP): expert parallelism is
# deliberately off, so the 896 routed experts are TP-sharded inside each
# pipeline stage. Node allocation = tp*pp/gpus_per_node = 8*2/8 = 2 nodes.
# Aggregated (single worker, decode num-worker 0) — no P/D split, no NIXL.
# VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION fuses the K3 LatentMoE tail path in
# the kimi-k3 bring-up image.
model:
path: "kimik3"
container: "vllm/vllm-openai:kimi-k3"
precision: "fp4"

identity:
model:
repo: "moonshotai/Kimi-K3"
container:
image: "vllm/vllm-openai:kimi-k3"
frameworks:
dynamo: "1.3.0.dev20260618"

dynamo:
install: true
wheel: "1.3.0.dev20260618"

slurm:
time_limit: "8:00:00"

health_check:
interval_seconds: 10
max_attempts: 1440

Check failure on line 34 in benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml

View check run for this annotation

Claude / Claude Code Review

Launcher clobbers recipe's 4h health-check timeout back to 2h 

The recipe deliberately sets `health_check.max_attempts: 1440` (4h) for the ~1.4TB Kimi-K3 MXFP4 checkpoint, but `runners/launch_b200-dgxc.sh:291` unconditionally runs `sed -i 's/^  max_attempts: [0-9]*/  max_attempts: 720/'` on the copied config right before `srtctl apply`, silently clobbering it back to 720 attempts (2h) — the same budget sized for DSR1-FP8 at roughly half this checkpoint's weight size. The recipe file will look correctly configured but the wider window never actually takes ef
Comment on lines +43 to +44

@claude claude Bot Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The recipe deliberately sets health_check.max_attempts: 1440 (4h) for the ~1.4TB Kimi-K3 MXFP4 checkpoint, but runners/launch_b200-dgxc.sh:291 unconditionally runs sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' on the copied config right before srtctl apply, silently clobbering it back to 720 attempts (2h) — the same budget sized for DSR1-FP8 at roughly half this checkpoint's weight size. The recipe file will look correctly configured but the wider window never actually takes effect at runtime; either make the sed a floor (only raise, never lower) or special-case kimik3 like the other model-prefix branches nearby.

Extended reasoning...

The bug: runners/launch_b200-dgxc.sh line 291 runs:

sed -i 's/^  max_attempts: [0-9]*/  max_attempts: 720/' "${CONFIG_FILE%%:*}"

right before srtctl apply -f "$CONFIG_FILE" inside the IS_MULTINODE branch. This is an unconditional hard-set, not a max()/floor operation — whatever numeric value follows max_attempts: (two-space indent) in the config file gets overwritten to exactly 720, no matter what it was before.

The interaction this PR introduces: the new recipe benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml sets:

health_check:
  interval_seconds: 10
  max_attempts: 1440

with a two-space indent that matches the sed's regex exactly. The recipe's own comments make clear this 1440 (= 4h at 10s/attempt) was deliberately sized up from the launcher's usual value, specifically because the native MXFP4 checkpoint is ~1.4TB and has to be pulled over shared Lustre across 2 nodes. But because the sed is unconditional, srtctl apply never sees 1440 — it sees the file after the sed has already rewritten it to 720 (= 2h). The comment directly above the sed ("Bump recipe health-check timeout from 360×10s to 720×10s so large-model loads ... finish in time") shows the sed was written with the mental model of bumping the common 360 default up to 720; it never anticipated a recipe explicitly setting a value above 720, so for this recipe the sed is actually a reduction, not a bump.

Why nothing else catches this: there's no validation step between writing the recipe's health_check block and srtctl apply that would surface the discrepancy — the sed runs silently, will echo the sed invocation but not diff the before/after content of the file, and the recipe file committed to the repo (and reviewed in this PR) legitimately shows 1440. Someone reading the recipe or this diff would have no way to know the value gets overwritten at runtime.

Step-by-step proof:

  1. CONFIG_FILE resolves to recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml (per the configs/nvidia-master.yaml entry's additional-settings: CONFIG_FILE=...), a copy of this PR's new recipe overlaid into the srt-slurm-nv clone.
  2. That file, as written in this PR, contains the line max_attempts: 1440.
  3. At runtime, line ~289 first does sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" (unrelated), then line 291 does sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}".
  4. The regex ^ max_attempts: [0-9]* matches max_attempts: 1440 (2-space indent, then digits), so the line becomes max_attempts: 720.
  5. srtctl apply -f "$CONFIG_FILE" runs immediately after, reading the now-mutated file — it applies max_attempts: 720, interval_seconds: 10 → a 7200s (2h) health-check window, not the 14400s (4h) the recipe declares.

Impact: if 2h genuinely isn't enough to pull ~1.4TB of MXFP4 weights and bring the engine up (workers loading in parallel off shared Lustre, contended by whatever else is running on the same filesystem), the health check gives up and fails the entire multinode job — the exact scenario this recipe's author was trying to prevent by widening the window, on the largest checkpoint in the fleet and its first-ever bring-up. Because the recipe file looks correct, anyone debugging a health-check failure here would have to know to check the launcher script rather than trusting the recipe as documentation of what will actually run — a debugging detour on a bring-up PR that's likely to already have plenty of other failure modes to sort through.

Fix: make the sed a floor instead of a hard-set, e.g. only replace when the existing value is less than 720 (or compare-and-max in the shell), or skip the sed entirely for kimik3/any recipe that already sets a larger value, similar to how other model prefixes get their own branch in this same script.


resources:
gpu_type: "b200"
gpus_per_node: 8
agg_nodes: 2
agg_workers: 1
gpus_per_agg: 16

infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 32

frontend:
type: dynamo
enable_multiple_frontends: false

backend:
type: vllm
connector: null
aggregated_environment:
VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION: "1"
VLLM_SERVER_DEV_MODE: "1"
# ~1.4TB of MXFP4 weights off shared Lustre: keep the engine-ready window
# generous, and let one long AgentX request hold a PP stage beyond vLLM's
# 300-second model-execution default.
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800"
NCCL_CUMEM_ENABLE: "1"
TILELANG_CLEANUP_TEMP_FILES: "1"
UCX_MEMTYPE_CACHE: "n"
UCX_MEMTYPE_REG_WHOLE: "n"
UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_10:1,mlx5_11:1"
HF_HUB_CACHE: "/hf_hub_cache"
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
vllm_config:
aggregated:
served-model-name: "moonshotai/Kimi-K3"
tensor-parallel-size: 8
pipeline-parallel-size: 2
trust-remote-code: true
load-format: fastsafetensors
moe-backend: auto
gpu-memory-utilization: 0.95
no-enable-flashinfer-autotune: true
enable-auto-tool-choice: true
tool-call-parser: kimi_k3
reasoning-parser: kimi_k3
# Full native 1M context: the AgentX trace carries multi-hundred-k-token
# requests, and K3's KDA layers keep per-token KV small (only the 24
# gated-MLA layers hold cache). Cap prefill chunks so a single long
# request cannot OOM a pipeline stage; let vLLM pick max-num-seqs.
max-model-len: 1048576
max-num-batched-tokens: 8192

sbatch_directives:
segment: "1"

srun_options:
container-remap-root: ""

benchmark:
type: custom
aiperf_server_metrics: true
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
# Keep the aggregate worker in the multinode result schema so ingestion
# uses the zero decode-worker count instead of duplicating TP into P and D.
IS_MULTINODE: "true"
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400"
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
HF_HUB_CACHE: "/hf_hub_cache"
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"
43 changes: 43 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8325,3 +8325,46 @@ qwen3.5-fp8-gb200-dynamo-sglang-mtp:
tp: 16
ep: 16
dp-attn: true

# Kimi-K3 MXFP4 B200 aggregated vLLM via Dynamo (TP8 x PP2, 2 nodes / 16
# GPUs), agentic bring-up. The native MXFP4 checkpoint (2.8T total params,
# ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense
# and PP2 splits layers. Plain TP (NOT TEP): ep 1, no expert parallelism —
# the 896 routed experts are TP-sharded within each pipeline stage. Node
# count = tp*pp/gpus_per_node = 8*2/8 = 2. Aggregated (prefill num-worker 1 +
# decode num-worker 0, RECIPES.md section 5) — the single worker serves both
# phases, so no P/D KV transfer. Dedicated kimi-k3 vLLM bring-up image with
# VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 and the kimi_k3 tool-call/reasoning
# parsers.
# Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
kimik3-fp4-b200-dynamo-vllm-agentic:
image: vllm/vllm-openai:kimi-k3
model: moonshotai/Kimi-K3
model-prefix: kimik3
runner: cluster:b200-dgxc
precision: fp4
framework: dynamo-vllm
router: { name: dynamo-router, version: "1.3.0.dev20260618" }
multinode: true
disagg: false
scenarios:
agentic-coding:
- search-space:
- spec-decoding: none
conc-list: [8, 16, 32]
prefill:
num-worker: 1
tp: 8
pp: 2
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml"
# The aggregate worker also performs decode; keep the decode worker
# count at zero so result aggregation counts the 16 GPUs only once.
decode:
num-worker: 0
tp: 8
pp: 2
ep: 1
dp-attn: false
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5116,3 +5116,14 @@
description:
- "Bump image from lmsysorg/sglang:v0.5.14-rocm720-mi35x to lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260726"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2349

- config-keys:
- kimik3-fp4-b200-dynamo-vllm-agentic
description:
- "Add Kimi-K3 MXFP4 B200 aggregated multinode Dynamo-vLLM agentic-coding bring-up (new model on B200; first kimik3 benchmark config)"
- "Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (NOT TEP: ep 1, no enable-expert-parallel) — the native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense and PP2 splits the 93 layers. Aggregated mode (prefill num-worker 1 + decode num-worker 0, RECIPES.md section 5): one worker serves prefill and decode, no P/D KV transfer"
- "Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 (fuses the K3 LatentMoE tail path), --load-format fastsafetensors, --moe-backend auto, --gpu-memory-utilization 0.95, --no-enable-flashinfer-autotune, --enable-auto-tool-choice, --tool-call-parser kimi_k3, --reasoning-parser kimi_k3, --trust-remote-code"
- "Full native 1M max-model-len for the AgentX trace (K3's KDA layers keep per-token KV small; only the 24 gated-MLA layers hold cache), max-num-batched-tokens 8192 so a single long prefill cannot OOM a pipeline stage; conc 8/16/32"
- "Model pre-staged at /lustre/fsw/models/Kimi-K3 (moonshotai/Kimi-K3); launcher launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay into the srt-slurm-nv clone, and the agentic default_mounts (/aiperf_mmap_cache, /hf_hub_cache) already used by the GB200/GB300 agentic paths"
- "Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml on the cluster:b200-dgxc pool"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
28 changes: 28 additions & 0 deletions runners/launch_b200-dgxc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ elif [[ $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" ]]; then
# NVFP4 checkpoint, pre-staged on the b200-dgxc scratch tree.
export MODEL_PATH="/scratch/fsw/models/MiniMax-M3-NVFP4"
export SRT_SLURM_MODEL_PREFIX="minimax-m3-nvfp4"
elif [[ $MODEL_PREFIX == "kimik3" && $PRECISION == "fp4" ]]; then
# Native MXFP4 checkpoint, pre-staged on the SRE-managed Lustre tree.
export MODEL_PATH="/lustre/fsw/models/Kimi-K3"
export SRT_SLURM_MODEL_PREFIX="kimik3"
else
echo "Unsupported model prefix/precision: $MODEL_PREFIX/$PRECISION"
echo "Available models under /lustre/fsw/models:"
Expand Down Expand Up @@ -107,6 +111,13 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
if [[ "$IS_AGENTIC" == "1" ]]; then
git clone --branch cam/sa-submission-q2-2026 --single-branch https://github.com/cquil11/srt-slurm-nv.git "$SRT_REPO_DIR"
cd "$SRT_REPO_DIR" || exit 1
# Overlay InferenceX-staged agentic recipes onto the clone (cp -rT so
# an upstream stub directory is merged rather than nested).
if [[ $MODEL_PREFIX == "kimik3" ]]; then
mkdir -p recipes/vllm/kimi-k3/agentic
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic" \
recipes/vllm/kimi-k3/agentic
fi
elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "dsv4" ]]; then
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
cd "$SRT_REPO_DIR" || exit 1
Expand Down Expand Up @@ -207,6 +218,22 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
export OSL="$OSL"
export EVAL_ONLY="${EVAL_ONLY:-false}"

# Agentic runs bind-mount two persistent caches into every worker
# container (Lustre, shared across nodes): aiperf's content-addressed
# dataset mmap cache and the HF hub cache holding the trace dataset
# download. The container-side paths are referenced by the agentic
# recipes' benchmark.env (AIPERF_DATASET_MMAP_CACHE_DIR=/aiperf_mmap_cache,
# HF_HUB_CACHE=/hf_hub_cache).
DEFAULT_MOUNTS_BLOCK=""
if [[ "$IS_AGENTIC" == "1" ]]; then
HF_HUB_CACHE_HOST_PATH="/lustre/fsw/gharunners/hf-hub-cache"
mkdir -p "$AIPERF_MMAP_CACHE_HOST_PATH" "$HF_HUB_CACHE_HOST_PATH"
chmod 777 "$AIPERF_MMAP_CACHE_HOST_PATH" "$HF_HUB_CACHE_HOST_PATH" 2>/dev/null || true
DEFAULT_MOUNTS_BLOCK="default_mounts:
${AIPERF_MMAP_CACHE_HOST_PATH}: /aiperf_mmap_cache
${HF_HUB_CACHE_HOST_PATH}: /hf_hub_cache"
fi

# Create srtslurm.yaml for srtctl (used by both frameworks)
SRTCTL_ROOT="${GITHUB_WORKSPACE}/${SRT_REPO_DIR}"
echo "Creating srtslurm.yaml configuration..."
Expand Down Expand Up @@ -234,6 +261,7 @@ containers:
"${IMAGE}": "${SQUASH_FILE}"
nginx-sqsh: "${NGINX_SQUASH_FILE}"
use_exclusive_sbatch_directive: true
${DEFAULT_MOUNTS_BLOCK}
EOF

echo "Generated srtslurm.yaml:"
Expand Down