Skip to content

[AMD] kimik2.5-fp4-mi355x-vllm-disagg: split P/D by concurrency (1D low, 2D high) / 优化 kimik2.5-fp4-mi355x-vllm-disagg：按并发拆分 P/D 拓扑（低并发 1D、高并发 2D） #2301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adibarra merged 2 commits into from
Jul 22, 2026
Merged

[AMD] kimik2.5-fp4-mi355x-vllm-disagg: split P/D by concurrency (1D low, 2D high) / 优化 kimik2.5-fp4-mi355x-vllm-disagg：按并发拆分 P/D 拓扑（低并发 1D、高并发 2D） #2301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6bbf17e
[AMD] kimik2.5-fp4-mi355x-vllm-disagg: split P/D by concurrency (1D l…
hongxiayang Jul 22, 2026
f930bc0
Merge branch 'main' into hy/kimi-fp4-vllm-disagg2
seungrokj Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/multi_node/amd_utils/job.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ export DOCKER_CONT_NAME="container_${ENGINE}_${SANITIZED_USER}_${MODEL_NAME}_${S
# vLLM external router container.
# NOTE: vllm/vllm-router only retains ~16 recent nightlies on Docker Hub; older
# dated tags are garbage-collected (manifest unknown)
VLLM_ROUTER_IMAGE="${VLLM_ROUTER_IMAGE:-vllm/vllm-router:nightly-20260629-e667ebb}"
VLLM_ROUTER_IMAGE="${VLLM_ROUTER_IMAGE:-vllm/vllm-router:nightly-20260716-1fbcde7}"
ROUTER_CONT_NAME="router_vllm_${SANITIZED_USER}_${SLURM_JOB_ID}"
# Separate agentic benchmark-client container (see CLIENT_IMAGE handling below).
CLIENT_CONT_NAME="container_${ENGINE}_${SANITIZED_USER}_client_${SLURM_JOB_ID}"
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/multi_node/amd_utils/models_vllm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ amd-Llama-3.3-70B-Instruct-FP8-KV:
env: "VLLM_USE_V1=1 VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 AMDGCN_USE_BUFFER_OPS=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_RMSNORM=1 VLLM_USE_AITER_TRITON_ROPE=1 TRITON_HIP_ASYNC_COPY_BYPASS_PERMUTE=1 TRITON_HIP_USE_ASYNC_COPY=1 TRITON_HIP_USE_BLOCK_PINGPONG=1 TRITON_HIP_ASYNC_FAST_SWIZZLE=1"

Kimi-K2.5-MXFP4:
prefill_flags: "--tensor-parallel-size 8 --compilation-config '{\"cudagraph_mode\":\"PIECEWISE\"}' --no-enable-prefix-caching --block-size 1 --gpu-memory-utilization 0.90 --mm-encoder-tp-mode data"
decode_flags: "--tensor-parallel-size 8 --enable-expert-parallel --all2all-backend mori_low_latency --compilation-config '{\"cudagraph_mode\":\"PIECEWISE\"}' --no-enable-prefix-caching --block-size 1 --gpu-memory-utilization 0.90 --mm-encoder-tp-mode data"
env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_PAGED_ATTN=0 VLLM_ROCM_USE_AITER_RMSNORM=1 VLLM_USE_AITER_TRITON_SILU_MUL=0 VLLM_ENGINE_READY_TIMEOUT_S=3600"
prefill_flags: "--tensor-parallel-size 8 --no-enable-prefix-caching --block-size 1 --gpu-memory-utilization 0.90 --max-model-len 32768 --mm-encoder-tp-mode data --kv-cache-dtype fp8 --max-num-seqs 256 --max-num-batched-tokens 32768"
decode_flags: "--tensor-parallel-size 8 --all2all-backend mori_low_latency --no-enable-prefix-caching --block-size 1 --gpu-memory-utilization 0.90 --max-model-len 32768 --mm-encoder-tp-mode data --kv-cache-dtype fp8 --max-num-seqs 256 --max-num-batched-tokens 32768"
env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_RMSNORM=1 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 HSA_NO_SCRATCH_RECLAIM=1 VLLM_ENGINE_READY_TIMEOUT_S=3600"
Comment on lines +28 to +30

@claude claude Bot Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The disagg recipe now runs Kimi-K2.5-MXFP4 at TP4 (down from TP8) via configs/amd-master.yaml, but benchmarks/multi_node/amd_utils/models_vllm.yaml:28-30 still hardcodes VLLM_ROCM_USE_AITER_RMSNORM=1 unconditionally with no per-TP branching. The sibling single-node recipe (benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_mi355x.sh:51-54) explicitly disables AITER RMSNorm for TP<8 due to known accuracy issues, so this multi-node recipe now silently ships a numerically-degraded config since the fixed-seq-len sweep only measures throughput. Fix: disable VLLM_ROCM_USE_AITER_RMSNORM when TP<8, matching the single-node guard.

Extended reasoning...

What the bug is

configs/amd-master.yaml's kimik2.5-fp4-mi355x-vllm-disagg recipe is changed by this PR from all-TP8 prefill/decode to all-TP4 for both new topologies (1P/1D and 1P/2D). However, benchmarks/multi_node/amd_utils/models_vllm.yaml:28-30 (the Kimi-K2.5-MXFP4 entry) still statically sets env: "... VLLM_ROCM_USE_AITER_RMSNORM=1 ..." with no conditional logic based on tensor-parallel size. That env string is exported verbatim for every worker regardless of TP.

Why this matters — the sibling recipe already flags this exact hazard

benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_mi355x.sh:51-54 serves the identical model (amd/Kimi-K2.5-MXFP4) and explicitly guards against this:

# Disable AITER RMSNorm for TP < 8 due to accuracy issues
if [ "${TP}" -lt 8 ]; then
  export VLLM_ROCM_USE_AITER_RMSNORM=0
fi

This is direct, authoritative evidence from AMD/the repo's own single-node recipe that the AITER RMSNorm kernel has a known numerical-accuracy problem at TP<8 for this specific model/hardware combination. The multi-node models_vllm.yaml config has no equivalent branching — it is a flat, static env string.

The code path that triggers it

server_vllm.sh loads the env field from models_vllm.yaml into MODEL_ENVS and its setup_vllm_env() does a simple for env_pair in ${MODEL_ENVS}; do export "$env_pair"; done — there is no TP-conditional logic anywhere in the multi-node vLLM path. The same script sed-rewrites --tensor-parallel-size to PREFILL_TP_SIZE/DECODE_TP_SIZE, which submit.sh computes as PREFILL_NODES*PREFILL_TP/PREFILL_WORKERS. With the new master config (tp: 4 for both prefill and decode, num-worker: 1 for prefill, num-worker: 1 or 2 for decode), every worker in both new topologies genuinely runs at TP4 — e.g. 1*4/1 = 4 for prefill, and 1*4/1 = 4 (1D) or 2*4/2 = 4 (2D) for decode. So this recipe now lands squarely in the TP<8 zone the single-node script deliberately avoids, but with AITER RMSNorm turned on.

Why nothing catches it

The fixed-seq-len scenario is a throughput-only sweep (RUN_EVAL defaults to false, and this PR makes no eval-harness changes). Accuracy is never measured, so a numerically-degraded serving config would still produce green throughput numbers and merge/publish silently.

Step-by-step proof

  1. Before this PR: kimik2.5-fp4-mi355x-vllm-disagg ran prefill and decode both at TP8 (old master config: tp: 8 for both worker rows). At TP8, AITER RMSNorm is fine per the sibling script's own guard (if TP<8 disable; TP8 is not <8, so it stays enabled — the known-good case).
  2. This PR changes the master config: prefill tp: 4, decode tp: 4 in both search-space rows (1P/1D and 1P/2D).
  3. submit.sh computes PREFILL_TP_SIZE = PREFILL_NODES*PREFILL_TP/PREFILL_WORKERS = 1*4/1 = 4, DECODE_TP_SIZE = 1*4/1 = 4 or 2*4/2 = 4. Every worker really runs at TP4.
  4. models_vllm.yaml's Kimi-K2.5-MXFP4.env still unconditionally sets VLLM_ROCM_USE_AITER_RMSNORM=1; setup_vllm_env exports it as-is, no TP check.
  5. Result: TP4 + AITER RMSNorm enabled — exactly the combination the single-node recipe's comment says causes accuracy issues, newly introduced by this PR's TP8→TP4 switch.
  6. The fixed-seq-len sweep only measures tok/s/latency, so this ships green with degraded output quality unmeasured.

How to fix

Add the same per-TP guard used in the single-node script — either branch in server_vllm.sh's setup_vllm_env (or equivalent) to unset VLLM_ROCM_USE_AITER_RMSNORM when the resolved TP is <8, or simply drop VLLM_ROCM_USE_AITER_RMSNORM=1 from the Kimi-K2.5-MXFP4 yaml env now that this recipe runs exclusively at TP4. This is a one-line fix and directly mirrors precedent already in the codebase.

Addressing the "is this just a PR-description mismatch" concern

This is not a stale-description nit — the code itself demonstrably runs a config (TP4 + RMSNorm=1) that the repo's own sibling script treats as a documented accuracy hazard, independent of anything the PR description claims. The PR description even states intent to "sync per-worker vLLM serve flags/env with the single-node recipe" — this is the one flag that was supposed to be synced but was missed.

hf_dir: "models--amd--Kimi-K2.5-MXFP4"

MiniMax-M2.5:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/multi_node/amd_utils/submit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ fi
# Optional: exclude specific nodes (e.g. nodes with broken Docker sockets).
# Set SLURM_EXCLUDE_NODES env var to a comma-separated list of hostnames.
EXCLUDE_OPT=()
SLURM_EXCLUDE_NODES="${SLURM_EXCLUDE_NODES:-mia1-p01-g11,mia1-p01-g12,mia1-p01-g15}"
SLURM_EXCLUDE_NODES="${SLURM_EXCLUDE_NODES:-mia1-p01-g09,mia1-p01-g11,mia1-p01-g12,mia1-p01-g14,mia1-p01-g15}"
if [[ -n "${SLURM_EXCLUDE_NODES:-}" ]]; then
EXCLUDE_OPT=(--exclude "$SLURM_EXCLUDE_NODES")
fi
Expand Down
29 changes: 25 additions & 4 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -798,7 +798,7 @@ dsr1-fp8-mi355x-sglang-disagg-mtp:
- "DECODE_MTP_SIZE=2"

kimik2.5-fp4-mi355x-vllm-disagg:
image: vllm/vllm-openai-rocm:v0.24.0
image: vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9
model: amd/Kimi-K2.5-MXFP4
model-prefix: kimik2.5
runner: mi355x-disagg
Expand All @@ -810,22 +810,43 @@ kimik2.5-fp4-mi355x-vllm-disagg:
disagg: true
scenarios:
fixed-seq-len:
# All workers TP4 (real-weight sweep: TP8 decode is no better than TP4).
# Split across P/D topologies: 1D is stable at low conc; the high-conc tail
# runs on 2D so decode load is spread across two engines.
- isl: 8192
osl: 1024
search-space:
# 1P(TP4) 1D(TP4) = 2 nodes. Low concurrency only.
- spec-decoding: "none"
conc-list: [ 1, 2, 4 ]
prefill:
num-worker: 1
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 1
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "DECODE_NODES=1"
# 1P(TP4) 2D(TP4) = 3 nodes. High concurrency (decode KV/load spread over 2D).
- spec-decoding: "none"
conc-list: [ 8, 16, 32, 64, 128, 256, 512 ]
prefill:
num-worker: 1
tp: 8
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 2
tp: 8
ep: 8
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "DECODE_NODES=2"
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5049,3 +5049,14 @@
- "Topologies: 1p4d-dep4-tep8 (conc 4/24), 1p5d-dep4-tep4 (conc 5/30/60/115/195), 1p1d-dep4-dep8 (conc 308), 2p1d-dep4-dep8 (conc 615), 3p1d-dep4-dep8 (conc 1127), 4p1d-dep4-dep8 (conc 2151)"
- "Runner updated to clone srt-slurm at sa-submission-q2-2026 and copy local recipes into recipes/trtllm/kimi-k25-nvfp4/b200-fp4/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2249

- config-keys:
- kimik2.5-fp4-mi355x-vllm-disagg
description:
- "Bump image to vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9"
- "All-TP4 prefill/decode (drop TP8): a real-weight sweep showed TP8 decode is no better than TP4 on tok/s/GPU; expert parallelism off (ep:1, single-node TP8 sweep showed EP -14% to -27% slower than dense)"
- "Split the 8k/1k P/D topology by concurrency: low conc (1,2,4) on 1P(TP4)/1D(TP4); high conc (8,16,32,64,128,256,512) on 1P(TP4)/2D(TP4) so decode load/KV is spread across two decode engines (the single-decode engine is unstable in the high-conc tail)"
- "Sync per-worker vLLM serve flags/env with the single-node recipe: --kv-cache-dtype fp8, --max-model-len 32768, --max-num-seqs 256, --max-num-batched-tokens 32768; VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4, HSA_NO_SCRATCH_RECLAIM=1, AITER defaults; drop the --compilation-config PIECEWISE pin (use vLLM default FULL_AND_PIECEWISE)"
- "Re-pin VLLM_ROUTER_IMAGE to vllm/vllm-router:nightly-20260716-1fbcde7 (previous nightly-20260629-e667ebb was garbage-collected from Docker Hub)"
- "Exclude known-bad nodes mia1-p01-g09,g14 from the disagg node pool"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2301