Skip to content

[CI][PD] Add optional/nightly DSv4 Disaggregated eval #42310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NickLucche merged 7 commits into from
Jul 15, 2026
+31 −5
Merged

[CI][PD] Add optional/nightly DSv4 Disaggregated eval #42310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0ac163
accuracy value ds
NickLucche May 11, 2026
a23e34c
wd
NickLucche May 11, 2026
2a298aa
increase timeout
NickLucche May 14, 2026
3a93bf9
change device to h100
NickLucche May 14, 2026
bb94e05
oom
NickLucche May 15, 2026
592333e
fix gpu assignment
NickLucche Jul 13, 2026
ffa5d1d
Merge branch 'main' into dsv4-disagg-test2
mergify[bot] Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .buildkite/test_areas/disaggregated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,3 +203,25 @@ steps:
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh

# P TP 4 - D DPEP 4 test case for DSv4-Flash
- label: DSv4-Flash Disaggregated DP EP
key: dsv4-flash-disaggregated
timeout_in_minutes: 60
device: h200
optional: true
working_dir: "/vllm-workspace/tests"
num_devices: 8
env:
ENABLE_HMA_FLAG: "1"
DP_EP: "1"
GPU_MEMORY_UTILIZATION: "0.85"
PREFILLER_TP_SIZE: "4"
DECODER_TP_SIZE: "4"
PREFILL_BLOCK_SIZE: "256"
DECODE_BLOCK_SIZE: "256"
MODEL_NAMES: "deepseek-ai/DeepSeek-V4-Flash"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--kv-cache-dtype,fp8"
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- bash v1/kv_connector/nixl_integration/run_accuracy_test.sh
Comment on lines +208 to +227

@gemini-code-assist gemini-code-assist Bot May 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The new test step is missing the source_file_dependencies block. This is inconsistent with all other steps in this file and is important for the CI system to correctly track changes and manage cache invalidation for this test.

- label: DSv4-Flash Disaggregated DP EP
  key: dsv4-flash-disaggregated
  timeout_in_minutes: 30
  device: h200
  optional: true
  working_dir: "/vllm-workspace/tests"
  num_devices: 8
  source_file_dependencies:
    - vllm/distributed/kv_transfer/kv_connector/v1/nixl/
    - tests/v1/kv_connector/nixl_integration/
  env:
    ENABLE_HMA_FLAG: "1"
    DP_EP: "1"
    GPU_MEMORY_UTILIZATION: "0.85"
    PREFILLER_TP_SIZE: "4"
    DECODER_TP_SIZE: "4"
    PREFILL_BLOCK_SIZE: "256"
    DECODE_BLOCK_SIZE: "256"
    MODEL_NAMES: "deepseek-ai/DeepSeek-V4-Flash"
    VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--kv-cache-dtype,fp8"
  commands:
    - uv pip install --system -r /vllm-workspace/requirements/kv_connectors.txt
    - bash v1/kv_connector/nixl_integration/run_accuracy_test.sh

Comment on lines +206 to +227

@claude claude Bot May 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new TP=4/4 configuration is the first call site of run_accuracy_test.sh to exercise its inner GPU-ID build loop more than once, exposing a latent Bash bug at tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh:142-204. Once GPU_ID becomes a comma-separated string, $(((GPU_ID + j) % NUM_GPUS)) triggers Bash's C-style comma operator, so prefill ends up with CUDA_VISIBLE_DEVICES=0,1,3,6 and decode with 7,0,2,5 — both collide on GPU 0 at gpu-memory-utilization=0.85 and the optional DSv4 step will OOM rather than produce accuracy numbers. Fix in run_accuracy_test.sh: accumulate GPU IDs in a Bash array and stringify only at the end, and reset NEXT_GPU between the prefill and decode loops (this is pre-existing, but only this PR triggers it).

Extended reasoning...

Bug

The optional DSv4-Flash CI step added here is the first call site of tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh to use PREFILLER_TP_SIZE / DECODER_TP_SIZE > 2. That configuration exposes a latent Bash bug in the script that builds CUDA_VISIBLE_DEVICES for each prefill/decode instance.

The relevant code (lines 142-150 for prefill, mirrored at 197-204 for decode) is:

GPU_ID=$((i % $(get_num_gpus)))
NEXT_GPU=${GPU_ID}
for (( j=1; j < PREFILLER_TP_SIZE; j++ )); do
  NEXT_GPU=$(((GPU_ID + j) % $(get_num_gpus)))
  GPU_ID="${GPU_ID},${NEXT_GPU}"
done

After the first iteration GPU_ID is a comma-separated string (e.g. 0,1). On the next iteration the arithmetic expansion $(((GPU_ID + j) % N)) evaluates GPU_ID + j inside Bash arithmetic, where comma is the lowest-precedence C-style comma operator. The expression evaluates left-to-right and yields only the last sub-expression, so the prior tokens are discarded and +j is applied only to the trailing element.

Step-by-step proof

With NUM_GPUS=8, PREFILLER_TP_SIZE=4, i=0 (one prefill instance):

j NEXT_GPU calculation NEXT_GPU GPU_ID
init i%8 0 0
1 (0 + 1) % 8 1 0,1
2 (0,1 + 2) % 8 → comma op → (0, 1+2) → 3 3 0,1,3
3 (0,1,3 + 3) % 8 → (0, 1, 3+3) → 6 6 0,1,3,6

Prefill is launched with CUDA_VISIBLE_DEVICES=0,1,3,6 (intended: 0,1,2,3). After the prefill loop, NEXT_GPU=6 leaks into the decode loop because it is never reset. With DECODER_TP_SIZE=4 and i=0:

j NEXT_GPU calculation NEXT_GPU GPU_ID
init (0 + 6 + 1) % 8 7
1 (7 + 1) % 8 0 7,0
2 (7,0 + 2) % 8 → (7, 0+2) → 2 2 7,0,2
3 (7,0,2 + 3) % 8 → (7, 0, 2+3) → 5 5 7,0,2,5

Decode is launched with CUDA_VISIBLE_DEVICES=7,0,2,5 (intended: 4,5,6,7). The disjoint allocations {0,1,2,3}/{4,5,6,7} implied by the comment "starting from after prefill GPUs" at line 198 are not what the loop actually produces.

Impact

Both prefill and decode processes are launched in parallel via eval "$FULL_CMD &" at lines 189 and 251. Both sets contain GPU 0, and both serve commands set --gpu-memory-utilization $GPU_MEMORY_UTILIZATION with GPU_MEMORY_UTILIZATION=0.85 from the new YAML — they will contend on / OOM GPU 0 (and waste GPU 4 entirely). The new optional CI step therefore cannot produce valid DSv4 accuracy numbers as configured, defeating its purpose.

Why this is the first trigger

Every other caller of run_accuracy_test.sh — config_sweep_accuracy_test.sh (tp_configs, dp_ep_configs, hybrid_ssm_configs) and all other YAML steps in this file — uses PREFILLER_TP_SIZE / DECODER_TP_SIZE ≤ 2. With TP=2 the j loop runs exactly once, GPU_ID is still a plain integer when the arithmetic happens, and the comma-operator path is never exercised. The TP=4/4 configuration introduced by this PR is the first to expose the latent bug, so flagging it here is appropriate even though the broken code lives in another file.

Fix

Two changes in run_accuracy_test.sh:

  1. Accumulate GPU IDs in a Bash array and stringify only at the end, e.g.: 
    GPUS=("$((i % $(get_num_gpus)))")
for (( j=1; j < PREFILLER_TP_SIZE; j++ )); do
  GPUS+=("$(((${GPUS[0]} + j) % $(get_num_gpus)))")
done
GPU_ID=$(IFS=,; echo "${GPUS[*]}")
NEXT_GPU=${GPUS[-1]}
  2. Reset / recompute NEXT_GPU at the start of the decode loop instead of leaking it from prefill.

13 changes: 8 additions & 5 deletions tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,12 +140,13 @@ run_tests_for_model() {
# Start prefill instances
for i in $(seq 0 $((NUM_PREFILL_INSTANCES-1))); do
# Calculate GPU ID - we'll distribute across available GPUs
GPU_ID=$((i % $(get_num_gpus)))
NEXT_GPU=${GPU_ID}
GPU_START=$((i % $(get_num_gpus)))
GPU_ID=$GPU_START
NEXT_GPU=$GPU_START
# Reserve TP*PP GPUs for the prefiller (TP shards across PP stages).
PREFILLER_WORLD_SIZE=$((PREFILLER_TP_SIZE * PREFILLER_PP_SIZE))
for (( j=1; j < PREFILLER_WORLD_SIZE; j++ )); do
NEXT_GPU=$(((GPU_ID + j) % $(get_num_gpus)))
NEXT_GPU=$(((GPU_START + j) % $(get_num_gpus)))
GPU_ID="${GPU_ID},${NEXT_GPU}"
done

Expand Down Expand Up @@ -195,10 +196,12 @@ run_tests_for_model() {
# Start decode instances
for i in $(seq 0 $((NUM_DECODE_INSTANCES-1))); do
# Calculate GPU ID - we'll distribute across available GPUs, starting from after prefill GPUs
GPU_ID=$(((i + NEXT_GPU + 1) % $(get_num_gpus)))
DECODE_START=$(((i + NEXT_GPU + 1) % $(get_num_gpus)))
GPU_ID=$DECODE_START
NEXT_GPU=$DECODE_START
# If DECODER_TP_SIZE is more than 1
for (( j=1; j < DECODER_TP_SIZE; j++ )); do
NEXT_GPU=$(((GPU_ID + j) % $(get_num_gpus)))
NEXT_GPU=$(((DECODE_START + j) % $(get_num_gpus)))
GPU_ID="${GPU_ID},${NEXT_GPU}"
done
# Calculate port number (base port + instance number)
Expand Down
1 change: 1 addition & 0 deletions tests/v1/kv_connector/nixl_integration/test_accuracy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"Qwen/Qwen3.5-0.8B": 0.33,
"google/gemma-4-E2B-it": 0.485,
"ai21labs/AI21-Jamba2-3B": 0.74,
"deepseek-ai/DeepSeek-V4-Flash": 0.95,
}

SIMPLE_PROMPT = (
Expand Down
Loading