[Bugfix][PD] Pad resumed speculative decode requests

[ZeldaHuang]

Purpose

Keep resumed speculative-decode requests on the uniform verifier shape when a data-parallel rank has no already-running requests.

In P/D serving, a synchronous KV connector can resolve a waiting consumer to all but the final prompt token. During that scheduling pass, the matched prefix exists in the local num_computed_tokens variable, while request.num_computed_tokens is updated only after admission. The current padding condition also requires local scheduled_running_reqs, so this rank schedules one token even when peer ranks run the 1 + num_spec_tokens verifier shape. A single qlen=1 rank then prevents the DP group from using the full decode CUDA graph.

This change treats a request with resolved computed tokens as resumed and applies the existing reject-only -1 padding, provided no prefill was scheduled. Ordinary prefills and diffusion speculative decoding retain their current behavior.

The regression test uses a synchronous mock KV connector with 32 matched tokens and does not manually mutate request state.

Why this is not duplicate work

I searched open PRs for speculative-decode padding, resumed KV-connector requests, P/D CUDA graphs, and the relevant scheduler condition. No open PR changes admission for a synchronously resumed request on a rank without running work.

• [SpecDecode] Preserve reject-only residual verifier padding #54102 addresses Mamba alignment, rejection sentinels, adaptive verification, and verifier graph topology; it retains the scheduled_running_reqs requirement.
• [Bugfix][Spec Decode] Don't pad a resumed decode request past max_model_len #53812 prevents resumed padding from exceeding max_model_len.
• [Bugfix][Core] Skip spec padding below prefill threshold #54028 prevents padding from conflicting with the long-prefill threshold.
• [Bugfix][Core][Spec Decode] Exclude scheduler padding from draft metrics #50518 separates reject-only padding from acceptance metrics.

Tests

Full scheduler test file, using an offline model redirect and a local pytest shim selecting CpuPlatform on the CPU-only test host:

uv run --python .venv/bin/python --no-project python -m pytest -q \
  -p cpu_platform_plugin tests/v1/core/test_scheduler.py

158 passed, 175 warnings in 43.74s

Changed-file repository hooks:

.venv/bin/pre-commit run --files \
  vllm/v1/core/sched/scheduler.py \
  tests/v1/core/test_scheduler.py

All applicable hooks passed, including Ruff check/format, mypy, SPDX, forbidden imports, and configuration validation. git diff --check origin/main...HEAD also passed.

Serving validation

The equivalent scheduler-only change was validated on four 4xGB300 nodes (DEP16/EP16). Set the model paths and host names for the target cluster before running the commands below.

Run the same server command on all four nodes, with a shared MASTER_ADDR. Each process owns four local DP ranks:

export MODEL=<path-to-Kimi-K3-model>
export DRAFT_MODEL=<path-to-K3-EAGLE3-draft-model>
export MASTER_ADDR=<first-node-hostname>
export VLLM_ATTENTION_BACKEND=FLASHINFER_MLA
export VLLM_USE_FLASHINFER_MOE_FP4=1
export VLLM_FLASHINFER_MOE_BACKEND=latency
export VLLM_USE_V2_MODEL_RUNNER=1

vllm serve "$MODEL" \
  --port 8000 \
  --distributed-executor-backend mp \
  --served-model-name moonshotai/Kimi-K3 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --safetensors-load-strategy lazy \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --quantization kimi_k3_dual \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"FLASHINFER"}' \
  --tensor-parallel-size 1 \
  --data-parallel-size 16 \
  --data-parallel-size-local 4 \
  --data-parallel-address "$MASTER_ADDR" \
  --data-parallel-rpc-port 13345 \
  --data-parallel-hybrid-lb \
  --enable-expert-parallel \
  --enable-ep-weight-filter \
  --all2all-backend flashinfer_nvlink_one_sided \
  --language-model-only \
  --mm-processor-cache-gb 0 \
  --moe-backend deep_gemm_mega_moe \
  --no-enable-prefix-caching \
  --max-num-seqs 12 \
  --max-model-len auto \
  --max-num-batched-tokens 48 \
  --reasoning-config '{"reasoning_parser":"kimi_k3","strict_thinking":false}' \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,3,4,5,6,7,8,9,10,11,12,16,20,24,28,32,36,40,44,48]}' \
  --speculative-config "{\"method\":\"eagle3\",\"model\":\"$DRAFT_MODEL\",\"num_speculative_tokens\":3,\"disable_eagle_block_drop\":true,\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_rates\":[0.70,0.49,0.3382]}" \
  --kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"fill_mean":0.015,"fill_std":0.0}}'

Route traffic across the four node-local API servers:

vllm-router \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 18080 \
  --prometheus-port 18081 \
  --intra-node-data-parallel-size 4 \
  --request-timeout-secs 900 \
  --worker-urls \
    http://<node-0>:8000 \
    http://<node-1>:8000 \
    http://<node-2>:8000 \
    http://<node-3>:8000

The unconditional per-position acceptance rates sum to 1.5282, giving a target mean acceptance rate of 50.94% and acceptance length of 2.5282 for K=3. Run the QPS sweep once on each revision:

for QPS in 2 3 4 5 6; do
  vllm bench serve \
    --backend openai \
    --base-url http://localhost:18080 \
    --model moonshotai/Kimi-K3 \
    --tokenizer "$MODEL" \
    --trust-remote-code \
    --dataset-name random \
    --random-input-len 120000 \
    --random-range-ratio 0.5 \
    --random-output-len 400 \
    --prompt-token-ids \
    --ignore-eos \
    --request-rate "$QPS" \
    --max-concurrency 192 \
    --num-prompts 480 \
    --num-warmups 24 \
    --seed "$((300 + QPS))" \
    --percentile-metrics ttft,tpot,itl,e2el \
    --metric-percentiles 50,90,95,99,99.5,99.9 \
    --result-dir "results/qps$QPS" \
    --save-detailed \
    --save-result \
    --result-filename result.json
done

Apply the base scheduler for the before sweep and this PR's scheduler for the after sweep. Every point completed 480 / 480 requests with zero failures. Observed acceptance rates were 49.74%–52.80%, with acceptance lengths of 2.483–2.574.

QPS	P50 ITL, before → after	P50 change	P99 ITL, before → after	P99 change
2	32.17 → 29.97 ms	-6.9%	250.68 → 41.52 ms	-83.4%
3	33.03 → 32.33 ms	-2.1%	240.85 → 49.97 ms	-79.3%
4	35.50 → 34.33 ms	-3.3%	232.84 → 56.42 ms	-75.8%
5	36.51 → 36.27 ms	-0.7%	68.40 → 57.87 ms	-15.4%
6	38.07 → 38.11 ms	+0.1%	60.31 → 65.34 ms	+8.3%

The P99 benefit is large through QPS 4, narrows at QPS 5, and is no longer visible at QPS 6. This matches the scheduler mechanism: once each rank stays busy, the original scheduled_running_reqs branch already applies the padding. P50 remains nearly unchanged throughout the sweep.

No accuracy evaluation was needed because this reuses existing reject-only placeholder semantics and does not change model outputs, draft acceptance, kernels, or graph dispatch.

AI assistance

OpenAI Codex assisted with root-cause analysis, implementation, tests, runtime experiment analysis, and PR preparation. I reviewed and understand every changed line and am responsible for the contribution.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[njhill]

Thanks @ZeldaHuang

[njhill]

/ci run

[github-actions]

✅ Triggered Buildkite CI #87121 for commit f4b27e0b2048.