[ldkhang1201]

Overview

Fixes #64652. In Model Runner V2, the speculator builds its own attention groups. It picked each KV cache group's kernel block size from the drafter's backends alone, so it can disagree with the kernel pages the KV cache was allocated with. When that happens, draft steps 2+ read the wrong KV. This PR passes the runner's kernel_block_sizes to the speculator, which is what Model Runner V1 already does for its drafter (gpu_model_runner.py, drafter.initialize_attn_backend(kv_cache_config, kernel_block_sizes)).

Claims

• EAGLE draft acceptance after the first position no longer collapses when the drafter shares a KV cache group with target layers that need smaller kernel pages than --block-size. On the exact config from [Bug]: EAGLE3 draft acceptance collapses after the first position with --block-size > 128 (MiniMax-M3, GB200) #64652 (MiniMax-M3-NVFP4 + EAGLE3, --block-size 512), acceptance length goes from 2.11 to 3.09, matching --block-size 128.
• Configs where the two choices already agree, including the default backends, are unchanged.
• A new e2e regression test fails on main and passes with this change.

Validation

Regression test (test_qwen3_eagle3_split_kernel_pages_keep_draft_acceptance):

• Setup: Qwen3-8B with AngelSlim/Qwen3-8B_eagle3, target FLASHINFER, drafter TRITON_ATTN.
• Below SM100, FlashInfer only offers 16/32/64-token kernel pages while Triton accepts 128, so a 128-token block is split for the target group but the drafter alone would not split it.
• The test asserts that per-position acceptance at block_size=128 matches block_size=64 within 0.05.
• Run on 1x GH200 (SM90) on vLLM 3966924d5 (the nightly from [Bug]: EAGLE3 draft acceptance collapses after the first position with --block-size > 128 (MiniMax-M3, GB200) #64652), without and with this patch.

pytest -v -s tests/v1/e2e/spec_decode/eagle/test_eagle_correctness.py -k test_qwen3_eagle3_split_kernel_pages_keep_draft_acceptance

Without the fix:

E         comparison failed. Mismatched elements: 2 / 3:
E         Index | Obtained            | Expected
E         1     | 0.15544692737430169 | 0.2896844088553933 ± 0.05
E         2     | 0.04902234636871508 | 0.15182917255456116 ± 0.05
FAILED tests/v1/e2e/spec_decode/eagle/test_eagle_correctness.py::test_qwen3_eagle3_split_kernel_pages_keep_draft_acceptance
=========== 1 failed, 12 deselected, 2 warnings in 836.31s (0:13:56) ===========

With the fix:

=========== 1 passed, 12 deselected, 2 warnings in 770.37s (0:12:50) ===========

Most of that time is a cold first read of the weights. Each engine takes about 70 s to profile, compile and warm up, plus 5 s of generation.

Serving, same setup, 1x GH200.

• Load: 24 coding prompts, greedy, 512 tokens each, 4 concurrent.
• Metric: per-position acceptance from /metrics, the method used in [Bug]: EAGLE3 draft acceptance collapses after the first position with --block-size > 128 (MiniMax-M3, GB200) #64652.

Drafter backend	--block-size	main	this PR
TRITON_ATTN	64	0.561 / 0.294 / 0.140 (2.00)	0.554 / 0.287 / 0.133 (1.97)
TRITON_ATTN	128	0.578 / 0.121 / 0.022 (1.72)	0.552 / 0.285 / 0.133 (1.97)
TRITON_ATTN	256	0.571 / 0.113 / 0.023 (1.71)	0.557 / 0.287 / 0.141 (1.99)
default (same as target)	default	0.568 / 0.289 / 0.140 (2.00)	0.563 / 0.283 / 0.136 (1.98)

Each cell shows positions 1 / 2 / 3, with the acceptance length in parentheses.

The configuration from #64652, 1x GB300 (SM103).

• Software: vLLM 335dd77b0 nightly with and without this patch; nvidia/MiniMax-M3-NVFP4 at 9014640, drafter Inferact/MiniMax-M3-EAGLE3-GQA at 9669248.
• Flags: the reporter's command, adapted to one GPU (--tensor-parallel-size 1 --offload-backend uva --cpu-offload-gb 40 --disable-flashinfer-autotune --gpu-memory-utilization 0.95).
• Load: the same /metrics measurement as above.

--block-size	main	this PR
128	0.861 / 0.687 / 0.526 (3.07)	0.864 / 0.692 / 0.536 (3.09)
256	not run	0.871 / 0.709 / 0.548 (3.13)
512	0.879 / 0.202 / 0.030 (2.11)	0.865 / 0.692 / 0.529 (3.09)

The 128 and 512 rows on main match the report (0.78 / 0.60 / 0.45 and 0.82 / 0.15 / 0.02 on GB200 at TP2).

Unit tests. These ran on GH200 against this branch's Python tree and its parent 1fbde73aa1:

• tests/v1/spec_decode/test_draft_attention_backend_override.py
• tests/v1/worker/test_gpu_model_runner_v2.py
• tests/v1/worker/test_attn_utils.py
• tests/v1/worker/test_gpu_target_dependent_ar_speculator.py

Both trees: 150 passed. pre-commit run passes on the changed files.

Details

Root cause

• Since [Model Runner V2] fix draft attention metadata generation #37364, the MRV2 speculator calls init_attn_backend(..., active_layer_names=<draft layers>) to build metadata only for the groups the drafter uses. [Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim #42766 later moved prepare_kernel_block_sizes into init_attn_backend. So the speculator now votes a kernel block size per group from the drafter's backends only, while the runner votes over all layers and allocates the KV cache with its own result.
• [KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots #57169 made each speculator group map its own block tables and KV views from its kernel_block_size, so a disagreement between the two now matters.
• In [Bug]: EAGLE3 draft acceptance collapses after the first position with --block-size > 128 (MiniMax-M3, GB200) #64652, the EAGLE3 drafter's dense layer shares a KV cache group with MiniMax-M3's sparse attention and indexer layers, whose backends support only 128-token pages. The runner splits 256/512-token blocks into 128-token pages. The speculator's FlashInfer vote keeps 512, so its group has no kernel_block_size.
• Draft prefill reuses the target's metadata, which is correct, so position 1 is healthy. Draft decode steps 2+ build metadata through the speculator's groups and read the 128-token pages with an unsplit block table.
• KV writes use flat token slots and stay correct. That is why outputs stay lossless and only acceptance drops.

Fix. init_attn_backend takes an optional kernel_block_sizes and skips the vote when given. DraftModelSpeculator.set_attn (with the DFlash and standalone pass-throughs) takes the runner's self.kernel_block_sizes and forwards it. The runner's vote already includes the draft layers' backends, so every backend in a group supports the chosen size. AttentionGroup.create_metadata_builders still narrows it per backend, the same way it does for the runner's groups.

CI coverage

• The test is in tests/v1/e2e/spec_decode/eagle/test_eagle_correctness.py. The -k qwen3_eagle3 filter of the H200 step "Spec Decode Eagle 1: DeepSeek + Qwen" selects it, and that step depends on vllm/v1/worker/gpu/spec_decode/.
• It is skipped on SM100, where FlashInfer offers pages up to 1024 tokens for GQA so 128-token blocks are not split, and on non-CUDA platforms.
• It starts two engines with cached models, which adds roughly 3 to 4 minutes to that step.

Not a duplicate. No open PR references #64652. #62800 changes FlashInfer's per-layer page vote and padded-page splitting. It does not touch the speculator's separate vote.

AI assistance (Claude Code) was used to investigate, implement and validate this change. The runs above were done on the hardware listed.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.