[ZJY0516]

Overview

Part 1 of 2 split from #50897 (implements #50438): lookahead block hashes for the local prefix cache. EAGLE, EAGLE3 and MTP drafters write the draft KV at a block's last position from the token after the block, so main drops the last matched block on every hit. This PR puts that token into the block hash and removes the drop when no KV connector is configured. KV connector support is in the stacked PR (part 2).

Claims

• Local prefix-cache hits with EAGLE, EAGLE3 and MTP reach floor((N - 1) / B) * B, the same as without speculative decoding, instead of one cache unit less (544–1600 tokens on hybrid attention + Mamba/GDN models). Multi-module MTP with k modules reaches floor((N - k) / B) * B.
• A block is reused only when its lookahead token matches, so a hit reads the same draft KV as a cold run.
• New SpeculativeConfig.disable_lookahead_block_hashes restores main's behavior. With any KV connector configured, this PR keeps the legacy hashes.
• KV events publish the lookahead token in a new BlockStored.lookahead_token_ids field, so consumers can reproduce the hashes.
• Model runners are unchanged; no new CPU/GPU synchronization.

Validation

Unit tests on this branch (one GB200 node):

python -m pytest tests/v1/core tests/v1/streaming_input/test_scheduler_streaming.py \
  tests/v1/kv_connector/unit/test_nixl_connector_hma.py tests/models/kimi_k3/test_kda_metadata.py \
  tests/distributed/test_kv_cache_events.py \
  --ignore tests/v1/core/prefix_cache/test_hybrid_prefix_cache_hit_rate.py -k "not ray"

Result: 1,256 passed. test_hybrid_prefix_cache_hit_rate.py is excluded because it launches the vllm CLI (see the TODO below).

Hash format CPU cost: one request_block_hasher call over a whole prompt, SHA-256, one pinned GB200 Grace core, median of 15 alternating-order rounds of 5 calls. Token IDs are random, and the prompt length is aligned + 1 so every format emits the same hashes; MM rows add a 512-token image every 1,024 tokens. Each variant is the real hasher with only its lookahead branch replaced; an unmodified copy run the same way stays within 0.2% of the real one. Percentages are versus main.

• nested: H((parent, tokens[s:e], (E(s, e), tokens[e], E(e, e + 1)))), the previous revision of this PR
• expanded window: H((parent, tokens[s:e + 1], E(s, e + 1)))
• this PR: H((parent, tokens[s:e], (E(s, e + 1), tokens[e])))

Input	Prompt tokens	B	main	nested	expanded window	this PR
text	32K+1	16	3.166 ms	4.107 ms (+29.7%)	3.394 ms (+7.2%)	3.566 ms (+12.6%)
text	32K+1	128	0.917 ms	1.033 ms (+12.6%)	0.940 ms (+2.5%)	0.963 ms (+5.0%)
text	32K+1	1024	0.636 ms	0.647 ms (+1.7%)	0.635 ms (-0.2%)	0.639 ms (+0.5%)
text	128K+1	16	13.007 ms	16.770 ms (+28.9%)	13.945 ms (+7.2%)	14.406 ms (+10.8%)
text	128K+1	128	3.805 ms	4.248 ms (+11.6%)	3.907 ms (+2.7%)	3.967 ms (+4.3%)
text	128K+1	1024	2.705 ms	2.786 ms (+3.0%)	2.717 ms (+0.4%)	2.746 ms (+1.5%)
MM	32K+1	16	3.777 ms	5.126 ms (+35.7%)	4.104 ms (+8.7%)	4.271 ms (+13.1%)
MM	32K+1	128	1.013 ms	1.194 ms (+17.9%)	1.065 ms (+5.1%)	1.087 ms (+7.3%)
MM	32K+1	1024	0.654 ms	0.675 ms (+3.2%)	0.658 ms (+0.6%)	0.660 ms (+0.9%)

One E call instead of two saves 7–17% at B ≤ 128 versus the nested form. The expanded window is cheapest, but its hash input equals a legacy hash with block size B + 1; keeping tokens[e] in the extra keys costs 3–5% more than it at B = 16 and at most 2.5% at B ≥ 128. With one lookahead token the hasher indexes tokens[e] instead of slicing, which builds the same tuple; the slice cost about 7 more points at B = 16.

End-to-end and draft-KV results from the combined PR (#50897, earlier revision; not yet re-run on this split):

Qwen/Qwen3.6-27B, MTP k=4, 1x GB200, prefix_repetition	main	lookahead hashes
Prefix-cache hit rate	72.7%	85.0%
Request throughput	9.09 req/s	12.43 req/s
Mean / median TTFT	404 / 304 ms	318 / 226 ms
GSM8K 5-shot	0.4359	0.4359

Hit length, Qwen3.5-2B + MTP k=1, unit 544, 1,367-token prompt sent twice: main without spec decode 1088, main with MTP 544, lookahead hashes 1088.

Draft-KV check (KV of the hit blocks vs a cold run, bf16 max abs diff): Qwen3-8B + EAGLE3 0.09–0.14 (negative control with a stale lookahead token: 7.55); Qwen3.5-2B MTP 0.19 (control: 14.53).

TODO before review: re-run tests/v1/e2e/general/test_mamba_prefix_cache.py, tests/v1/core/prefix_cache/test_hybrid_prefix_cache_hit_rate.py and the benchmark on GPU.

Details

Enablement. get_num_hash_lookahead_tokens (v1/core/kv_cache_utils.py) returns L = num_prefill_lookahead_tokens (1, or k for multi-module MTP with k modules) when use_eagle_block_drop(), not disable_lookahead_block_hashes, prefix caching is on and there is no KV transfer config; otherwise 0, which keeps the legacy hashes and the block drop. DFlash and DSpark build draft KV from target hidden states alone; #54163 removes them from use_eagle_block_drop(), after which they keep the legacy hashes with no drop. Until then they get lookahead hashes too, which is correct but unnecessary. num_hash_lookahead_tokens replaces the former boolean flag through the KV cache manager, coordinator, block pool and scheduler. Part 2 replaces the last condition with a per-connector capability.

Hashing. get_request_block_hasher(..., num_hash_lookahead_tokens=L) hashes a block [s, e) only once tokens up to e + L - 1 are in all_token_ids, as H((parent, tokens[s:e], (E(s, e + L), *tokens[e:e + L]))); for L = 1 that is (E(s, e + 1), tokens[e]). E is generate_block_hash_extra_keys over the block and its lookahead tokens, called once per block, so MM inputs or prompt embeddings there are covered too; the MM cursor then steps back to the first input the next block overlaps. The hashed tokens stay the block's tokens, and the (E, token) shape keeps these hashes apart from legacy ones; hashing tokens[s:e + L] instead would equal a legacy hash with block size B + L. Different L give different tuple lengths. All KV cache groups share these hashes.

KV events. BlockStored.lookahead_token_ids holds tokens[e:e + L] per block, and extra_keys keeps its untagged per-block shape but covers [s, e + L). A block hash is reproducible from token_ids, extra_keys and lookahead_token_ids (test_kv_events_reconstruct_lookahead_block_hashes). Consumers that ignore the new field still decode events, but blocks with the same tokens at the same position can now carry different hashes, so routers that index one hash per position need to key on the lookahead tokens. The extra field widens the msgpack map header of every BlockStored (fixmap to map16); decoded contents are unchanged. examples/features/kv_events/kv_events_subscriber.py includes the field.

Caching. Blocks are cached at allocation, as without lookahead hashes; SingleTypeKVCacheManager.cache_blocks stops at the last hashed block. At request finish, update_from_output caches the block whose lookahead the last sampled token completes.

Drop removal. The scheduler clears use_eagle_block_drop. KVCacheCoordinator.drop_eagle_group_ids is empty in this mode and drives the per-group drop and replay_boundaries, so the last-block drop, the Mamba checkpoint back-off and the replay-boundary shift are all skipped from one place. eagle_group_ids stays, since mamba_shared_prefix_checkpoint still applies.

Docs. A new "Prefix caching" section in docs/features/speculative_decoding/README.md covers the default, disable_lookahead_block_hashes and BlockStored.lookahead_token_ids.

Why it is correct. The drafter writes the draft KV at e - 1 (input (h_{e-1}, token e)) in the step that prefills or samples token e. The hash exists only once token e is in all_token_ids, which never holds draft or in-flight tokens, so a cached block's draft KV is written by the end of that step. With k-module MTP, module j writes its KV at p from token p + j, so the hash waits for tokens up to e + k - 1. MultiModuleMTPSpeculator re-prefills a slot written from a rejected draft in the step that samples its replacement, and the coordinator already keeps the last k - 1 computed tokens out of the cache.

Limitations

• Multi-module MTP lookahead hashes are covered by unit tests only; the draft-KV check against a cold run has not been run for them yet.
• HiSparse requires HiSparseConnector, so like other connectors it keeps the legacy hashes and the block drop. Its host tier publishes pages once their host copies finish; whether those copies include the last position's draft KV needs GPU validation before part 2 lets HiSparseConnector opt in.
• Streaming-input sessions are not supported. When a turn ends on a lookahead boundary, the block cached with the discarded sampled token keeps that hash (its draft KV was written from that token), so later blocks of the session are unreachable by prefix lookups from other requests.
• Hybrid models lose the hit when a prompt diverges exactly at the producer's last retained Mamba boundary (0 vs one unit on main); see [Core][Spec Decode] Add lookahead-aware prefix cache hashing for EAGLE-style draft models #50897 for the reproducer.

Duplicate-work check

Split from #50897 by the same author. No other open PR adds the successor token to the prefix-cache key (#44986, #45614, #48375 and #39171 change when the drop applies, not the key).

AI assistance (Claude Code) was used to split, implement, test and describe this change. The submitting human reviews every changed line and is responsible for it end to end.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ZJY0516]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #97197 for commit d04e4db5fca9.

⚠️ This PR is 25 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

[andakai]

/ci run

[github-actions]

❌ @andakai, Only reviewers with write access can use CI commands before CI is delegated to the PR author.

[ivanium]

/ci run

[github-actions]

✅ Triggered Buildkite CI #97218 for commit 9cc08bfe366e.