Repository navigation
[Spec Decode] DSpark confidence-scheduled verification #47808
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
vllm-bot
merged 45 commits into
vllm-project:main
from
neuralmagic:codex/dspark-capacity-realloc
Aug 12, 2026
Merged
Changes from 36 commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
0d35edb
[Spec Decode] DSpark adaptive verification
LucasWilkinson a772e95
[Spec Decode] Keep non-adaptive request ordering identical to main
LucasWilkinson c4c8eec
[Spec Decode] Source decode max_q_len from attention metadata
LucasWilkinson ae28ecf
[Spec Decode] Prune implementation-focused unit tests
LucasWilkinson f8c5702
[Spec Decode] Rename adaptive verification config knobs
LucasWilkinson 9c70855
[Spec Decode] Fall back to fixed-length verification without cost pro…
LucasWilkinson cb068aa
[Spec Decode] Cap draft budget to one rejection-sampler chunk
LucasWilkinson 4cbe277
[Spec Decode] Do not treat async draft placeholders as grammar-rejected
LucasWilkinson 75b1522
[Spec Decode] Gate adaptive verification on LoRA, confidence head, mo…
LucasWilkinson 48daecc
cleanup
LucasWilkinson 5105a57
[Spec Decode] Drop cudagraph prefill-padding trim from SWA builder
LucasWilkinson 5b6ce97
[Spec Decode] Resolve profiling step timings without a per-step sync
LucasWilkinson 1ea4f64
cleanup
LucasWilkinson 4db5223
cleanup
LucasWilkinson aad37e1
fix precommit
LucasWilkinson ea368ee
[Spec Decode] Only require a topk buffer on compressed DSV4 layers
LucasWilkinson 45a8201
Revert "[Spec Decode] Only require a topk buffer on compressed DSV4 l…
LucasWilkinson e565583
[Spec Decode] Require a topk buffer only where it is read
LucasWilkinson e6e8bea
minor cleanup
LucasWilkinson da62b61
cleanup
LucasWilkinson af4751b
wip: review comments
LucasWilkinson 70596c8
[Spec Decode] Inherit target quantization for the DSV4 DSpark draft
LucasWilkinson eed2eef
[Spec Decode] Compute draft confidences in the topk sampling path too
LucasWilkinson 3c59d1e
[Spec Decode] Rename max_chunk_logits helper to avoid shadowing
LucasWilkinson 7ea349d
[Spec Decode] Align DSpark GSM8K eval config with the vLLM recipe
LucasWilkinson 387b084
review comments
LucasWilkinson e82c2b4
cleanup
LucasWilkinson 49f9571
review comments
LucasWilkinson e593351
Update vllm/config/speculative.py
LucasWilkinson f7c227c
Merge remote-tracking branch 'upstream/main' into dspark-final
LucasWilkinson 01db35a
review comments
LucasWilkinson 3becde2
review comments
LucasWilkinson d80ff5b
remove slop tests
LucasWilkinson 6636dfe
CI fixes
LucasWilkinson 10c132f
Merge branch 'main' into codex/dspark-capacity-realloc
LucasWilkinson 1160bce
Apply suggestion from @benchislett
benchislett 1854fdf
[Doc] Fix adaptive verification default in docs
LucasWilkinson a6ef5ac
review comments
LucasWilkinson 24fbcfb
Merge remote-tracking branch 'upstream/main' into pr47808
LucasWilkinson b67f231
review comments
LucasWilkinson a48eccc
Merge branch 'main' into codex/dspark-capacity-realloc
mergify[bot] aa72154
Merge branch 'main' into codex/dspark-capacity-realloc
njhill 7b032b5
fux test
LucasWilkinson 5cd5b8a
review refactors
LucasWilkinson e2e3353
[Bugfix] Skip adaptive sizing for no-draft batches
LucasWilkinson File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
47 changes: 47 additions & 0 deletions
47
docs/features/speculative_decoding/adaptive_verification.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Adaptive Verification | ||
|
|
||
| Speculative decoding buys fewer decode steps with more compute. At batch size 1 that is a good trade: the GPU is memory-bound with spare compute, so the extra draft tokens are close to free. At batch size 256 it is a much more delicate one. Draft tokens now compete with real tokens for the same compute, and every rejected token is compute wasted; with enough of them, throughput drops. | ||
|
|
||
| That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies. | ||
|
|
||
| Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two. | ||
|
|
||
| The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second. | ||
|
|
||
| The practical effect is that one configuration holds up across the whole load range, which removes most of the need to tune `num_speculative_tokens` per deployment. | ||
|
|
||
| ## Support | ||
|
|
||
| Adaptive verification needs per-position acceptance estimates, so today its currently only supported for DSpark with a **confidence head**. | ||
|
|
||
| ## Usage | ||
|
|
||
| It is enabled by default for DSpark: | ||
|
|
||
| ```bash | ||
| vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \ | ||
| --tokenizer-mode deepseek_v4 --trust-remote-code \ | ||
| --speculative-config '{ | ||
| "method": "dspark", | ||
| "model": "deepseek-ai/DeepSeek-V4-Flash-DSpark", | ||
| "num_speculative_tokens": 7, | ||
| "draft_sample_method": "probabilistic", | ||
| "enable_adaptive_verification": true | ||
| }' | ||
| ``` | ||
|
|
||
| Set `enable_adaptive_verification: false` to verify the full block for every request. | ||
|
|
||
| ## Requirements and limitations | ||
|
|
||
| - The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend. | ||
| - Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup. | ||
| - Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries), pipeline parallelism (cost curves and confidences exist only on the last rank), or output logprobs (to be fixed). | ||
|
|
||
| ## Tuning the cost profile | ||
|
|
||
| Step costs are profiled against a synthetic KV context, 8192 tokens by default. Deployments serving much longer contexts may want to raise it so the profiled step reads a more realistic amount of cache (this matters a bit less for sparse attention models like DeepSeek-v4 since the cheap indexer is the main cost that scales with context length). | ||
|
|
||
| ```bash | ||
| export VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=131072 | ||
| ``` |
25 changes: 25 additions & 0 deletions
25
tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| model_name: "deepseek-ai/DeepSeek-V4-Flash-DSpark" | ||
| accuracy_threshold: 0.92 | ||
| # lower acceptance length since trimming drafts at higher concurrency reduces | ||
| # acceptance length, this is very loose lower bound for smoke testing | ||
| min_acceptance_length: 1.2 | ||
| num_questions: 1319 | ||
| num_fewshot: 5 | ||
| startup_max_wait_seconds: 1800 | ||
| server_args: >- | ||
| --tokenizer-mode deepseek_v4 | ||
| --trust-remote-code | ||
| --max-model-len 8192 | ||
| --tensor-parallel-size 4 | ||
| --enable-expert-parallel | ||
| --block-size 256 | ||
| --gpu-memory-utilization 0.5 | ||
| --kv-cache-dtype fp8 | ||
| --attention_config.use_fp4_indexer_cache=True | ||
| --max-num-batched-tokens 16384 | ||
| --max-num-seqs 128 | ||
| --speculative-config '{"method":"dspark", | ||
| "model":"deepseek-ai/DeepSeek-V4-Flash-DSpark", | ||
| "attention_backend":"FLASH_ATTN","num_speculative_tokens":7, | ||
| "draft_sample_method":"probabilistic", | ||
| "enable_adaptive_verification":true}' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # SPDX-FileCopyrightText: Copyright contributors to the vLLM project | ||
|
|
||
| from types import SimpleNamespace | ||
|
|
||
| import numpy as np | ||
|
|
||
| from vllm.v1.worker.gpu.async_utils import StepTimingSample | ||
| from vllm.v1.worker.gpu.spec_decode.adaptive_verification import ( | ||
| AdaptiveVerificationManager, | ||
| ) | ||
|
|
||
|
|
||
| def make_manager( | ||
| confidences: np.ndarray, verify_cost_ms: np.ndarray | ||
| ) -> AdaptiveVerificationManager: | ||
| num_reqs, num_steps = confidences.shape | ||
| manager = AdaptiveVerificationManager.__new__(AdaptiveVerificationManager) | ||
| manager.num_speculative_steps = num_steps | ||
| manager._stale_confidences = [SimpleNamespace(np=confidences)] | ||
| manager._stale_idx = 0 | ||
| manager.req_states = SimpleNamespace( | ||
| req_id_to_index={"low": 0, "high": 1}, | ||
| num_computed_tokens_np=np.ones(num_reqs, dtype=np.int32), | ||
| prefill_len=SimpleNamespace(np=np.ones(num_reqs, dtype=np.int32)), | ||
| ) | ||
| manager.cost_tables = (np.zeros(num_reqs + 1), verify_cost_ms) | ||
| manager._max_total_logits = 1 << 30 | ||
| manager.num_bonus_tokens = 1 | ||
| return manager | ||
|
|
||
|
|
||
| def test_budget_stops_where_marginal_drafts_stop_paying_for_themselves(): | ||
| # Verification is cheap up to two extra tokens, then jumps 100x; only the | ||
| # highest-confidence draft is worth the cheap slot. | ||
| manager = make_manager( | ||
| np.array([[0.1, 0.1], [0.9, 0.9]], dtype=np.float32), | ||
| np.array([1.0, 1.0, 1.0, 1.0, 100.0, 100.0, 100.0]), | ||
| ) | ||
|
|
||
| manager.get_num_tokens( | ||
| {"low": 3, "high": 3}, | ||
| {"low": [1, 2], "high": [3, 4]}, | ||
| ) | ||
| valid_drafts, num_non_draft_tokens, draft_budget = manager._batch_budget | ||
|
|
||
| assert draft_budget == 1 | ||
| assert valid_drafts == {"low": 2, "high": 2} | ||
| assert num_non_draft_tokens == {"low": 1, "high": 1} | ||
|
|
||
|
|
||
| def test_profiled_batches_seed_cost_curves_via_consumer(): | ||
| manager = AdaptiveVerificationManager.__new__(AdaptiveVerificationManager) | ||
| manager.req_states = SimpleNamespace(max_num_batched_tokens=4096, max_num_reqs=64) | ||
| manager.num_speculative_steps = 7 | ||
| manager.num_bonus_tokens = 1 | ||
| curves: dict[str, list[tuple[int, float]]] = {} | ||
| manager.set_cost_curves = lambda draft, verify: curves.update( | ||
| draft=draft, verify=verify | ||
| ) | ||
|
|
||
| timings = [ | ||
| StepTimingSample( | ||
| forward_ms=float(batch["num_tokens"]), | ||
| drafter_ms=1.0, | ||
| num_target_tokens=batch["num_tokens"], | ||
| num_reqs=batch["num_tokens"] // 8, | ||
| # Only the captured sizes replay a graph; the tail sizes run eager. | ||
| full_cudagraph=batch["num_tokens"] <= 1024, | ||
| ) | ||
| for batch in manager.batches_to_profile([8, 1024]) | ||
| ] | ||
| manager.set_initial_cost_curves(timings) | ||
|
|
||
| # Tail beyond the last capture size: 1.5x then doubling to the max. | ||
| assert curves["verify"] == [ | ||
| (8, 8.0), | ||
| (1024, 1024.0), | ||
| (1536, 1536.0), | ||
| (2048, 2048.0), | ||
| (4096, 4096.0), | ||
| ] | ||
| # Eager batches must not contribute to the draft curve: keyed by request | ||
| # count they would land inside the captured range and, once made monotonic, | ||
| # smear that eager cost across every larger request count. | ||
| assert curves["draft"] == [(1, 1.0), (128, 1.0)] | ||
|
|
||
|
|
||
| def test_compact_batch_preserves_totals_and_bounds(): | ||
| # The CPU placeholder layout must keep the batch total equal to the GPU | ||
| # total and every verification row within decode_query_len, or downstream | ||
| # CPU metadata desyncs from the reallocated GPU boundaries. | ||
| manager = make_manager( | ||
| np.array([[0.9, 0.9], [0.9, 0.9], [1.0, 1.0]], dtype=np.float32), | ||
| np.array([1.0] * 44 + [100.0] * 3), | ||
| ) | ||
| manager.req_states.req_id_to_index["prefill"] = 2 | ||
| manager.req_states.num_computed_tokens_np = np.zeros(3, dtype=np.int32) | ||
| manager.req_states.prefill_len.np = np.array([0, 0, 60], dtype=np.int32) | ||
| num_tokens = manager.get_num_tokens( | ||
| {"low": 3, "high": 3, "prefill": 40}, | ||
| {"low": [1, 2], "high": [3, 4]}, | ||
| ) | ||
| scheduled = np.array([3, 3, 40], dtype=np.int32) | ||
| drafts = np.array([2, 2, 0], dtype=np.int32) | ||
| compacted = manager.compact_batch(drafts, scheduled) | ||
|
|
||
| assert int(compacted.sum()) == num_tokens | ||
| num_steps = manager.num_speculative_steps | ||
| assert (compacted[:2] <= 1 + num_steps).all() | ||
| assert compacted[2] == 40 | ||
|
|
||
|
|
||
| def test_budget_caps_at_one_rejection_sampler_chunk(): | ||
| # The chunked verification path cannot address the compacted logits | ||
| # layout, so the budget must keep total logits within a single chunk. | ||
| manager = make_manager( | ||
| np.array([[0.9, 0.9], [0.9, 0.9]], dtype=np.float32), | ||
| np.ones(7), | ||
| ) | ||
| manager._max_total_logits = 3 # 2 bonus logits + at most 1 draft | ||
| manager.get_num_tokens( | ||
| {"low": 3, "high": 3}, | ||
| {"low": [1, 2], "high": [3, 4]}, | ||
| ) | ||
| _, _, draft_budget = manager._batch_budget | ||
| assert draft_budget <= 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.