Skip to content
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
0d35edb
[Spec Decode] DSpark adaptive verification
LucasWilkinson Jul 28, 2026
a772e95
[Spec Decode] Keep non-adaptive request ordering identical to main
LucasWilkinson Jul 30, 2026
c4c8eec
[Spec Decode] Source decode max_q_len from attention metadata
LucasWilkinson Jul 30, 2026
ae28ecf
[Spec Decode] Prune implementation-focused unit tests
LucasWilkinson Jul 30, 2026
f8c5702
[Spec Decode] Rename adaptive verification config knobs
LucasWilkinson Jul 30, 2026
9c70855
[Spec Decode] Fall back to fixed-length verification without cost pro…
LucasWilkinson Jul 30, 2026
cb068aa
[Spec Decode] Cap draft budget to one rejection-sampler chunk
LucasWilkinson Jul 30, 2026
4cbe277
[Spec Decode] Do not treat async draft placeholders as grammar-rejected
LucasWilkinson Jul 30, 2026
75b1522
[Spec Decode] Gate adaptive verification on LoRA, confidence head, mo…
LucasWilkinson Jul 30, 2026
48daecc
cleanup
LucasWilkinson Jul 30, 2026
5105a57
[Spec Decode] Drop cudagraph prefill-padding trim from SWA builder
LucasWilkinson Jul 30, 2026
5b6ce97
[Spec Decode] Resolve profiling step timings without a per-step sync
LucasWilkinson Jul 30, 2026
1ea4f64
cleanup
LucasWilkinson Jul 30, 2026
4db5223
cleanup
LucasWilkinson Jul 31, 2026
aad37e1
fix precommit
LucasWilkinson Jul 31, 2026
ea368ee
[Spec Decode] Only require a topk buffer on compressed DSV4 layers
LucasWilkinson Jul 31, 2026
45a8201
Revert "[Spec Decode] Only require a topk buffer on compressed DSV4 l…
LucasWilkinson Jul 31, 2026
e565583
[Spec Decode] Require a topk buffer only where it is read
LucasWilkinson Jul 31, 2026
e6e8bea
minor cleanup
LucasWilkinson Jul 31, 2026
da62b61
cleanup
LucasWilkinson Jul 31, 2026
af4751b
wip: review comments
LucasWilkinson Aug 4, 2026
70596c8
[Spec Decode] Inherit target quantization for the DSV4 DSpark draft
LucasWilkinson Aug 4, 2026
eed2eef
[Spec Decode] Compute draft confidences in the topk sampling path too
LucasWilkinson Aug 4, 2026
3c59d1e
[Spec Decode] Rename max_chunk_logits helper to avoid shadowing
LucasWilkinson Aug 5, 2026
7ea349d
[Spec Decode] Align DSpark GSM8K eval config with the vLLM recipe
LucasWilkinson Aug 5, 2026
387b084
review comments
LucasWilkinson Aug 6, 2026
e82c2b4
cleanup
LucasWilkinson Aug 6, 2026
49f9571
review comments
LucasWilkinson Aug 6, 2026
e593351
Update vllm/config/speculative.py
LucasWilkinson Aug 7, 2026
f7c227c
Merge remote-tracking branch 'upstream/main' into dspark-final
LucasWilkinson Aug 7, 2026
01db35a
review comments
LucasWilkinson Aug 7, 2026
3becde2
review comments
LucasWilkinson Aug 7, 2026
d80ff5b
remove slop tests
LucasWilkinson Aug 7, 2026
6636dfe
CI fixes
LucasWilkinson Aug 9, 2026
10c132f
Merge branch 'main' into codex/dspark-capacity-realloc
LucasWilkinson Aug 10, 2026
1160bce
Apply suggestion from @benchislett
benchislett Aug 10, 2026
1854fdf
[Doc] Fix adaptive verification default in docs
LucasWilkinson Aug 10, 2026
a6ef5ac
review comments
LucasWilkinson Aug 11, 2026
24fbcfb
Merge remote-tracking branch 'upstream/main' into pr47808
LucasWilkinson Aug 11, 2026
b67f231
review comments
LucasWilkinson Aug 11, 2026
a48eccc
Merge branch 'main' into codex/dspark-capacity-realloc
mergify[bot] Aug 11, 2026
aa72154
Merge branch 'main' into codex/dspark-capacity-realloc
njhill Aug 12, 2026
7b032b5
fux test
LucasWilkinson Aug 12, 2026
5cd5b8a
review refactors
LucasWilkinson Aug 12, 2026
e2e3353
[Bugfix] Skip adaptive sizing for no-draft batches
LucasWilkinson Aug 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .buildkite/test_areas/lm_eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,26 @@ steps:
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-pcp.txt

- label: LM Eval Spec Decode (4xB200)
key: lm-eval-spec-decode-4xb200
timeout_in_minutes: 120
device: b200-k8s
num_devices: 4
optional: true
source_file_dependencies:
- tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml
- tests/evals/gsm8k/configs/models-spec-decode.txt
- vllm/models/deepseek_v4/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/v1/spec_decode/metrics.py
- vllm/v1/worker/gpu/spec_decode/adaptive_verification.py
- vllm/v1/worker/gpu/spec_decode/dflash/
- vllm/v1/worker/gpu/spec_decode/dspark/
- vllm/v1/worker/gpu/spec_decode/rejection_sampler.py
- vllm/v1/worker/gpu/spec_decode/speculator.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-spec-decode.txt

- label: LM Eval Large Models EP (2xB200)
key: lm-eval-large-models-ep-2xb200
timeout_in_minutes: 60
Expand Down
2 changes: 2 additions & 0 deletions docs/features/speculative_decoding/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ vLLM supports a variety of methods of speculative decoding. Model-based methods
- [Hidden State Extraction](extract_hidden_states.md)
- [Custom Proposer Backend (Experimental)](#custom-proposer-backend-experimental)
- [Dynamic Speculative Decoding](dynamic_speculative_decoding.md)
- [Adaptive Verification](adaptive_verification.md)

## Method Selection at a Glance

Expand All @@ -35,6 +36,7 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
| Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
| Dynamic Speculative Decoding | High gain | Higher than base SD method | Useful for RL or workload with fluctuating QPS |
| Adaptive Verification | High gain | Higher than base SD method | Sizes verification per request from drafter confidence; currently DSpark only. |

For reproducible measurements in your environment, use
[`examples/features/speculative_decoding/spec_decode_offline.py`](../../../examples/features/speculative_decoding/spec_decode_offline.py)
Expand Down
47 changes: 47 additions & 0 deletions docs/features/speculative_decoding/adaptive_verification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Adaptive Verification

Speculative decoding buys fewer decode steps with more compute. At batch size 1 that is a good trade: the GPU is memory-bound with spare compute, so the extra draft tokens are close to free. At batch size 256 it is a much more delicate one. Draft tokens now compete with real tokens for the same compute, and every rejected token is compute wasted; with enough of them, throughput drops.

That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies.

Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.

The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second.

The practical effect is that one configuration holds up across the whole load range, which removes most of the need to tune `num_speculative_tokens` per deployment.

## Support

Adaptive verification needs per-position acceptance estimates, so today its currently only supported for DSpark with a **confidence head**.

## Usage

It is enabled by default for DSpark:

```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
--tokenizer-mode deepseek_v4 --trust-remote-code \
--speculative-config '{
"method": "dspark",
"model": "deepseek-ai/DeepSeek-V4-Flash-DSpark",
"num_speculative_tokens": 7,
"draft_sample_method": "probabilistic",
"enable_adaptive_verification": true
}'
```

Set `enable_adaptive_verification: false` to verify the full block for every request.

## Requirements and limitations

- The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend.
- Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup.
- Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries), pipeline parallelism (cost curves and confidences exist only on the last rank), or output logprobs (to be fixed).

## Tuning the cost profile

Step costs are profiled against a synthetic KV context, 8192 tokens by default. Deployments serving much longer contexts may want to raise it so the profiled step reads a more realistic amount of cache (this matters a bit less for sparse attention models like DeepSeek-v4 since the cheap indexer is the main cost that scales with context length).

```bash
export VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=131072
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
model_name: "deepseek-ai/DeepSeek-V4-Flash-DSpark"
accuracy_threshold: 0.92
# lower acceptance length since trimming drafts at higher concurrency reduces
# acceptance length, this is very loose lower bound for smoke testing
min_acceptance_length: 1.2
num_questions: 1319
num_fewshot: 5
startup_max_wait_seconds: 1800
server_args: >-
--tokenizer-mode deepseek_v4
--trust-remote-code
--max-model-len 8192
--tensor-parallel-size 4
--enable-expert-parallel
--block-size 256
--gpu-memory-utilization 0.5
--kv-cache-dtype fp8
--attention_config.use_fp4_indexer_cache=True
--max-num-batched-tokens 16384
--max-num-seqs 128
--speculative-config '{"method":"dspark",
"model":"deepseek-ai/DeepSeek-V4-Flash-DSpark",
"attention_backend":"FLASH_ATTN","num_speculative_tokens":7,
"draft_sample_method":"probabilistic",
"enable_adaptive_verification":true}'
Comment thread
benchislett marked this conversation as resolved.
Resolved
1 change: 1 addition & 0 deletions tests/evals/gsm8k/configs/models-spec-decode.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml
35 changes: 35 additions & 0 deletions tests/evals/gsm8k/test_gsm8k_correctness.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import shlex

import pytest
import requests
import yaml

from tests.utils import RemoteOpenAIServer
Expand Down Expand Up @@ -65,6 +66,28 @@ def run_gsm8k_eval(eval_config: dict, server_url: str) -> dict:
return results


def get_acceptance_length(server_url: str) -> float:
"""Mean tokens emitted per verification step, from the server's counters.

1.0 means every draft was rejected (speculation bought nothing); the
theoretical maximum is 1 + num_speculative_tokens.
"""
response = requests.get(f"{server_url.rstrip('/').removesuffix('/v1')}/metrics")
response.raise_for_status()
counters: dict[str, float] = {}
for line in response.text.splitlines():
if line.startswith("vllm:spec_decode_num_"):
name, _, value = line.partition(" ")
counters[name.split("{")[0]] = float(value)

num_drafts = counters.get("vllm:spec_decode_num_drafts_total", 0.0)
num_accepted = counters.get("vllm:spec_decode_num_accepted_tokens_total", 0.0)
assert num_drafts > 0, (
"no drafts recorded; speculative decoding did not run for this config"
)
return 1.0 + num_accepted / num_drafts


def test_gsm8k_correctness(config_filename):
"""Test GSM8K correctness for a given model configuration."""
eval_config = yaml.safe_load(config_filename.read_text(encoding="utf-8"))
Expand Down Expand Up @@ -168,4 +191,16 @@ def test_gsm8k_correctness(config_filename):
f"{expected_metric:.4f} - {tol:.4f} = {expected_metric - tol:.4f}"
)

# Speculative configs additionally assert that drafts are actually
# landing: accuracy alone passes even when every draft is rejected.
min_acceptance_length = eval_config.get("min_acceptance_length")
if min_acceptance_length is not None:
acceptance_length = get_acceptance_length(server_url)
print(f" Mean acceptance length: {acceptance_length:.3f}")
print(f" Minimum acceptance length: {min_acceptance_length:.3f}")
assert acceptance_length >= min_acceptance_length, (
f"Acceptance length too low: {acceptance_length:.3f} < "
f"{min_acceptance_length:.3f}"
)

print(f"✅ GSM8K test passed for {eval_config['model_name']}")
127 changes: 127 additions & 0 deletions tests/v1/spec_decode/test_adaptive_verification.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from types import SimpleNamespace

import numpy as np

from vllm.v1.worker.gpu.async_utils import StepTimingSample
from vllm.v1.worker.gpu.spec_decode.adaptive_verification import (
AdaptiveVerificationManager,
)


def make_manager(
confidences: np.ndarray, verify_cost_ms: np.ndarray
) -> AdaptiveVerificationManager:
num_reqs, num_steps = confidences.shape
manager = AdaptiveVerificationManager.__new__(AdaptiveVerificationManager)
manager.num_speculative_steps = num_steps
manager._stale_confidences = [SimpleNamespace(np=confidences)]
manager._stale_idx = 0
manager.req_states = SimpleNamespace(
req_id_to_index={"low": 0, "high": 1},
num_computed_tokens_np=np.ones(num_reqs, dtype=np.int32),
prefill_len=SimpleNamespace(np=np.ones(num_reqs, dtype=np.int32)),
)
manager.cost_tables = (np.zeros(num_reqs + 1), verify_cost_ms)
manager._max_total_logits = 1 << 30
manager.num_bonus_tokens = 1
return manager


def test_budget_stops_where_marginal_drafts_stop_paying_for_themselves():
# Verification is cheap up to two extra tokens, then jumps 100x; only the
# highest-confidence draft is worth the cheap slot.
manager = make_manager(
np.array([[0.1, 0.1], [0.9, 0.9]], dtype=np.float32),
np.array([1.0, 1.0, 1.0, 1.0, 100.0, 100.0, 100.0]),
)

manager.get_num_tokens(
{"low": 3, "high": 3},
{"low": [1, 2], "high": [3, 4]},
)
valid_drafts, num_non_draft_tokens, draft_budget = manager._batch_budget

assert draft_budget == 1
assert valid_drafts == {"low": 2, "high": 2}
assert num_non_draft_tokens == {"low": 1, "high": 1}


def test_profiled_batches_seed_cost_curves_via_consumer():
manager = AdaptiveVerificationManager.__new__(AdaptiveVerificationManager)
manager.req_states = SimpleNamespace(max_num_batched_tokens=4096, max_num_reqs=64)
manager.num_speculative_steps = 7
manager.num_bonus_tokens = 1
curves: dict[str, list[tuple[int, float]]] = {}
manager.set_cost_curves = lambda draft, verify: curves.update(
draft=draft, verify=verify
)

timings = [
StepTimingSample(
forward_ms=float(batch["num_tokens"]),
drafter_ms=1.0,
num_target_tokens=batch["num_tokens"],
num_reqs=batch["num_tokens"] // 8,
# Only the captured sizes replay a graph; the tail sizes run eager.
full_cudagraph=batch["num_tokens"] <= 1024,
)
for batch in manager.batches_to_profile([8, 1024])
]
manager.set_initial_cost_curves(timings)

# Tail beyond the last capture size: 1.5x then doubling to the max.
assert curves["verify"] == [
(8, 8.0),
(1024, 1024.0),
(1536, 1536.0),
(2048, 2048.0),
(4096, 4096.0),
]
# Eager batches must not contribute to the draft curve: keyed by request
# count they would land inside the captured range and, once made monotonic,
# smear that eager cost across every larger request count.
assert curves["draft"] == [(1, 1.0), (128, 1.0)]


def test_compact_batch_preserves_totals_and_bounds():
# The CPU placeholder layout must keep the batch total equal to the GPU
# total and every verification row within decode_query_len, or downstream
# CPU metadata desyncs from the reallocated GPU boundaries.
manager = make_manager(
np.array([[0.9, 0.9], [0.9, 0.9], [1.0, 1.0]], dtype=np.float32),
np.array([1.0] * 44 + [100.0] * 3),
)
manager.req_states.req_id_to_index["prefill"] = 2
manager.req_states.num_computed_tokens_np = np.zeros(3, dtype=np.int32)
manager.req_states.prefill_len.np = np.array([0, 0, 60], dtype=np.int32)
num_tokens = manager.get_num_tokens(
{"low": 3, "high": 3, "prefill": 40},
{"low": [1, 2], "high": [3, 4]},
)
scheduled = np.array([3, 3, 40], dtype=np.int32)
drafts = np.array([2, 2, 0], dtype=np.int32)
compacted = manager.compact_batch(drafts, scheduled)

assert int(compacted.sum()) == num_tokens
num_steps = manager.num_speculative_steps
assert (compacted[:2] <= 1 + num_steps).all()
assert compacted[2] == 40


def test_budget_caps_at_one_rejection_sampler_chunk():
# The chunked verification path cannot address the compacted logits
# layout, so the budget must keep total logits within a single chunk.
manager = make_manager(
np.array([[0.9, 0.9], [0.9, 0.9]], dtype=np.float32),
np.ones(7),
)
manager._max_total_logits = 3 # 2 bonus logits + at most 1 draft
manager.get_num_tokens(
{"low": 3, "high": 3},
{"low": [1, 2], "high": [3, 4]},
)
_, _, draft_budget = manager._batch_budget
assert draft_budget <= 1
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ def mock_model_runner_with_req_states():
runner.prompt_logprobs_worker = None
runner.is_last_pp_rank = False
runner.pooling_runner = None
runner.adaptive_verification = None

# Mock staged writes — they use Triton kernels that require GPU
runner.req_states.apply_staged_writes = Mock()
Expand Down
6 changes: 3 additions & 3 deletions tests/v1/worker/test_gpu_batch_ordering.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,22 +40,22 @@ def _make_common_attn_metadata(query_lens: list[int]) -> CommonAttentionMetadata
def test_sort_batch_req_ids_no_spec():
# decode_query_len == 1: plain ascending order (decodes first).
num_tokens_per_req = {"p1": 100, "d1": 1, "p2": 7, "d2": 1}
assert sort_batch_req_ids(num_tokens_per_req, 1) == ["d1", "d2", "p2", "p1"]
assert sort_batch_req_ids(num_tokens_per_req, {}, 1) == ["d1", "d2", "p2", "p1"]


def test_sort_batch_req_ids_spec_decode():
# decode_query_len == 2 (MTP k=1): uniform decodes lead, then the 1-token
# chunked-prefill tail, then longer prefills.
num_tokens_per_req = {"tail": 1, "d1": 2, "p1": 100, "d2": 2}
assert sort_batch_req_ids(num_tokens_per_req, 2) == ["d1", "d2", "tail", "p1"]
assert sort_batch_req_ids(num_tokens_per_req, {}, 2) == ["d1", "d2", "tail", "p1"]


def test_spec_decodes_lead_short_prefill_tail():
# With the fixed ordering, split_decodes_and_prefills classifies the
# uniform 2-token decodes as decodes even when a 1-token prefill tail is
# in the batch (indexer-style: require_uniform, threshold=1+k).
num_tokens_per_req = {"tail": 1, **{f"d{i}": 2 for i in range(8)}}
req_ids = sort_batch_req_ids(num_tokens_per_req, 2)
req_ids = sort_batch_req_ids(num_tokens_per_req, {}, 2)
query_lens = [num_tokens_per_req[r] for r in req_ids]
assert query_lens == [2] * 8 + [1]

Expand Down
11 changes: 11 additions & 0 deletions vllm/config/speculative.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,10 @@ class SpeculativeConfig:
synthetic_acceptance_rates. Only valid when rejection_sample_method is 'synthetic'.
Mutually exclusive with synthetic_acceptance_rates."""

enable_adaptive_verification: bool = False
"""Whether to adaptively size the draft-verification budget from per-request
confidence. Currently only supported for method="dspark"."""

@staticmethod
def _acceptance_length_to_rates(length: float, n: int) -> list[float]:
"""Mean acceptance length to unconditional per-position rates, using
Expand Down Expand Up @@ -998,6 +1002,9 @@ def __post_init__(self):
self.draft_model_config.hf_config.architectures = [
"DSparkDraftModel"
]
self.draft_model_config.quantization = (
self.target_model_config.quantization
)
self.update_arch_()
elif (
self.method == "dspark"
Expand Down Expand Up @@ -1143,6 +1150,10 @@ def __post_init__(self):
self.target_parallel_config, self.draft_tensor_parallel_size
)
)

if self.method != "dspark" and self.enable_adaptive_verification:
raise ValueError("Adaptive verification only supported with DSpark")

return self

def _validate_suffix_decoding(self):
Expand Down
Loading
Loading