[Spec Decode] Cache adaptive verification profiles

[calvarado2004]

Purpose

DSpark adaptive verification profiles target-forward and drafter costs at every captured and eager-tail shape during each engine boot. The calibration is deterministic for a fixed model, runtime, backend, hardware, and serving configuration, but it currently runs five replays per shape every time.

This PR adds an opt-in persistent cache for the exact calibrated draft and verification curves. Enable it with:

export VLLM_ENABLE_ADAPTIVE_VERIFICATION_PROFILE_CACHE=1

Cache entries live under $VLLM_CACHE_ROOT/adaptive_verification/.

Why this is not a duplicate

I searched open and closed PRs for adaptive verification cache, adaptive verification profile, DSpark profile cache, and the proposed environment variable. No existing PR persists these curves.

• [Spec Decode] DSpark confidence-scheduled verification #47808 introduced adaptive verification and its startup calibration; it intentionally recalibrates on each boot.
• [MRV2] Worker-Level Timings #50919 adds worker-level runtime timing observability, not persistent startup-profile reuse.
• [Spec Decode] Add D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding #47131 and [MRV2][Spec Decode] Adaptive Speculative Decoding - Initial Support #48692 implement related adaptive verification policies, not caching.

Design

Cache reuse is fail-closed:

• Rank 0 atomically saves the exact median curves already broadcast by set_cost_curves.
• The cache key includes an immutable model identity (resolved remote revision or local config/index digest), GPU identity and compute capability, TP/PP/DP/DCP, vLLM/PyTorch/FlashInfer/CUDA/HIP/cuDNN versions, resolved attention/KV and quantization kernel implementations, K, KV dtype, scheduler limits, CUDA graph mode/capture grid, and profile context.
• Entries carry a schema version, their full fingerprint inputs, strictly validated positive curves, and a SHA-256 payload checksum.
• All TP ranks must independently produce the same fingerprint.
• A hit runs two full-CUDA-graph sentinel replays. Both target and drafter timings must resolve to exact cached curve points and remain within a deliberately generous 0.4x-2.5x range on every TP rank.
• Any missing revision/index, I/O error, corruption, version mismatch, heterogeneous TP fingerprint, or sentinel anomaly falls back to the existing full profile and refreshes the cache.

The feature is disabled by default. Cold-start behavior and the cost-table construction are unchanged; a valid hit passes the same curves through the existing set_cost_curves path. This does not change token sampling or output correctness.

Startup impact

For the DeepSeek-V4 DSpark configuration that motivated this change (12 captured shapes, 8 eager-tail shapes, 5 replays each), a cold calibration performs 100 dummy runs while a warm hit performs 2 sentinel runs: 98% fewer profiling runs.

In a saved TP2 DGX Spark startup log, graph capture plus adaptive calibration took 353 seconds, with approximately 340 seconds after DSpark graph capture attributable to the calibration phase. The projected warm graph/profile phase is about 13-15 seconds. These are projected warm-cache savings from the observed cold log; this branch has not yet been benchmarked end-to-end on that GPU, so the PR remains draft.

Test plan

Passed:

/Users/carlos/exo/.venv/bin/python -m pytest -q --confcutdir=tests/v1/spec_decode tests/v1/spec_decode/test_adaptive_verification_cache.py
# 4 passed

/Users/carlos/exo/.venv/bin/ruff check <all changed Python files>
# All checks passed

/Users/carlos/exo/.venv/bin/python -m compileall -q <all changed Python files>
git diff --check
# passed

The focused tests cover exact round trips/fingerprint misses, payload corruption, schema-version rejection, and sentinel point/mode/timing rejection.

I also attempted the pre-existing tests/v1/spec_decode/test_adaptive_verification.py suite. It could not collect in this macOS development environment because the partial environment lacks the full vLLM test dependency set (einops was the final reported missing dependency after installing tblib, pyzmq, cbor2, and openai into a temporary target). No output/model evaluation was run because this is an opt-in startup-only change that reuses exact calibrated inputs and fails closed; a warm/cold GPU startup comparison is still required before marking the PR ready.

AI assistance

AI assistance was used to port the implementation, write tests, and prepare this description. I reviewed every changed line and confirm that I understand the design and can defend the implementation. A warm/cold GPU validation is still required before moving this PR out of draft.

[mergify]

Documentation preview: https://vllm--52233.org.readthedocs.build/en/52233/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @calvarado2004.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[calvarado2004]

The merge conflicts with main are now resolved, and the branch is up to date with the latest main. Ruff checks pass locally and the diff is limited to the PR's original scope (6 files). This PR is ready for review — could a maintainer please take a look when they have a chance? Thank you!

[calvarado2004]

Requesting a ready label from a maintainer so CI can actually run here.

pre-run-check is red, but not on the code — it is the contributor gate:

PR must have the 'verified', 'ready', or 'ready-run-all-tests' label to run pre-commit, or the author must have at least 4 merged PRs (found 1).

pre-commit is therefore skipped, and the PR has had no CI signal since it was opened on Aug 14. I have one merged PR (#46177), so the count path will not clear on its own.

Rechecked against main at a0f1b9ad05 today:

git merge-tree main <this branch>                                    no conflict
pytest tests/v1/spec_decode/test_adaptive_verification_cache.py      4 passed

(The teardown RuntimeError: Allocator for mps is not a DeviceAllocator is a macOS artefact from tests/conftest.py::cleanup_fixture, not an assertion failure.)

The change is opt-in and off by default: the cache is gated behind a new env var in vllm/envs.py, and adaptive_verification.py falls back to the existing profiling path whenever the cache misses, is corrupt, or carries a different schema version — the three cases the tests above cover. No default behaviour changes.

Happy to rebase, split it, or adjust the scope if a reviewer would prefer a different shape. AI assistance was used on this PR; I reviewed every changed line and ran the commands above.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @calvarado2004.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork