[Pb314314]

Overview

Model runner V2 captures CUDA graphs from one token list (cudagraph_capture_sizes, capped by max_cudagraph_capture_size, 1024 tokens on GB200), shared by prefill and decode. With speculative decoding a decode request holds 1 + K tokens, so decode graphs stop at 1024 / (1 + K) requests (128 at K = 7), and every prefill or mixed step above 1024 tokens runs eagerly.

This PR gives each kind its own setting in its own unit: prefill graphs in tokens, decode graphs in requests, plus a token budget per decode graph. On SM100 (GB200, B200) the defaults cover prefill and mixed steps up to 8192 tokens and decode batches up to max_num_seqs, within 4096 tokens per decode graph (1024 requests for K ≤ 3, 512 at K = 7). The default grids are coarser than main's list, so at the GB200 defaults every runner captures fewer graphs than main. On other GPUs the prefill and target decode defaults stay as on main.

This is the "separate prefill/decode graph sizes" item of the Q4 roadmap (vllm-project#61024).

Claims

All on GB200 with the vllm serve defaults (max_num_seqs 1024, max_num_batched_tokens 16384), against main 60a7d568f4.

• Throughput at matched SD settings (this head, untraced; not a comparison of each build's best configuration). DeepSeek-V4-Flash with DSpark K=7: ×1.03 / ×3.7 / ×2.3 output tok/s at 16 / 256 / 512 concurrent requests. MiniMax-M3 with EAGLE3 K=3: ×1.02 / ×2.0 / ×2.7. Additional configurations: ×0.99–1.05 at 1 request, ×1.0–1.08 at 16, ×1.2–1.8 at 256 (dense Qwen3.8-27B on one GPU excepted, appendix F). All on ShareGPT prompts (table under Results).
  The gain comes from steps that are eager on main: mixed steps of 1K–8K tokens (PIECEWISE here) and, at 256–512 requests, decode batches above 1024 / (1 + K) requests (FULL here).
• Graph counts at the GB200 defaults. 39 PIECEWISE sizes up to 8192 tokens (main: 83 up to 1024). Target decode: 64 sizes up to 1024 requests for K ≤ 3, and 48 up to 512 at K = 7 (main: 80–93, reaching 128–1024 requests). Every drafter runner also captures fewer graphs than main.
  Not for every setting: with max_num_seqs 256 and no SD, target decode is 37 graphs (main 35); off SM100, one-token draft decodes are 64 (main 51) and dynamic SD target decode 176 (main 147).
• KV cache. The graphs are fewer but wider. M3 keeps 0.94× of main's KV cache. DeepSeek-V4-Flash DSpark K=7 keeps 0.81×; the other models 0.93–0.99×, except adaptive-verification configs (0.66–0.78×), with both physical-memory growth and budgeting issues; measured breakdown and fixes. The section Closing the DeepSeek-V4 KV cache gap reports an earlier combined follow-up prototype reaching 0.989×; those fixes are not included here. Main runs out of memory at 1024 concurrent requests on that model; this PR does not.
• Correctness: no regression observed in the completed evaluations of this head. GSM8K accuracy, answer agreement and acceptance length were checked against main on DeepSeek-V4-Flash (DSpark K=7, K=2, dynamic SD, adaptive verification), DeepSeek-V4.1-Flash (DSpark K=5 with adaptive verification), MiniMax-M3 (EAGLE3 K=3), Qwen3.8-Flash-Next (MTP K=3), Qwen3.8-27B (MTP K=3) and Kimi-Linear-48B-A3B.

Configuration

Five fields on CompilationConfig (--compilation-config '{...}'):

field	unit	controls	default on SM100	default on other GPUs
max_prefill_cudagraph_capture_tokens	tokens	Cap on PIECEWISE graphs: every batch that is not a uniform decode, and uniform decodes no FULL graph covers	min(8192, max_num_batched_tokens)	the platform's max_cudagraph_capture_size, as on main (512 on H100/H200)
prefill_cudagraph_capture_token_sizes	tokens	The exact token counts that get a PIECEWISE graph	the prefill grid up to the cap	main's list
max_decode_cudagraph_capture_requests	requests	Cap on FULL graphs for uniform decode batches, where every request has the same query length q (1 + K for the target)	max_num_seqs	main's reach, 512 / (1 + K)
decode_cudagraph_capture_request_sizes	requests	The exact request counts that get a decode graph. A runner captures n requests as n · q tokens	the decode grid up to the cap	main's request counts
max_decode_cudagraph_capture_tokens	tokens	Capture uniform decode only when n · q ≤ budget, including q=1. Larger batches use PIECEWISE where supported, otherwise eager; AV is exempt. Memory safeguard and follow-ups	4096 (interim default)	none

Default grids, and how they differ from main. On main, one token list (1, 2, 4, then every 8 up to 256, then every 16 up to 1024: 83 sizes) serves both kinds. A decode graph is a list entry divisible by 1 + K, so decode reaches 1024 / (1 + K) requests: 1024 / 512 / 336 / 256 / 168 / 128 at K = 0 / 1 / 2 / 3 / 5 / 7.

Grid	Rule	Graph count	Largest captured batch
Prefill	1, 2, 4, …, 256; then every 256 tokens through 7680; finally 8192. General rule: Appendix A	39 (main: 83)	8192 tokens (main: 1024 tokens)
Uniform decode, K ≤ 3	Every request count through 6, then increasing power-of-two strides; see Appendix A	64	1024 requests
Uniform decode, K=7	Same request grid, limited by the 4096-token budget	48	512 requests × q=8 = 4096 tokens
Uniform decode, K=15	Same request grid, limited by the 4096-token budget	37	256 requests × q=16 = 4096 tokens
Adaptive verification, K=5	Separate dense token grid; not limited by the uniform-decode budget	64 (main: 85)	6144 tokens (main: 1024 tokens)

Graph count here counts capture sizes per routine; one PIECEWISE size can contain multiple CUDA graph objects. GB200 defaults: 1024 sequences, 16384 batched tokens. AV uses 1, 2, 4, then four sizes per doubling through 1024, every 128 through 4096, and every 256 above, capped at min(decode cap × (1 + K), max_num_batched_tokens).

How to configure.

• Set a cap to use the default grid, or a list to choose sizes. If both are set, the list must fit within the cap.
• Set a list to [] to disable that kind; set both lists to [] to disable CUDA graphs.
• Prefill and decode have separate settings. DSpark/DFlash-family draft coverage also depends on the recorded maximum graph size; see the details below.
• The new fields require Model Runner V2. Legacy sizing fields remain supported, but must not be mixed with the new prefill/decode list/cap fields.

Size limits, graph-mode exceptions, and legacy compatibility

• Limits and modes. Oversized entries are dropped with a warning: prefill ≤ max_num_batched_tokens, decode ≤ max_num_seqs. FULL and PIECEWISE use the prefill grid for all batches.
• Drafter coupling. DSpark, DFlash, DFlash2 and LiLiCorr are additionally bounded by the resolved max_cudagraph_capture_size / q. That recorded maximum covers all graph kinds, so changing the prefill cap can change draft FULL coverage. Example: DFlash K=7 with dynamic SD 7/3/1, prefill cap 128 → 8192: 37 → 48 draft graphs, reaching 256 → 512 requests. The other runners use their separate prefill/decode settings as described in the runner table.
• Legacy fields. cudagraph_capture_sizes and max_cudagraph_capture_size initialize both kinds with a deprecation warning. After resolution, they record the captured token sizes / maximum for buffer sizing. Compatibility does not mean identical graphs for AV or DSpark: with a legacy power-of-two token list through 512 at K=5, AV gets 25 sizes through 258 tokens (main: 10 through 512), and DSpark reaches 43 requests (main: 52). Model Runner V1 is unchanged and rejects the new prefill/decode fields.

Examples.

# Before: one cap for both kinds, in tokens.
--compilation-config '{"max_cudagraph_capture_size": 1024}'
# After: delete it to take the defaults, or set each kind in its own unit.
# Example for K=7: PIECEWISE up to 8192 tokens; target FULL up to 512 requests.
# 512 x (K+1) = 4096 tokens. Requires max_num_seqs >= 512; illustrates the budget, not an optimal tuning choice.
--compilation-config '{"max_prefill_cudagraph_capture_tokens": 8192, "max_decode_cudagraph_capture_requests": 512}'
# Prefill-only worker that does not execute decode: disable dedicated decode graphs.
--compilation-config '{"decode_cudagraph_capture_request_sizes": []}'

Results at the current defaults

Performance at matched SD settings. These measurements isolate the effect of this PR at the listed K; they do not establish gains over a tuned main baseline. Output tok/s, this PR / main, one untraced launch per build (--numa-bind, after a throwaway launch). ShareGPT_V3 prompts with their own lengths, --ignore-eos, temperature 0, 1024 output tokens:

Output throughput (tokens/s; higher is better).

Concurrent requests	DeepSeek + DSpark K=7, main → PR	Speedup	MiniMax + EAGLE3 K=3, main → PR	Speedup
16	4,759 → 4,916	1.03×	3,233 → 3,309	1.02×
64	10,127 → 11,799	1.17×	7,655 → 8,105	1.06×
128	10,157 → 12,448	1.23×	9,334 → 10,054	1.08×
256	6,683 → 24,582	3.68×	8,339 → 16,992	2.04×
512	12,120 → 28,358	2.34×	7,736 → 21,192	2.74×

Time per output token (TPOT, ms; lower is better).

Concurrent requests	DeepSeek + DSpark K=7, main → PR	MiniMax + EAGLE3 K=3, main → PR
16	3.05 → 3.00	4.42 → 4.41
64	5.65 → 4.67	7.29 → 6.88
128	10.34 → 5.66	11.16 → 10.17
256	34.1 → 8.3	29.1 → 12.4
512	33.1 → 14.3	56.6 → 19.1

More configurations (this head, one launch per build, ShareGPT through the chat endpoint with each model's template, 1024 output tokens; 16 / 64 / 512 prompts at concurrency 1 / 16 / 256, 1024 at 512):

model, config	c1	c16	c256	c512	GSM8K main → PR (same answers)	KV vs main
DeepSeek-V4.1-Flash, 256K, DSpark K=5 + adaptive verification	1.02	1.04	1.21	–	0.890 → 0.897 (91.3%)	0.66
DeepSeek-V4-Flash, DSpark K=2	1.00	1.02	1.76	–	0.950 → 0.948 (97.5%)	0.93
DeepSeek-V4-Flash, DSpark K=7, dynamic SD 7 / 3 / 1	0.99	1.03	1.57	1.54	0.952 → 0.951 (97.4%)	0.94
DeepSeek-V4-Flash, DSpark K=7 + adaptive verification	0.96	1.02	1.48	1.55	0.950 → 0.948 (96.8%)	0.78
Qwen3.8-Flash-Next-FP8, MTP K=3 (max_num_seqs 256)	0.99	1.08	1.42	–	0.967 → 0.968 (98.9%)	0.99
Kimi-Linear-48B-A3B, TP1, no SD	1.05	1.00	1.00	–	0.897 → 0.901 (97.4%)	0.99

• AV memory: the 0.66× / 0.78× values are measured KV capacity ratios. New allocator measurements show both larger graph pools and idle profiling cache; see the breakdown and fixes.

Correctness with AV enabled: this checks whether the new graph configuration reduces GSM8K accuracy. DeepSeek-V4 K=7 scored 95.00% on main → 94.77% on PR; no statistically significant drop was detected in this paired run. The table’s 96.8% is answer agreement, not accuracy. Paired results.

Run conditions. One untraced launch per build; small gains need repeated runs. Main: 60a7d568f4; PR: 30c305c944. Build details and commands are folded below; engine step times are in Appendix C.

Build details, commands and logs

AV paired correctness check: main 1253/1319 correct; PR 1250/1319. Of the changed outcomes, 18 went correct→wrong and 15 wrong→correct; exact two-sided McNemar p=0.73. This tests for an accuracy difference in this run, not bitwise identity or a proof of equivalence.

Concurrency	Prompts	Mean input tokens
16	192	216–218
64	384	245–251
128	256	242–244
256	768	235
512	1024	232–238

C64/C128 are single-launch measurements with finite request counts; throughput includes startup/drain effects. DeepSeek used an unmeasured warm-up to exclude a cold Triton compilation observed in the first attempt. These points are not independent repetitions of the other rows.

Build details. The original DeepSeek-V4 K=7 and MiniMax rows use a4975e7fd4, whose differences from 30c305c944 are tests and the AV grid, unused by these runs. C64/C128 use the current head. Both builds use the same compiled binaries. Earlier repeat launches varied by up to 10%; this is observed variability, not a confidence interval for these single-launch results.

# DeepSeek-V4-Flash (TP4 + EP, DSpark K=7)
vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --enable-expert-parallel --tensor-parallel-size 4 --attention_config.indexer_kv_dtype mxfp4 \
  --moe-backend deep_gemm_mega_moe --tokenizer-mode deepseek_v4 --numa-bind --cudagraph-metrics \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
# MiniMax-M3 (TP4, EAGLE3 K=3); env VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
#   NCCL_CUMEM_ENABLE=1 NCCL_MNNVL_ENABLE=1 NCCL_NVLS_ENABLE=1
vllm serve nvidia/MiniMax-M3-NVFP4 --block-size 128 --tensor-parallel-size 4 --trust-remote-code --kv-cache-dtype fp8 \
  --attention_config.backend FLASHINFER --attention_config.use_trtllm_attention true \
  --attention_config.indexer_kv_dtype fp8 --attention_config.minimax_m3_msa_decode_backend cutlass \
  --enable-prefix-caching --max-num-batched-tokens 16384 --gpu-memory-utilization 0.9 \
  --disable-flashinfer-autotune --stream-interval 20 --numa-bind --cudagraph-metrics \
  --speculative-config '{"method":"eagle3","model":"Inferact/M3-EAGLE3-GQA-V2","num_speculative_tokens":3,"attention_backend":"FLASHINFER","use_local_argmax_reduction":true}'
# Benchmark (one line per row; concurrency / prompts / output / seed = 16/192/1024/16, 256/768/1024/256,
# 512/1024/1024/512)
vllm bench serve --backend vllm --model <model> --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --ignore-eos --temperature 0 \
  --sharegpt-output-len 1024 --max-concurrency 16 --num-prompts 192 --seed 16

Server logs, bench JSONs, step traces and the run scripts are kept on our cluster and available on request.

Memory: three causes and their fixes

The memory increase is substantial today. Follow-up prototypes have reduced it in measured runs; the current-grid combination is being validated separately.

Model and experiment	Fixes applied	Added GPU memory before → after (GiB/GPU)	What this establishes
DeepSeek-V4 + DSpark K=7, earlier grid	Indexer row chunks + step-wise draft logits + aux-output release + shared capture stream	25.35 → 6.00 (main: 5.81)	KV capacity recovered from 0.65× to 0.989× of main
MiniMax-M3 + EAGLE3 K=3, earlier grid	Aux-output release only	9.33 → 4.51	Total graph pool fell 7.58 → 2.76 GiB; replay working storage remains
DeepSeek / MiniMax, current grid	Applicable fixes above	18.66 / 7.72 → pending measurement	These before values are current; earlier-grid after values are not substituted

The two completed experiments used earlier grids without the 4096-token decode limit. They demonstrate that the identified allocations can be reduced; they do not establish the current-grid after total. Current-grid validation measures memory and checks GSM8K again. The estimator fix is a separate accounting change: it can recover KV budget without shrinking graph tensors.

The three physical-memory issues are sparse-indexer logits, auxiliary hidden states retained across capture sizes, and DSpark draft logits. Each section names the affected implementation, explains the cause and fix, then gives a measured model example. Memory estimation is a separate accounting issue.

GB200, GiB per GPU (rank 0), with default grids and a 4096-token decode budget; no adaptive verification.

Added GPU memory means device usage after graph setup minus before it (the log calls this actual). It includes graph pools and other allocations. For AV, graph setup also runs the cost-curve profiling forwards. Weights and KV already allocated before this step are excluded.

Measurement details and source logs

DeepSeek-V4-Flash: TP4+EP, DSpark K=7, 1M context. MiniMax-M3: TP4, EAGLE3 K=3. Allocator breakdown: one instrumented startup per build, PR 79fb06ad41 (30c305c944 + instrumentation), main 9ac2d91b11 (60a7d568f4 + instrumentation). Sources: p0final/out/{dsv4,m3}_memb_{main,pr}/analysis.rank0.json and allocator snapshots, retained on the cluster. These are current-budget measurements; the earlier no-budget optimization stack is folded below.

KV counts are from the serving comparison (a4975e7fd4, equivalent to this head for these non-AV paths); the instrumented DeepSeek PR startup had 6,208,515 tokens. Rounding can affect the last digit. The outside-pool residual includes graph/runtime/library allocations; it has not been fully separated into graph-object bytes and other owners. It is not all graph objects. Allocations already retained from startup profiling are outside this real-capture delta: outside-pool memory retained after profiling was DeepSeek 2.20 → 1.40 GiB and MiniMax 1.49 → 1.14 GiB. These lifecycle measurements are not independent charges to add blindly. A complete pure graph-object attribution remains open. Here profiling means startup memory estimation; prefill means a model execution phase.

Actual added GPU memory (GiB/GPU; no AV)	DeepSeek main	DeepSeek PR	MiniMax main	MiniMax PR
Total	5.81	18.66	3.15	7.72
CUDA graph pools	4.326	17.051	1.799	6.127
Other PyTorch buffers and cached memory	0.195	0.891	0.598	0.854
CUDA / library allocations outside PyTorch	1.293	0.721	0.750	0.738

KV budgeting is separate from physical memory usage. The 29.19 GiB below is a software estimate, not another allocation: DeepSeek PR actually adds 18.66 GiB above. An inaccurate estimate changes the KV budget; the estimator follow-up addresses this separately.

Budget and resulting KV capacity	DeepSeek main	DeepSeek PR	MiniMax main	MiniMax PR
Estimated CUDA-graph memory, GiB	8.56	29.19	4.60	9.23
Measured KV capacity, tokens	7,660,426	6,208,214	3,575,296	3,358,336
KV capacity relative to main	1.00×	0.81×	1.00×	0.94×

Pool reservations and live-tensor peaks are different views of the same memory; do not add them together.

1. Sparse-attention indexer: chunk token-selection logits

Tensor and cause: logits = fp8_fp4_paged_mqa_logits(...) materializes BF16 scores over candidate KV positions; as_fp32_logits creates an FP32 copy for the selected top-k backend. Top-k selection consumes these logits before sparse attention uses the selected positions. The workspace grows with query rows × context length. This affects that implementation path, not every sparse-attention model.

Measured example — DeepSeek-V4-Flash: BF16 indexer logits and their FP32 conversion trigger 10.02 GiB of the 10.53 GiB target FULL pool increment, with the 4096-token budget already enabled (512 requests × q=8, 1M context). These are KV-selection scores, distinct from the vocabulary logits in issue 3.

Shape behind the 8.016 GiB: 512 requests × (K+1=8) = 4096 query tokens, with a 1M-token context limit. The indexer scores the compressed KV positions for each row. The allocator trace attributes two 4104 MiB pool segments to FP32 conversion (8.016 GiB total); this is reserved pool growth, not one 8 GiB live tensor. Capturing this shape keeps up to 512 concurrent uniform requests on FULL graphs. It is a coverage choice, not a mandatory size: a lower decode cap saves memory but sends larger batches to PIECEWISE/eager.

TODO: compute and convert logits in row chunks, reusing smaller buffers. #62169 covers SM90/SM120; the SM100 prototype is published in this fork for inspection, but is not merged or included in this PR. This applies to the affected sparse-indexer implementation.

Allocation that triggers new reserved space	Added reserved GiB	Why / next action
Indexer logits: BF16 → FP32 conversion	8.016	Large logits and limited reuse; chunk by rows (indexer row chunking).
Indexer logits: original BF16 output	2.004	Full-size logits; chunk by rows (indexer row chunking).
Attention output, _alloc_attn_out	0.250	Required workspace; investigate reuse.
Query normalization / RoPE / KV insertion, _fused_qnorm_rope_kv_insert	0.250	Required workspace; investigate reuse.
Remaining FULL capture sizes after the widest capture	0.006	Small additional pool growth
Total target FULL pool increment	10.525	Total reserved increase.

Measured result of indexer chunking: on the earlier grid, this fix alone reduced added GPU memory from 25.35 to 15.35 GiB/GPU. The current-grid after value is being measured. Replay still needs working storage; #62860 separately budgets eager workspace and does not itself shrink the logits.

2. Auxiliary hidden states: release outputs retained across capture sizes

Scope and cause: drafters that consume target auxiliary hidden states are affected when nested output references keep those tensors alive across PIECEWISE capture sizes. The current draft round needs its hidden states; retaining separate outputs for every capture size is unnecessary.

Pending fix: use weak references for nested auxiliary outputs (#63427, open and not merged; not included in this PR).

Measured examples: MiniMax + EAGLE3 retains 4.378 GiB within its 4.619 GiB target PIECEWISE pool; DeepSeek + DSpark retains 2.918 GiB. These are two instances of the same output-lifetime issue.

MiniMax target PIECEWISE memory	Main, GiB	PR, GiB	Cause / action
Auxiliary hidden states kept across capture sizes	1.224	4.378	Unnecessary retention; fix nested references (#63427)
Other reserved pool space	0.050	0.241	Remainder; not a separate workspace peak
Total target PIECEWISE pool	1.273	4.619	Mostly retained auxiliary outputs; fix above

The final pool saving depends on the working storage needed for replay. The 4.378 GiB is retained outputs, not MoE workspace; no MoE-chunking change is proposed from these measurements.

Memory after the aux-output fix: current-grid after-fix measurements are still TODO. Earlier-grid tests confirm the fix reduces memory, but their totals cannot be substituted for today's grids.

Measurement, GiB/GPU	Before fix	After fix
Current grid: MiniMax target PIECEWISE pool	4.619	TODO: measure
Current grid: DeepSeek target PIECEWISE pool	3.531	TODO: measure
Earlier grid: MiniMax total graph pool	7.58	2.76
Earlier grid: DeepSeek total graph pool	23.29	19.35

Earlier-grid validation: 72 PIECEWISE sizes, no 4096-token decode limit, aux-output fix only (design/FINAL_VALIDATION.md, job 39109). Retained aux tensors fell to zero; replay working storage still remains. Do not subtract today's 4.378 / 2.918 GiB from today's reserved pool to predict the after-fix total.

3. DSpark drafter: compute logits per step and reuse temporary storage

Problem: the generic DSpark sampling path computes all K steps' vocabulary logits before sequential sampling. Temporary logits grow with requests × K × vocabulary size. This affects models using that path, not just DeepSeek; custom sampling implementations can differ.

TODO: compute one step's logits, sample it, then reuse the temporary storage for the next step. A local step-chunking prototype exists; no public PR yet. The persistent logits cache needed for probabilistic verification remains separate.

Measured logits shape and memory (DeepSeek-V4, TP4, DSpark K=7): the largest draft graph covers 585 requests, or 4095 draft rows. With a 129,280-token vocabulary and BF16 (2 bytes/value), the trace shows:

Logits storage at the measured peak	Shape	GiB/GPU
Local TP shard	4095 × 32320	0.247
All-gather buffer	16380 × 32320 (same element count as 4095 × 129280)	0.986
Contiguous full-vocabulary copy	4095 × 129280	0.986
Logits subtotal		2.219

Shapes follow the code and BF16 configuration; the allocator trace confirms their storage sizes and simultaneous lifetimes. The drafter's total tensor peak is 2.313 GiB; its reserved pool is 2.994 GiB. These are overlapping views, not costs to add together. Step-wise logits can reduce the temporary buffers; the probabilistic-verification cache is separate.

DSpark measurement on DeepSeek-V4-Flash, K=7	Main	PR	Meaning / action
Draft-logits live peak, GiB	0.55	2.22	Temporary logits; step chunking
Drafter graph-pool increment, GiB	0.740	2.994	Includes storage serving those logits; not an extra logits charge

Additional reuse fix: capture target and drafter sequentially on one capture stream per GPU (#59537), reducing stream-local workspace duplication and allowing pool reuse. This is separate from step chunking. Savings with the current grids remain to be measured; the table is one configuration, not a universal DSpark cost. In the earlier combined prototype, after indexer chunking, aux-output release and step-wise logits were already applied, the shared-stream fix reduced measured capture memory from 8.43 to 6.00 GiB/GPU. This is an incremental result on that earlier stack; the current-grid after-fix total remains TODO.

Measured graph counts, capture-stage memory, and remaining workspace

Each stage reports the additional memory reserved by the graph pool, in GiB/GPU; the total is the sum of those increments. Graph count counts capture sizes; one PIECEWISE size may contain multiple CUDA graph objects. These tables cover the measured DeepSeek + DSpark and MiniMax + EAGLE3 deployments, so each includes its own drafter.

DeepSeek-V4-Flash + DSpark K=7: capture stage	Widest shape, main → PR	Graph count, main → PR	Main: added pool memory (GiB)	PR: added pool memory (GiB)
Target PIECEWISE: prefill, mixed, decode fallback	1024 → 8192 tokens	83 → 39	0.912	3.531
Target FULL: uniform verification/decode	128 → 512 requests, q=8	80 → 48	2.674	10.525 (breakdown below)
DSpark FULL: drafter's block routine	144 → 585 requests, q=7	80 → 51	0.740	2.994
Total graph pool		243 → 138	4.326	17.051

MiniMax-M3 + EAGLE3 K=3: capture stage	Widest shape, main → PR	Graph count, main → PR	Main: added pool memory (GiB)	PR: added pool memory (GiB)
Target PIECEWISE: prefill, mixed, decode fallback	1024 → 8192 tokens	83 → 39	1.273	4.619
Target FULL: uniform verification/decode	256 → 1024 requests, q=4	81 → 64	0.229	0.662
EAGLE3 step 0 PIECEWISE: after target prefill/mixed	1024 → 8192 tokens	83 → 39	0.088	0.662
EAGLE3 step 0 FULL: after compatible target uniform decode	256 → 1024 requests, q=4	81 → 64	0.025	0.000
EAGLE3 steps 1…K−1 FULL: one-token draft decode	1024 → 1024 requests, q=1	83 → 64	0.184	0.184
Total graph pool		411 → 270	1.799	6.127

For the measured MiniMax K=3 example, target and draft step 0 use q=4: main reaches 256 × 4 tokens, PR reaches 1024 × 4. Target capture sizes total 164 → 103; EAGLE3 drafter sizes total 247 → 167. A zero step-0 FULL pool increment means reuse of already-reserved space.

MiniMax target uniform-decode graph pool (FULL mode, 0.662 GiB):

Allocation	Added pool space, GiB	Cause / action
FlashInfer MoE tactic	0.56250	Working storage (live peak: 0.290 GiB); no chunking change proposed yet
MXFP8 GEMMs	0.09375	Working storage; reuse where possible
Other capture sizes	0.00586	Small additional reservation
Total target FULL pool	0.66211	Much smaller than auxiliary-output retention

MiniMax's PIECEWISE MoE live peak is 0.678 GiB; the FULL peak is 0.290 GiB. These use the reported pools and are not additional charges. Required working storage remains after fixing retention.

Graph/runtime objects and other outside-pool allocations are included in the total-memory table, but a complete pure graph-object attribution remains open. These two configurations do not establish the same breakdown for other models.

The separate memory-budget estimator determines how much KV cache is admitted; it does not change the size of the three allocations above. Its measurements follow the graph inventory.

Graphs per runner (real config resolution and CudaGraphManager on CPU, this head vs main; FULL_AND_PIECEWISE; GB200: 1024 sequences / 16384 batched tokens with the decode budget; off SM100: 1024 / 8192). Cells give graph count (capture sizes) and maximum coverage; parentheses explicitly label the main baseline. Target and drafter capture their own graphs. Adaptive verification changes how many draft tokens the target verifies per request; its target FULL graphs therefore support variable query lengths. Draft-generation graphs remain in the drafter rows.

runner	graph type, when it runs	sizes from	GB200: PR (main)	Other GPUs: PR (main)
Target (with or without SD)	PIECEWISE (tokens): prefill, mixed, and decode steps no FULL graph covers	prefill list / cap	39 up to 8192 tokens (main: 83 up to 1024)	51 up to 512 (main: same)
Target, uniform decode	FULL, uniform decode at q = 1 + K (requests)	decode list / cap, budget	no SD / K=1 / K=2 / K=3: 64 up to 1024 (main: 83 / 82 / 93 / 81 up to 1024 / 512 / 336 / 256); K=7: 48 up to 512 (main: 80 up to 128); K=15: 37 up to 256 (main: 64 up to 64)	K=3: 49 up to 128 (main: same); K=7: 48 up to 64 (main: same)
Target verification with AV enabled	FULL varlen (tokens): decode steps that mix query lengths up to 1 + K	adaptive-verification grid up to decode cap × (1 + K); exempt from max_decode_cudagraph_capture_tokens	K=5: 64 up to 6144 (main: 85 up to 1024)	K=5: 28 up to 498 (main: 53 up to 512)
EAGLE, EAGLE3, single-module MTP, Gemma 4 MTP: draft step 0	Own PIECEWISE / FULL captures: PW for prefill or non-uniform batches; FULL for compatible uniform decode (q=K+1)	Prefill token grid / decode request grid, budget	K=3: 39 + 64 (main: 83 + 81); K=7: 39 + 48 (main: 83 + 80)	K=3: 51 + 49 (main: same)
EAGLE / EAGLE3 / single-module MTP, K ≥ 2: later draft steps	FULL, q = 1, draft tokens 2 … K	Own request grid to max_num_seqs, common token budget; [] disables	64 up to 1024 (main: 83, or 97 at K=2)	64 up to 1024 (main: 51 up to 512)
DSpark (q = K)	FULL drafting routine (q=K)	Decode request grid; budget and recorded max / q	K=7: 51 up to 585 (main: 80 up to 144); K=3: 64 up to 1024 (main: 81 up to 336)	K=7: 48 up to 64 (main: 48 up to 71)
DFlash, DFlash2, LiLiCorr (q = K + 1)	FULL drafting routine (q=K+1)	Decode request grid; budget and recorded max / q	K=7: 48 up to 512 (main: 80 up to 128); K=15: 37 up to 256 (main: 64 up to 64)	K=7: 48 up to 64 (main: same)
multi-module MTP (q = K + 1)	Own FULL capture with all draft modules; used after a compatible uniform target decode	the target's decode list	K=3: 64 up to 1024 (main: 81 up to 256); K=7: 48 up to 512 (main: 80 up to 128)	K=3: 49 up to 128 (main: same)
dynamic SD 7 / 3 / 1 (EAGLE3)	target decode, one q per tier; one-token decode per K ≥ 2	per tier, up to the widest batch that drafts it	target 112 (main: 243); one-token 128 (main: 166)	target 176 (main: 147); one-token 128 (main: 102)
n-gram, draft model, extract_hidden_states	none, the drafter runs eagerly

• A drafter graph object costs 0.4–1.2 MiB, a target graph object 10–20 MiB. At 256 sequences and 8192 batched tokens, target decode is 37 graphs (main 35 without SD). GPT-OSS keeps main's 1024-token PIECEWISE sizes off SM100.

Closing the DeepSeek-V4 KV cache gap

The current-budget physical breakdown is above: DeepSeek-V4-Flash DSpark K=7 retains 0.81× of main's KV capacity, and MiniMax-M3 EAGLE3 K=3 retains 0.94×. The follow-ups address different owners and conditions; indexer chunking is specific to the affected sparse-indexer path, aux-output weak references apply where those outputs are retained, and the estimator needs checking across models.

Why the decode budget exists. The SM100 default of 4096 tokens is an interim memory safeguard against wide FULL captures, especially the target sparse-indexer workspace (issue 1); multi-token drafter allocations are also bounded. It is not a drafter-specific limit. Reassess the default after workspace fixes and measurements on the current grids; the configurable budget remains useful. Large K also has a compute cost (Appendix B).

Adaptive verification: measured memory and fixes

GB200, GiB/GPU. Graph setup includes target/drafter capture and AV cost-curve profiling. The original serving runs below measured the total memory increase and KV capacity; the new allocator measurements then separate what occupies that memory.

Configuration	Added GPU memory from graph setup (GiB), main → PR	Estimated CUDA-graph memory (GiB), main → PR	Measured KV tokens, main → PR	KV ratio
DeepSeek-V4-Flash, DSpark K=7 + AV	34.85 → 34.50	31.61 → 50.23	6,038,123 → 4,727,854	0.78×
DeepSeek-V4.1-Flash, DSpark K=5 + AV	33.38 → 33.79	25.36 → 45.63	14,955,942 → 9,908,490	0.66×

New measured breakdown — DeepSeek-V4, DSpark K=7 + AV. The similar totals hide a larger graph pool and less idle ordinary-allocator cache in the PR.

Added GPU memory during graph setup, GiB/GPU	Main	PR
Total	34.854	34.496
CUDA graph pools	4.326	20.180
Ordinary PyTorch allocator	29.186	13.176
CUDA / libraries outside PyTorch	1.342	1.141
Of ordinary allocator: completely idle AV profiling cache	28.990	12.285

Graph-pool component, GiB/GPU	Main	PR	Fix direction
Target PIECEWISE	0.912	3.531	Release retained aux outputs
Target FULL verification	2.674	13.654	Chunk indexer logits; current AV coverage is 1024 → 8192 tokens
DSpark FULL	0.740	2.994	Step-wise logits and shared capture stream

Conclusion: AV has both a real graph-pool increase and a cache/estimation problem. Clearing the ordinary idle cache (#61509) does not remove the persistent graph pool or runtime workspace requirements. The physical fixes above and the estimator fix must be evaluated separately; cache bytes cannot simply be converted into promised KV gains.

One MEMB-instrumented startup per build, frozen main 60a7d568f4 / PR 30c305c944; rank 0 and rank 3 agree. Totals reproduce the original DeepSeek-V4 runs. Source: avmem/out/job_39639/{dsv4_main,dsv4_pr} and design/AV_MEMORY_BREAKDOWN.md. AV’s 8192-token target grid is exempt from the uniform-decode budget; its 13.654 GiB FULL pool is a different configuration from the no-AV 4096-token / 10.525 GiB example. DeepSeek-V4.1’s paired allocator breakdown is still running.

TODO: release AV profiling cache (vllm-project#61509), correct the full-initialization estimate, then remeasure actual memory, KV capacity and serving performance. The fixes and their status are listed below.

Issue	Fix / status
AV cost-profiling cache contaminates the capture measurement	Release it before capture (#61509, merged upstream; absent from the frozen baseline)
Extrapolated FULL samples include cache releases; KV-init buffers are missed	Measure retained memory across complete initialization; local estimator prototype, separate follow-up
Remaining retained buffers / pool duplication	Attribute by stage and owner, then fix lifetimes and evaluate shared capture streams (#59537)

Source: design/P1_RESULTS_RAW.md, the underlying P1_dsv4sd / P1_dsv41 logs, and design/EST_BLOCKER.md. These totals are measured; a per-owner AV pool breakdown must not be inferred from them.

Memory-estimator TODO across models

Separate issue: total memory budget error after initialization. This includes graph-memory estimation and other unbudgeted buffers. Values are actual usage minus the configured GPU memory budget: positive = budget exceeded; negative = unused budget that could hold KV. Final head, one launch per build. Diagnostic instrumentation calls empty_cache() to separate terms, so these runs must be kept separate from the uninstrumented AV totals above.

Usage minus budget, GiB/GPU	DeepSeek-V4 K=7	DeepSeek-V4.1 + AV	MiniMax-M3	Qwen3.8-Flash-Next	Qwen3.8-27B	Kimi-Linear
main	+4.73	11.84 over budget	+2.54	+1.03	+3.48	+0.62
this PR	−3.81	5.55 below budget	+2.09	+0.61	+3.62	+0.27
this PR with the estimate fix	+0.59	+2.42	+0.42	+1.10	+0.54	+0.08

• Over-reservation comes from (a) sampled FULL-graph windows that include empty_cache() and are extrapolated (large only with DeepSeek's indexer warmup) and (b) the adaptive-verification profiling cache ([Bugfix][MRV2] Don't count adaptive verification profiling cache as CUDA graph memory vllm-project/vllm#61509). Under-reservation comes from (c) runner buffers allocated with the KV cache that the budget never counts (all models: 4.6 GiB DeepSeek-V4, 3.1 GiB Qwen3.8-27B, 1.6 GiB M3) and (d) runtime buffers larger than the profile (DeepSeek's eager indexer logits, [Bugfix][DSA] Reserve the decode indexer logits in the memory profile vllm-project/vllm#62860).
• On DeepSeek-V4.1 + AV, the combined estimator prototype recovered 32% KV capacity in one startup. This is a follow-up result, not a fix included in this PR or a saving attributable to [Bugfix][MRV2] Don't count adaptive verification profiling cache as CUDA graph memory vllm-project/vllm#61509 alone.
• The fix is independent of this PR (follow-up 5): about 70 changed lines in profile_cudagraph_memory, plus [Bugfix][MRV2] Don't count adaptive verification profiling cache as CUDA graph memory vllm-project/vllm#61509 and [Bugfix][DSA] Reserve the decode indexer logits in the memory profile vllm-project/vllm#62860, and 6–59 s more startup.

How much memory the follow-up fixes saved in earlier experiments

These measurements explain possible remedies, not the current 18.66 GiB breakdown. The 0.989× KV result is a combined prototype on an older head, not achieved by this PR alone. Savings overlap and cannot be subtracted independently from current totals.

On earlier grids, the budget reduced the logged capture delta from 25.4 to 15.9 GiB (KV 0.83×). With the final grids it is 18.66 GiB despite about 1 GiB less live memory: allocator snapshots show unreused target-pool space growing from 7.0 to 10.9 GiB. Capture layout and reuse matter as well as live-tensor size.

Earlier-grid prototype 4a3270c7ad, one launch per setting, rank 0. Physical fixes are applied cumulatively below; the final estimator row uses a separate accounting scope.

Measured step by step:

DeepSeek-V4-Flash DSpark K=7, 1M context	Measured GPU memory added during capture, GiB	KV vs main
Earlier prototype 4a3270c7ad, before the 4096-token budget	25.35	0.65×
+ indexer chunking (1)	15.35	0.90×
+ DSpark step-chunked logits (2) + #63427 (3)	8.43	0.966×
+ one capture stream per device, #59537 (2)	6.00 (main 5.81)	0.989×
+ full-initialization memory estimate (4), on both builds	estimate 12.82 for 13.34 held	0.9995× of main with the same fix

The stack keeps this PR's throughput (×3.0 at 256 requests), completes 1024 requests where main crashes, and scores GSM8K 0.949–0.955 against main's 0.950. On Kimi-K3 (8-layer dummy, DSpark K=7), #63427 plus step-chunked draft logits take KV from 0.82× to 0.94×.

Limitations

• Padding overhead. A batch rounds up to the next captured size. Against the earlier denser grid, the largest measured engine-time increase was 0.3% on DeepSeek and 1.2% on MiniMax’s prefill-heavy workload. Indexer scheduling remains a follow-up to reduce this cost.
• DeepSeek-family KV below main until the follow-ups land: 0.81× (DeepSeek-V4-Flash K=7, budget on), 0.66× (DeepSeek-V4.1-Flash with adaptive verification; the uniform-decode budget does not bound AV; physical memory and estimation both require follow-ups).
• Adaptive verification keeps a dense token grid (64 graphs at K = 5). It picks its verify budget from the captured sizes, so gaps cost acceptance. On DeepSeek-V4.1 DSpark K=5 (same server, grid switch): the prefill grid (nothing between 64 and 128 tokens) cut verify tokens per request 5.9 → 4.2 at 16 requests (acceptance −10.6%); multiples of 1 + K lacked powers of two (+0.8–1.2% engine at 64 / 256 requests); the dense grid and powers of two ∪ multiples of 1 + K stayed within ±0.25%.
• Off SM100 prefill, target decode, draft-prefill, DFlash and multi-module MTP defaults equal main's. Differences: one-token draft decodes reach max_num_seqs (64 graphs, main 51); DSpark reaches 64 requests at K = 7 (main 71); adaptive verification 28 graphs (main 53); dynamic SD 176 target decode graphs (main 147); GPT-OSS decode 64 (main 83) at the same reach. Computed, not measured on H100/H200.
• The memory estimate sizes the KV cache from a wrong graph-memory number, on main and on this PR (see Closing the DeepSeek-V4 KV cache gap); KV numbers here include its error until follow-up 5 lands.
• Deprecated fields with adaptive verification or DSpark do not reproduce main's graphs (Configuration).
• The widest one-token draft graph holds the drafter's logits (+0.25–0.37 GiB EAGLE3 / MTP, +1.02 GiB Gemma 4 E2B). Persistent capture outputs are sized for the widest graph (0.25–0.66 GiB at 8192 tokens) and not counted by the current estimate. Interactivity mode adds every token count up to 32 to the prefill grid (65 sizes); the decode grid has every count only up to 6 (main: 32).

Test plan and results

Validation: on GB200, configuration tests passed (110/110) and CUDA-graph manager tests passed (28/28); pre-commit is clean. Model correctness results are in Results. Commands, CPU-only limitations and detailed counts are folded below.

Test commands and detailed results

pytest tests/compile/test_config.py
pytest tests/v1/cudagraph/test_cudagraph_manager.py tests/v1/spec_decode/test_dynamic_sd_cug.py \
       tests/v1/worker/test_gpu_ubatch_slicing.py

• CPU, 30c305c944 (this head, CUDA_VISIBLE_DEVICES=). test_config.py: 98 passed; the other 12 start a vllm_runner and need a GPU. The other three files: 53 passed, 9 skipped, 1 failed (needs a CUDA device). Main fails the same GPU-only tests on CPU.
  • The coverage test helper honors the decode budget (K=7 covers 512 requests).
• GPU (GB200), 30c305c944: test_config.py 110 passed, test_cudagraph_manager.py 28 passed.
• Graph counts in this description: the real config resolution and CudaGraphManager on CPU, at 30c305c944 and at main.
• pre-commit on 60a7d568f4..30c305c944: clean.
• Evals: GSM8K and acceptance length per configuration on this head, in the Results tables; earlier-grid runs in appendix D.

Not a duplicate

• [Refactor][MRV2] Split decode and mixed CUDA graph capture planning vllm-project/vllm#61882 refactors decode/mixed planning without a config change.
• [Feature][MRV2] Allow sparse prefill CUDA graph captures vllm-project/vllm#62209 adds an opt-in prefill subset of the shared list.
• [Bugfix][Spec Decode] Ladder cudagraph candidates per uniform query length vllm-project/vllm#56136 and [Bugfix][Spec Decode] Keep a full uniform-decode batch reachable by FULL graphs vllm-project/vllm#62040 patch reachability for dynamic SD and DSpark. This PR covers both by construction, and their cases are in the tests.
• [Perf] Keep a 512-token floor on the default CUDA graph capture size vllm-project/vllm#60265 changes the default floor.
• [Bugfix][CUDA Graph] Weak-ref nested outputs of captured graphs vllm-project/vllm#63427, [Core] Reuse one CUDA graph capture stream per device vllm-project/vllm#59537, [Bugfix][Attention] Chunk sparse indexer decode logits by request vllm-project/vllm#62169, [Bugfix][DSA] Reserve the decode indexer logits in the memory profile vllm-project/vllm#62860, [Bugfix][MRV2] Measure the CUDA graph pool for the memory estimate, not the capture's free-memory delta vllm-project/vllm#59368 and [Core] Count V2's KV cache init memory against the KV cache budget vllm-project/vllm#60186 change memory behavior, not capture sizes. They are complementary, and are listed above and in the follow-ups.
• None of them separates the two kinds' units and caps.

---

Appendix A. How the grids were chosen

Data. Measured on earlier PR heads, traced. The candidate comparison used one server holding every candidate's graphs, without the decode budget.

The rule. Each grid captures a graph wherever padding to the next size would cost more than a fixed share ε of the step's time. A batch runs on the smallest graph that holds it, and padded rows are not free, so the spacing between sizes follows how fast step time grows with batch size, regime by regime.

Graph count costs little:

• per PIECEWISE size, a 15–20 MiB graph object and about 1 s of capture;
• per target FULL graph, 10–16 MiB and 0.5 s.

With an aux-state drafter, every PIECEWISE size also keeps 24–36 KiB per captured token until vllm-project#63427 lands.

(a) What a padded row costs, per operator class. Measured inside FULL decode replays with per-segment GPU timing:

operator class	padded row vs real row	why
dense GEMMs	~0 below ~300 rows, then ~1.0	memory-bound below the compute ridge
attention	~0 (sparse attention 0.3–0.75)	rows with seq_len 0 are skipped
DeepSeek sparse indexer logits (sm100_paged_mqa_logits)	~87 µs per padded request, 300–800× a real one	padded rows get context 0 and out-of-range request indices, so the schedule puts them all on the last SM, which runs them in series
routed experts, unmasked (TRT-LLM MoE: M3, Qwen3.8-Flash-Next, Kimi)	1.0–1.4	padded tokens are routed and computed
routed experts, masked (DeepGEMM mega-MoE)	0.09	padded tokens are masked
norms, activations, collectives	~1.0	row-wise
whole replay	DeepSeek-V4 1.44–1.65; M3 0.85–0.96; Qwen3.8-Flash-Next 0.53–0.56; dense 0.45–0.50	DeepSeek drops to 0.54–0.57 with the indexer scheduled over real rows only

(b) Decode step time, by regime in tokens t = q · n. Marginal cost per request, as a multiple of the linear-tail slope:

tokens	4–32	32–64	64–128	128–256	256–512	≥ 512
marginal cost	×3–9	×3–4.6	×1.8–2.1	×1.1–1.7	×0.9–2.2	×0.9–1.0

• Expert activation (below about 75–120 tokens): step time grows with the number of distinct experts.
• Dense GEMMs below the ridge (up to ~300–512 tokens).
• Linear above that: T = T0 + s · n, with an intercept of 650–950 tokens.
• Kernel steps at power-of-two token counts (for example +3.8 ms at 2048 → 2176 tokens on DeepSeek-V4). Both grids contain every power of two in their own unit. A decode batch runs n · q tokens, though, so it can still cross a step: at q = 3, 673 requests pad to 704, which is 2019 → 2112 tokens and crosses 2048.

(c) Decode step rule.

• Padding a batch of n by δ costs about e(n) · δ / n of its step time, where e(n) = n · c′(n) / T(n).
• Bounding the mean cost by ε gives the local step Δ(n) = 2εn / e(n).
• One envelope over all measured models, e ≈ n^(1/3) / 5, gives Δ(n) = 10 ε n^(2/3). Rounded down to a power of two, with ε = 0.065, that is the decode rule: 64 sizes at 1024.

(d) PIECEWISE rule. PIECEWISE step time is flat below 1–4K tokens and linear above (11.8–14.0 ms per 1K tokens per rank). The same rule with e_PW(t) = min(1, t / 4096) and ε = 0.035 gives the prefill grid: 39 sizes at 8192, which sum to 127,487 tokens (main: 35,335).

The coefficient 0.07 is 2 × ε, with ε = 0.035, used to choose grid spacing; it is not a measured 7% slowdown or a universal padding bound. Assuming batches land roughly uniformly inside a grid interval, average padding is about half its width. Under the fitted cost model, the nominal stride is min(t, 0.07 × max(t, 4096)), rounded down to a power of two; the next size is the next multiple of that stride, with the cap always included. For example, at t=4096, 0.07×4096=286.72 gives stride 256; at t=7680, 0.07×7680=537.6 gives stride 512 and the next size is 8192. At the default cap the exact sequence is the one listed in Configuration. This is an empirical cost-model choice, checked by the candidate-grid measurements below, not an end-to-end performance guarantee.

(e) Measured candidates. One server held the union of the candidates' graphs and switched grids at idle, in rotated order. Each cell is the worst change in engine GPU time against the earlier 72 / 72 grids:

decode / PIECEWISE sizes	DeepSeek-V4 DSpark K=7	MiniMax-M3 EAGLE3 K=3
64 / 39 (this PR)	≤ 0.3%	≤ 0.7%; 1.2% on a prefill-heavy mix
47 / 39	+2.4%	+1.9%
36–32 / 33–30	+4.1–5.0%	+3.0–4.2%
22 / 24	+5.4%	+4.9%

• Repeats of the reference grid read −0.88% to +0.14% (DeepSeek-V4) and −0.15% to +0.69% (M3) engine time, so the 64 / 39 row is inside that range except for M3's prefill-heavy mix.
• Prefill cap 8192. It is the largest cap that gains on every measured configuration and regresses none. Above it, DP > 1 with a gather-based MoE computes the other ranks' padded rows (14–20% slower on Qwen3.8-Flash-Next DP4+EP). On a dense model at TP1, PIECEWISE stops paying off above about 1.5K tokens.
• What would allow fewer graphs. The padded-row multiplier, not the grid's shape. With DeepSeek's indexer scheduled over real rows only, the same ε gives about 36 decode graphs instead of 64 (derived).

Appendix B. Why high concurrency + large K is not a target

Data. Earlier grids, no budget, untraced, one launch per configuration. DeepSeek-V4-Flash DSpark, tok/s on semantic prompts (chat, code, math; acceptance 3.4–4.3 at K=7):

requests (chat / code / math)	no SD	K=1	K=3	K=7	dynamic 7/3/1	adaptive verification
256	14.5k / 13.4k / 13.4k	20.3k / 19.8k / 23.2k	21.5k / 22.7k / 26.3k	13.6k / 19.6k / 24.1k	21.1k / 21.4k / 25.7k	21.4k / 21.7k / 25.4k
512	26.3k / 22.6k / 27.3k	30.6k / 25.9k / 33.5k	31.2k / 23.6k / 36.5k	22.9k / 17.7k / 28.3k	26.4k / 22.5k / 28.2k	29.6k / 26.5k / 34.0k
1024	28.8k / 23.5k / 29.3k	38.2k / 29.0k / 40.1k	36.4k / 27.6k / 41.5k	22.4k / 22.5k / 29.4k	31.4k / 25.7k / 32.7k	32.4k / 26.4k / 34.2k

• At 256–512 requests, K=3 or K=1 is fastest; on code at 512 requests, adaptive verification is (26.5k, against K=1 25.9k and K=3 23.6k). At 1024 requests, K=1 or K=3 is fastest.
• A fixed K=7 trails both, and at 512–1024 requests it is no faster than no speculative decoding. The verify step is compute-bound on n · (1 + K) tokens, so a large K spends compute on drafts that get rejected.
• MiniMax-M3 EAGLE3: K=3 is fastest from 64 to 512 requests and ties K=1 at 1024.
• At 16–64 requests, K=7, dynamic speculative decoding and adaptive verification are on par and fastest.

Why use a 4096-token decode budget? It trades some FULL-graph coverage for lower memory. In the earlier DeepSeek-V4 + DSpark K=7 tests below, 4096 preserved throughput at 512 requests within single-run noise; lowering it to 2048 caused OOM without indexer chunking. AV is exempt because bounding it reduced throughput without saving memory. One launch per setting:

• A 4096-token budget reduces measured capture memory to 15.9 from 25.4 GiB and KV 0.83× instead of 0.65×. Throughput is within single-launch noise at 512 requests, and −4–5% at 1024 on chat and math.
• Bounding adaptive verification cost 8–12% at 1024 requests and did not lower its memory, so it is exempt.
• A 2048-token budget is not safe without indexer chunking. Wide decode batches then run the indexer eagerly, and the memory profile no longer reserves its buffer: it ran out of memory at 1024 requests, as main does.

Appendix C. Where the performance comes from

Data. Earlier grids (72 / 72, PR 4a3270c7ad with tracing), no budget. Every engine step of one launch per build was traced, and steps were bucketed by kind and size. The per-bucket savings reproduce the measured engine-time difference within 1–10%. Gains come only from steps that run eagerly on main and on a graph here.

regime	what differs from main	example	per step, main → PR	engine time, main → PR
concurrency 1	nothing	DeepSeek-V4 / M3, 1K/1K	equal	68.5 → 68.5 s / 66.3 → 66.4 s
4–16 requests	the mixed step (1,036–1,144 tokens) is eager on main and PIECEWISE here	DeepSeek-V4, 1K/1K at 16: 450 mixed steps	163 → 44 ms	148.1 → 88.9 s
		M3, 1K/1K at 16: 351 mixed steps	229 → 57 ms	171.5 → 110.4 s
high concurrency, small K	mixed steps of 1K–8K tokens become PIECEWISE. Decode is FULL on both builds up to main's reach (336 requests at K=2, 256 at K=3)	M3 K=3, 1K/1K at 256: 465 mixed steps	213–215 → 68–104 ms	135.1 → 71.3 s
above 1024 / (1 + K) requests	decode is eager on main and FULL here	DeepSeek-V4 K=7, 1K/1K at 256	decode 157 → 37 ms; mixed 193 → 61–74 ms	141.8 → 45.7 s
		M3 K=3, 1K/1K at 512	decode 180 → 59 ms; mixed 214–216 → 96–116 ms	138.9 → 60.8 s
prompts above 8192 tokens	none: the step exceeds the cap and is eager on both builds	8K/256 at 16	equal	12.4 → 12.4 s

Final grids. Build 147a0ced88 (this PR's grids before the decode budget), tracing on, 2 launches per build; an earlier run than the Results table. Medians, main → PR:

workload	eager steps	decode step, ms	mixed step, ms	padded requests per decode step
DeepSeek-V4, 16 requests	2–3 → 0 of ~2,870	12.2 → 12.3	47 → 43–44	0.00 → 0.08–0.10
DeepSeek-V4, 64 requests	57–62 → 1	15.3 → 15.5–16.3 (+4%)	69–70 → 43–44	0.2 → 0.6
DeepSeek-V4, 256 requests	672–675 → 3–4 of ~910	94–97 → 26–27	175–178 → 52–53	0.09 → 2.2–2.7
M3, 256 requests	300–308 → 3–4	25–27 → 26	212–216 → 60–61	0.6–0.7 → 1.7–1.9
M3, 512 requests	671–674 → 7–8	114–119 → 38	214–219 → 85–88	0.35 → 3.4–3.5

• On DeepSeek-V4 at 64 requests, decode steps are 4% slower, because main's list is finer at 33–64 requests. Mixed steps −37% outweigh it.
• Dense model at TP1 (Qwen3.8-27B). A PIECEWISE graph shortens a mixed step of ≥ 1.5K tokens by 14–21%, but the next step absorbs the GPU work. The gain is limited to steps of about 1.1K tokens (−5% engine time at 16 requests).

Appendix D. Per-model validation on the earlier grids (superseded by the Results tables)

Data. Earlier grids, PR 4a3270c7ad (72 / 72 graphs, the same widest graphs and coverage as this head), no budget, untraced. Main is 60a7d568f4.

• Launches. DeepSeek-V4 and M3 ran 3 launches per build in rotated order after a throwaway launch. The other models ran one launch per build.
• GSM8K. 1319 questions, greedy, at concurrency 256. Acceptance length is measured over the GSM8K window only.

Model / config; parentheses = ISL/OSL in tokens	c1 throughput PR/main	c16 throughput PR/main	c256 throughput PR/main	GSM8K PR / main (same answers; same-build floor)	KV vs main
DeepSeek-V4-Flash, TP4+EP, DSpark K=7	1.00	1.72 (1K/1K)	3.48 (1K/1K)	0.948 / 0.949 (97.3%; 97.1–97.4%); acceptance 4.445 / 4.445	0.65 (0.83 with the budget)
DeepSeek-V4-Flash, DSpark K=2	1.00	1.39	2.13	0.946 / 0.946 (97.1%; 97.1–97.4%)	0.91
DeepSeek-V4.1-Flash, 256K, DSpark K=5 + adaptive verification	0.99	1.43	2.42	0.898 / 0.899 (90.5%; 90.9%)	0.63
MiniMax-M3-NVFP4, TP4, EAGLE3 K=3	1.00	1.55 (1K/1K)	1.99 (1K/1K)	0.864 / 0.869 (84.1%; 84.8%); acceptance 2.561 / 2.560	0.91
Qwen3.8-Flash-Next-FP8, TP4+EP, MTP K=3 (max_num_seqs 256)	1.00	1.27	1.84	0.969 / 0.969 (99.1%; 98.3%)	0.99
Qwen3.8-27B, TP1, MTP K=3	1.00	1.05	0.97 (TPOT equal)	0.964 / 0.964 (99.5%)	0.95
Kimi-Linear-48B-A3B, TP1, no SD	1.00	1.00	1.00 (TTFT p50 −24%)	0.898 / 0.903 (3 servers each; below)	0.98
Kimi-K3 architecture, 8-layer dummy weights, DSpark K=7	0.97–1.00	0.99	1.16	n/a	0.82

Kimi-Linear correctness: no regression observed on the current head. GSM8K accuracy is 89.7% on main → 90.1% on PR, one evaluation per build. The earlier-grid results in the table above are historical measurements, not a current correctness failure.

Other notes.

• MiniMax correctness: no regression observed on the current head. Across three GSM8K evaluations per build, mean accuracy is 86.20% on main → 86.58% on PR. Each evaluation uses 1,319 questions, 5-shot prompts and a 256-token output limit. Logs: p0final/out/m3_g_{main,pr}{2,3,4}/gsm8k.json.
• Noise. Same-build spreads on the single-launch rows: up to ±10% (Qwen3.8-27B ShareGPT), and up to ±25% on main DeepSeek-V4.
• Stability. No OOM, crash or startup failure on the PR in any launch.

Appendix E. Follow-ups

#	follow-up	problem	motivation and expected gain	link
1	SM100 indexer decode logits chunked by rows	The logits (rows × max_model_len / 4, BF16 + FP32) are allocated whole: 12 GiB at 1024 × 8 rows	Graph pool 25.35 → 15.35 GiB, KV 0.65× → 0.90× (measured). Chunking happens only at 1M context with K ≥ 3 and many requests; otherwise it is today's single call. Whether the current decode budget can be lifted still needs measurement	branch fix/dsa-indexer-decode-chunk-sm100; extends vllm-project#62169 (SM90/SM120); vllm-project#62860
2	DSpark / DFlash step-chunked draft logits	The draft logits of all K rows are computed before the step loop	DSpark stream 5.22 → 2.25 GiB (DeepSeek), 5.65 → 2.29 GiB (Kimi); GSM8K and acceptance unchanged	ours, not yet a PR
3	Aux-state retention	weak_ref_tensors converts only the outer tuple	−3.9 / −4.8 / −11.4 GiB on DeepSeek-V4 / M3 / Kimi (earlier grids); also helps main	vllm-project#63427
4	One capture stream per device	A drafter's widest graph cannot reuse the target's free blocks	−2.43 GiB on the DeepSeek stack	vllm-project#59537 (draft)
5	Measure memory across the full initialization sequence	Error at init end is −5.6 to +11.8 GiB across six models on main and this PR: extrapolated sample windows that include empty_cache(), the AV profiling cache, and runner buffers outside the budget	Error +0.08 to +2.42 GiB after the fix; KV +32% on DeepSeek-V4.1 + AV; no OOM where main crashes. Open: up to 2.4 GiB of profiling residue on capture streams (DeepSeek-V4.1), which needs freeing at profiling teardown	ours, not yet a PR; replaces vllm-project#59368 + vllm-project#60186; needs vllm-project#61509 (merged) and vllm-project#62860 or follow-up 1
6	DeepSeek indexer schedule over real rows	Each padded decode row costs ~87 µs in series	Padded / real 1.44–1.51 → 0.54–0.57, TPOT −1 to −3%; about 36 decode graphs instead of 64 (derived)	prototype
7	A smaller adaptive-verification grid	AV keeps 64 dense token sizes, because gaps shrink its verify budget	Powers of two plus n · (1 + K) measured within ±0.25% engine time; a rule derived from the decode regimes in tokens might need fewer	not started
8	DSpark dynamic SD drafting only k(n) tokens	DSpark still drafts K=7 under dynamic SD: 11–23% behind fixed K=1 at 512–1024 requests	Dynamic SD at K=1–3 throughput (guess)	not started
9	GLM-5.x and Hy4 validation	Both carry the DSA indexer, so follow-up 1 should matter	–	runs incomplete
10	Defaults for dense single-GPU models	Wider graphs cost a dense 27B on one GPU 4.7% of KV cache for no throughput gain (Appendix F)	A lower prefill cap and decode budget when PIECEWISE stops paying off (about 1.5K tokens here)	not started

Appendix F. Dense models on one GPU (Qwen3.8-27B): measured trade-offs and tuning evidence

What we measured. Qwen3.8-27B with MTP K=3 on one GB200, final head against main, ShareGPT prompts with 1024 output tokens:

• Throughput: the initial untraced launch measured PR/main ratios of 0.84 at c16 and 0.87 at c256. A traced rerun measured 1.01 and 0.995 (2266 vs 2243 tok/s and 6440 vs 6474 tok/s), with similar per-step GPU times. These differing runs are reported here rather than selecting one pair for the summary table.
• KV cache is 4.7% smaller (1.532M vs 1.608M tokens), because the wider graphs take memory on a GPU that has little to spare: the 8192-token PIECEWISE graph and the 4096-token decode graphs (1024 requests × 4) account for about half each (same-build A/B: main's capture sizes restore 1.608M tokens; main's PIECEWISE sizes alone give 1.572M).
• At 256 concurrent requests the scheduler then runs about 149 instead of 156 requests at once (0.955, the KV ratio). Requests queue in waves, so the TTFT median moves from about 12 s to about 20 s while the mean and throughput barely change.

Why the wider graphs do not pay off here. A dense 27B model on one GPU is GPU-bound from about 1.5K tokens per step: a PIECEWISE graph shortens a mixed step by 14–21%, but the next step grows by the same amount, so the pair takes the same time (1.5–2K tokens: 147 vs 151 ms; 2–3K: 188 vs 188 ms). The host launch time that the graph removes was already hidden behind GPU work. Large MoE models with tensor or expert parallelism are the opposite: many small kernels per layer make the host the bottleneck, which is where this PR's gains come from. Padding costs this model nothing measurable (decode batches padded to 160 requests run as fast as main's 152 / 156 graphs; GEMM tile quantization).

Tuning direction for dense models on one or a few GPUs: if wider graph coverage reduces KV capacity without improving throughput, try lowering the prefill capture cap. Consider lowering decode coverage only when the workload rarely reaches those batch sizes. Validate throughput, tail latency and KV capacity on the intended workload before choosing limits; no specific preset is recommended yet.

[Pb314314]

Review of e01a8e5 (commit 1 d76fffc5e3 has the same capture plans as the earlier cg/pr-a@9b2c7c2db3; I checked 2880 CPU configs). Code is in good shape; most findings are about the description's claims, plus one real behavior delta (adaptive verification, inline).

Description: claims that need fixing

1. "Capture plans vs main on CPU (240 configurations) ... identical" is not backed. design/diff_capture_plans.py documents python -I, which ignores PYTHONPATH, so both sides imported main. Rerun with -P -s and the current API on 2880 configs (SM100 on/off × max_num_seqs {1,8,72,128,256,512} × K {0,1,3,5,7,16} × dynamic SD {none, 3-tier} × mnbt {2048, 8192} × {default, max1024, max8192, pow2 list} × 3 modes), commit 1 vs main 60a7d568f4:

   graphs	vs main
   PIECEWISE / token-keyed	identical, 2880/2880
   target uniform decode, static K	identical except explicit deprecated lists, which only gain coverage (e.g. pow2, max_num_seqs=8, K=5: main stops at 6 requests, PR adds 8)
   target uniform decode, dynamic SD	differs in 512 configs: unreachable graphs dropped (q=4 above 8 requests), tiers denser; per tier no lost coverage; −13 graphs on average
   one-token draft decodes	narrower in 348 (the documented delta), denser at the same max in 972
   adaptive verification	differs, see inline on get_varlen_decode_cudagraph_token_sizes
   GPT-OSS override, off SM100	PIECEWISE identical; FULL decode grid changes, max never lower

2. "other GPUs, nothing set; or only the deprecated fields: same as main" holds for prefill and static-K target decode only. Add dynamic SD, explicit lists, GPT-OSS (all gains or neutral) and AV (losses, inline).

3. Test command lists test_lilicorr.py but not tests/v1/worker/test_gpu_ubatch_slicing.py, which this PR edits. On a GB200 node: test_cudagraph_manager.py 27 passed, test_dynamic_sd_cug.py 6 passed, test_lilicorr.py 33 passed, test_gpu_ubatch_slicing.py 29 passed, test_config.py 97 passed / 4 failed (test_use_cudagraphs[*], which also fails on main on this node: the model never compiles, num_graphs_seen 0). Running all files in one process makes 12 forked tests fail with "Cannot re-initialize CUDA in forked subprocess"; run test_config.py on its own.

4. "The renaming and refactor commits change no capture plan for these configurations": verified. 42fbe5b0ac vs e01a8e5: target and draft plans identical in 2880/2880. AV plans are identical for the validated SM100 default configs (K=5, max_num_seqs 256/512); elsewhere the only change is one extra graph at the cap when the cap is off the grid.

5. Commit 2 scope: verified. Off SM100, or with any deprecated field set, commit 2 changes no plan (2520 + 840 AV configs identical to commit 1). Only SM100 with nothing set changes.

[Pb314314]

Review summary (e01a8e5)

Data and the six inline comments are in the review above: #2 (review)

Verified

• Commit 1 (d76fffc5e3) has the same capture plans as cg/pr-a@9b2c7c2db3 in all 2880 configs.
• Commit 2 changes only SM100 with nothing set. Off SM100, or with a deprecated field set, 2520 configs (plus 840 AV configs) match commit 1.
• "The run used 42fbe5b0ac; the renaming commits change no capture plan" holds:
  • target and draft plans match in 2880/2880;
  • AV plans match for the validated SM100 default configs (K=5, max_num_seqs 256/512).
• Unit tests, per file on GB200:
  • test_cudagraph_manager.py: 27 passed
  • test_dynamic_sd_cug.py: 6 passed
  • test_lilicorr.py: 33 passed
  • test_gpu_ubatch_slicing.py: 29 passed
  • test_config.py: 97 passed, 4 failed. The failures are test_use_cudagraphs[*], which also fail on main on this node: the model never compiles, so num_graphs_seen is 0.
  • The tests that failed on SM100's ceiling now pass with off_sm100.

To fix, by priority

1. The description's "240 configurations identical to main" is not backed. diff_capture_plans.py runs python -I, which ignores PYTHONPATH, so both sides imported main. Replace the table with the 2880-config results in the review.

2. Adaptive verification differs from main off SM100 by default and with the deprecated fields. This is the only regression. The token list is converted to a request cap, multiplied back by q, and the grid is regenerated. That loses information in two ways:

   • Cap rounding. When q does not divide the cap, the top is lost. K=5: 1024 → 1008, 512 → 498.
   • Explicit list ignored. An explicit deprecated list is replaced by the default grid. pow2 list, K=5: the cap falls from 512 to 258.

   Either use main's resolved token list on these two paths (capped at max_num_seqs × q), or list both deltas under Known trade-offs.

3. "Other GPUs, or only the deprecated fields: same as main" holds only for prefill and static-K target decode. Dynamic SD, explicit deprecated lists and the GPT-OSS override also change. All of these are gains or neutral, but state them.

4. Test command. It omits tests/v1/worker/test_gpu_ubatch_slicing.py, which this PR edits. Also, running test_lilicorr.py in the same process as test_config.py makes 12 forked tests fail with "Cannot re-initialize CUDA in forked subprocess". Run them separately.

5. Nits.

   • prefill_cudagraph_capture_grid duplicates the inline default list in _set_cudagraph_sizes.
   • decode_cudagraph_capture_grid and main's request_counts are two request grids; say why.
   • The legacy/new mixing check passes when the lists happen to match. Add a comment that re-resolution needs this.
   • DSv4.1's metadata graphs ignore dynamic SD. Add an assert, or use the same mapping as the manager.

[Pb314314]

Adaptive verification (AV): full list of capture-plan differences vs main

Follow-up to the inline thread on vllm.py:747, which listed two of these. Plan: keep the code as is and document the differences under Known trade-offs. Text to paste is at the end.

Method. CPU capture-plan dump (no model and no capture), comparing main (60a7d568f4) with this PR (bf156180e9, the same tree as e01a8e5). Each side runs CudaGraphManager(..., varlen_decode=True) for the target model.

• Grid: SM100 on/off × max_num_seqs {1, 8, 72, 128, 256, 512} × K {1, 3, 5, 7, 16} × max_num_batched_tokens {2048, 8192} × sizing {default, max_cudagraph_capture_size=1024, =8192, cudagraph_capture_sizes=[1, 2, 4, …, 512]}.
• That gives 480 configs per mode, all with static K. Dynamic SD + AV is not covered.
• FULL_AND_PIECEWISE and FULL_DECODE_ONLY give identical results.

Result. 254/480 configs are identical. 226 differ: in 76 the AV cap goes down, in 66 it goes up, and in the rest the cap is the same but the graph set differs.

Why

In this PR the AV sizes are prefill_cudagraph_capture_grid(min(max_decode_requests × q, max_num_batched_tokens)). Main took them from the resolved shared token list, capped at max_num_seqs × q. That gives four differences:

#	Difference	When	Example (q = K+1)	Effect
1	Cap rounds down to a multiple of q (token → request → token round trip)	q does not divide the shared max	off SM100 default: 512 → 498 (K=5), 510 (K=16) max_cudagraph_capture_size=1024: 1024 → 1008 (K=5), 1020 (K=16)	the top 2–16 AV token counts run eager
2	Explicit cudagraph_capture_sizes is ignored for AV; the default grid is used instead	deprecated explicit list	[1, 2, 4, …, 512], K=5, max_num_seqs ≥ 128: cap 512 → 258 (K=16: 272); K=3/7, max_num_seqs ≥ 128: 10 → 51 graphs	254 token counts run eager; more graphs than the user asked for
3	Main's uniform-decode sizes are dropped from the AV list (main appended request_counts × q)	q is not a power of two (K=5, 16 here)	K=5: 6 and 12 are gone, so 1-request and 2-request steps pad to 8 and 16 K=16: 17, 34, 68 and 408 are gone	slightly more padding at low concurrency; no lost coverage
4	Above 1024 the grid steps by 256 (prefill grid); main stepped by 16	sizing cap above 1024 (e.g. max_cudagraph_capture_size=8192)	K=5, max_num_seqs=256, cap 1536: 115 → 85 graphs; worst padding goes from 15 to 255 tokens	fewer graphs, more padding for AV steps above 1024 tokens

Separate from these: on SM100 with nothing set, the AV cap grows with the new decode default. For example, K=5 with max_num_seqs=512 and max_num_batched_tokens=8192 goes from 1024 to 3072. This comes from commit 2, adds coverage, and is the intended change.

AV plans for the validated GB200 configs (DSv4.1 DSpark + AV, K=5, max_num_seqs 256/512) are the same as 42fbe5b0ac, the commit the model runs were done on.

Correction to my compilation.py:71 comment

I wrote that prefill_cudagraph_capture_grid and the inline default list in _set_cudagraph_sizes "agree today". That holds only up to 1024 tokens. Above 1024 the inline list steps by 16 and prefill_cudagraph_capture_grid steps by 256, and by 512 above 4096. That gap is difference vllm-project#4.

PIECEWISE plans on the default (off SM100) and deprecated paths still match main, because those paths keep the inline list. Commit 1 alone matched main in 2880/2880 configs. So the duplication point stands, but the inline path cannot simply call prefill_cudagraph_capture_grid without changing PIECEWISE plans above 1024.

Suggested Known trade-offs text

Adaptive verification graphs. AV graph sizes are now prefill_cudagraph_capture_grid(min(max_decode_cudagraph_capture_requests × q, max_num_batched_tokens)) on every path. Main used the shared token list. Compared with main:
(1) the AV cap is a multiple of q, so when q does not divide the old cap, the top few token counts run eager (off SM100 default with K=5: 512 → 498);
(2) an explicit cudagraph_capture_sizes no longer shapes AV graphs (e.g. a power-of-two list up to 512 with K=5 now caps AV at 258 tokens);
(3) the uniform sizes q and 2q are no longer added, so with K=5, 1-request and 2-request steps pad from 6 and 12 to 8 and 16;
(4) above 1024 tokens the AV grid steps by 256 instead of 16.
On SM100 defaults the AV cap grows with max_num_seqs (K=5, 512 seqs: 1024 → 3072 tokens). AV plans for the validated configs are unchanged from the benchmarked commit.

[Pb314314]

[Claude] v3 is at 3f707d4f0f. Item by item, against the review summary and the AV follow-up:

1. The "240 configurations" claim is replaced by the 2880-config results from the review, together with the scope of commit 2 and commit 3.
2. Adaptive verification: all four differences from main, and the growth on SM100, are under Known trade-offs, as planned. The code is unchanged.
3. "Same as main" now says per kind what differs. Gains or neutral: dynamic SD, explicit deprecated lists, the GPT-OSS override, one-token draft decodes. Losses: adaptive verification, and DSpark's q = K graphs (below).
4. Test commands run in two processes, and tests/v1/worker/test_gpu_ubatch_slicing.py is added.
5. Nits: the mixing-check comment and the DSv4.1 metadata-graph note are in f940752. The two prefill grids and the two request grids are explained in the description and in the thread replies.

New in v3:

• Commit 3: one-token draft decodes (EAGLE/EAGLE3/MTP) cover max_num_seqs on every GPU. It closes the gap that was measured at −9% / −7% / −2% (c128 / c256 / c512, Qwen3-8B EAGLE3 K=7, off-SM100 graph sets). The same rule as a prototype measured at main's throughput, for +0.16–0.44 GiB per GPU. A GB200 run of the head commit is queued.
• A per-kind graph-count table in the description: main vs PR, target vs drafter, GB200 vs H100/H200 defaults. It also surfaced one more delta. DSpark's q = K graphs follow the target's request cap, so off SM100 they cover slightly fewer requests than main: 71 → 64 at K=7, and 100 → 83 at K=5. This is listed under Known trade-offs. Extending multi-token drafters waits for [Core] Reuse one CUDA graph capture stream per device vllm-project/vllm#59537 (follow-up 1).

Commits 1 and 2 differ from e01a8e5 only in those comments.