[knapcio]

Summary

• add opt-in DFLASH_VERIFY_TOKENS={1,3,7} while DFlash2 retains its trained DFLASH_TOKENS=7 proposal block
• publish one batch-uniform proposal prefix to the target and advertise the same width through vLLM's native target-graph schedule
• backport the active runtime-width GDN/KDA metadata fix proposed in vLLM PR #53542 for the exact pinned image source
• preserve DFLASH_VERIFY_TOKENS=0 as an exact stock/full-width rollback: the new overlays are neither mounted nor applied
• keep structured-output publication on the existing stock path
• preserve inline DFlash overrides over .env, validate before graph arithmetic, and generate all required capture sizes up to the configured concurrency

This is a small graph-aligned integration for the pinned GLM/V2/sparse-MLA stack, not a new speculative-decoding algorithm. The wide-proposal/narrow-verification pattern has upstream prior art in the still-open adaptive-verification PR #52559.

Controlled A/B

Two DGX Sparks, TP=2, max_num_seqs=2, EXL3 target, FP8 target KV, DFlash2 draft TP=1. Each row is the median of two fixed English-prose prompts with one 512-token completion per prompt, temperature 1, top-p 0.95, thinking off, single stream, and a warmed idle endpoint.

Mode	Proposal / verify	Median decode tok/s	Observed delta vs stock 7/7
Speculative decoding off	—	13.91	-15.58%
Stock DFlash2	7 / 7	16.48	—
Static narrow block	3 / 3	19.25	+16.83%
Wide proposal, narrow verify	7 / 1	19.29	+17.10%
Wide proposal, narrow verify	7 / 3	20.04	+21.64%

Both final 7/3 prompts improved over stock (+22.25%, +21.03%) and static 3/3 (+4.49%, +3.74%). This is deliberately reported as a small workload-specific observation, not a population estimate or a claim about other prompts, languages, concurrency, or hardware.

A two-prompt code smoke had a +4.44% median but mixed per-prompt movement (-4.19%, +14.27%), so this PR does not claim a code speedup. The GDN backport's independent timing contribution is also not isolated; it is included as the active-width correctness fix and validated as part of the final stack.

Correctness and rollback

The target still performs ordinary standard rejection sampling over the published prefix, so the sampling distribution is unchanged by construction. This stack was not bitwise repeatable across identical temperature-0 runs, so output hashes are not presented as a proof of equivalence.

Live validation on the final 7/3 server:

• ordinary requests published exactly 3 draft tokens per verification step (687 / 229)
• strict JSON-schema output returned the exact requested 1..40 sequence
• required tool choice returned get_weather({"city":"Warsaw"})
• structured output retained stock grammar handling without an error
• head and worker were healthy with zero container restarts
• corrected proposal-k=7 boot warmup passed 13/13 requests, including concurrency 2
• observed GPU peaks were 67 C / 61 C; the run used a 90 C pause threshold and never approached it

Tests

python3 tests/test_dflash2_verify_width.py
python3 tests/test_gdn_runtime_width.py
python3 tests/test_start_overrides.py
bash -n start.sh
python3 -m compileall -q overlay tests
git diff --check

The tests cover fixture application, idempotence, incomplete-marker rejection, source-drift failure, ordinary/structured publication behavior, invalid widths, stock rollback, caller-over-.env precedence, dynamic graph sizes beyond 32, recovery subcommands with malformed tuning, and wiring on both TP ranks. The image's live GPU overlay self-check also passed on both nodes.

Upstream references

• vLLM adaptive verification (open draft): [Spec Decode] Add graph-aware adaptive K for DFlash vllm-project/vllm#52559
• vLLM active runtime-K GDN metadata fix (open): [Perf][SpecDecode] Preserve active runtime-K width in GDN metadata vllm-project/vllm#53542

[MiaAI-Lab]

Live A/B on this 2× DGX Spark kit

We applied this branch on the recipe cluster (TP=2, EXL3, fp8 target KV, DFlash2 draft TP=1, max_num_seqs=4) and compared stock 7/7 against live 7/3 (DFLASH_TOKENS=7, DFLASH_VERIFY_TOKENS=3, overlays applied, num_speculative_tokens_per_batch_size=[[1,4,3]]).

Mechanism: confirmed

7/3 really does publish a 3-token verify prefix. Ordinary prose steps were exactly 3.0 draft tokens/step (e.g. 495/165, 537/179). Spec per-pos accept was non-zero only on positions 0–2; 3–6 were 0. Stock 7/7 stayed at 7.0 draft tokens/step. Strict JSON-schema sequence = 1..40 still returned the exact array. Health stayed 200.

Prose: not a +21% on this kit

Same protocol as the PR body (two English-prose prompts, 512 tokens, temperature 1, top-p 0.95, thinking off, single stream, warmed idle endpoint). Prompts were a hash-map explanation and a photosynthesis essay (the PR does not list the original two).

Arm	hashmap tok/s	photosynthesis tok/s	mean
Stock 7/7	25.87	26.05	25.96
7/3 round 1	30.55	25.80	28.18
7/3 round 2	27.82	27.23	27.52

Mean of four 7/3 completions: 27.85 tok/s (~+7.3%). Hashmap averaged about +13%; photosynthesis about +2% (round 1 was slightly slower than stock). This is the same ~26 tok/s prose regime this kit already measures at temp 0, not the 16.48 → 20.04 table. n is still tiny and temp 1 makes the completions different each run, but we did not reproduce a uniform ~+22% on both prompts.

Structured / high-accept: large regression

This is the blocker for using 7/3 as a recipe default. Count-1-to-200, temp 0, thinking off, 5×400 — the same tests/bench_decode.py --structured protocol as the kit notes (stock median 61.7 tok/s, accept 0.918, 6.43 accepted/step):

Arm	Median tok/s	Accept	Accepted / step
Stock 7/7 (this kit, prior lab)	61.7	0.918	6.43
Live 7/3 (this boot)	40.6	1.000	3.00

All five 7/3 runs sat in 40.46–40.71. Per-pos was 1.0 / 1.0 / 1.0 / 0 / 0 / 0 / 0. Verify is saturating at K=3 and throwing away tokens 4–7 that stock still accepts at 0.91–0.83 on this workload. That is about −34% vs the structured/code path this recipe publishes.

We did not same-session A/B structured on 7/7 after the 7/3 restart (stock structured number is the existing lab 5×400 median). The accepted/step drop 6.43 → 3.00 matches the overlay, so we do not think this is just boot noise.

What we need from you

We are not merging this as a default. Default 0 / opt-in is still a discussion, but the PR body currently leads with +21% English prose and does not show the structured hit.

Please:

1. Publish the two prose prompts from the PR table, or re-run that table on this public recipe (max_num_seqs=4 as well as 2) so we can match it.
2. Same-session structured + code A/B (count-1-to-200 and the sparkDash-style code prompt) for 7/7 vs 7/3 vs static 3/3. If 7/3 loses ~30% there, that needs to be in the summary, not only “we do not claim a code speedup.”
3. How should operators choose? If this is only for low-accept prose, say that explicitly and keep DFLASH_VERIFY_TOKENS=0 as the recipe default. If you have a path that keeps high-accept K=7 for structured/grammar batches without the advertised-K=3 mismatch, describe it.
4. Mixed ordinary + structured batches. The overlay keeps full width when has_structured_output_reqs is set, while the scheduler still advertises K=3. JSON-only worked; mixed batches were not tested here.

Until (2) is measured and the write-up matches the workloads this image actually serves, we would not merge, and we would not recommend anyone set DFLASH_VERIFY_TOKENS=3 on this kit.

[knapcio]

Thanks — this is a valid blocker, and I agree the current write-up overweights the two-prompt prose result.

The exact prompts behind my table were:

1. Write at least 900 words of polished magazine prose about how a city changes during a summer thunderstorm. Use concrete sensory detail, varied sentence rhythm, and continuous paragraphs. Do not use headings, lists, quotations, or discuss these instructions. (SHA-256 20e59ae8…)
2. Write at least 900 words of engaging narrative nonfiction about a night train crossing Europe. Keep it factual in tone but vivid, with continuous paragraphs and natural transitions. Do not use headings or lists. (SHA-256 5a727eb5…)

My existing code smoke used two different temperature-1 generative implementation prompts. It was not the public sparkDash clamp_00…clamp_49 workload, and I do not yet have the requested same-session 7/7 vs 7/3 vs 3/3 structured/code matrix.

More importantly, your item 4 is correct: the handler keeps K=7 for any batch containing a guided request while num_speculative_tokens_per_batch_size globally advertises K=3. A single guided-JSON smoke passing does not validate that mixed-batch contract.

Please do not merge this revision. DFLASH_VERIFY_TOKENS=0 remains the only recommended/default setting. I am setting up a bracketed public-prompt run at max_num_seqs=4: stock 7/7 A1 → 7/3 → static 3/3 → stock 7/7 A2, each with the count-1-to-200 and exact sparkDash code prompts (warmup + 5×400, temperature 0, thinking off), plus a synchronized ordinary/guided mixed-batch check on 7/3.

Unless that overturns the high-accept regression, I will put the negative matrix and the mixed-batch mismatch prominently in the write-up and close the PR rather than turn this small integration into a batch-adaptive scheduler project.

[knapcio]

Bracket complete; I am closing this rather than recommending the width split.

Same-session protocol on the public recipe: max_num_seqs=4, one 400-token warmup plus 5×400 measured completions, temperature 0, top-p 1, thinking off, fixed seeds. A1 and A2 are stock 7/7 controls; B is proposal 7 / verification 3; C is static 3/3.

Workload	A1 7/7	B 7/3	C 3/3	A2 7/7	B vs control midpoint
Count 1–200 structured	66.9603	41.6436	41.4392	66.8721	-37.77%
sparkDash-style code	63.5715	39.9767	40.1336	63.2061	-36.93%

Control drift was only -0.13% structured and -0.57% code. Every arm produced exactly 400 tokens with finish_reason=length, and the output hash was identical across all four arms within each workload, so content differences do not explain the result.

The counters show the mechanism clearly. B and C reached 100% acceptance but were capped at exactly 4 output tokens per verification step. Stock reached 7.82/7.86 on structured A1/A2 and exactly 8.0 on code. B was only +0.49% versus C on structured and -0.39% on code: keeping a 7-token proposal does not recover the useful accepted positions once verification is capped at 3.

I also ran three fail-closed ordinary/guided mixed probes. Despite sub-millisecond request-start skew and valid outputs/counters, actual decode overlap was 0.0 s, 12.404 s, and 0.0 s. Admission is nondeterministic here, so I am not selecting the favorable attempt or publishing an aggregate A/B claim.

The independently observed low-accept prose gain (~7.3%) is real enough to motivate the experiment, but it is not a simple general recipe win against these repeatable ~37% high-accept regressions and the mixed-batch contract complication. DFLASH_VERIFY_TOKENS=0 remains the only recommended setting. Both ranks stayed healthy; bracket peak was 65°C/60°C.

Thanks for catching the missing workload coverage and for the live retest.