[jianc99]

This is component 7 of the DFlash adaptive-verification series. Original implementation and component-only review: jianc99#7. Integration and full-system results: #46182.

Depends on #46187. Until those land, the diff against upstream main includes prerequisite changes. Compact verification on GDN additionally needs #46189.

Updated onto upstream main (6a319e7). All applicable pre-commit checks passed on this branch. The GPU tests and benchmark results below were run previously on the original source branch; they have not been rerun after this upstream update. Submission preparation used AI assistance.

---

Stacked on #46187 (online acceptance estimator). On a GDN target (Qwen3.8), compact mode also needs #46189 (packed ragged GDN verify).

Motivation

DSpark already has adaptive verification. With SGLANG_RAGGED_VERIFY_MODE=cap-accept or compact, the scheduler plans a per-step verify budget from a measured cost table and the drafts' confidence and gives the budget to the most likely drafts (global top-k by survival); compact then runs the packed ragged verify in token-tier CUDA graphs, while cap-accept verifies the whole block and stops acceptance at the planned length. That path is open to DSPARK only, and it needs a trained confidence head. DFLASH drafts, including DFlash2 selector drafts, have no confidence head, so DFLASH always verifies the whole block, even where most of it gets rejected.

Modifications

DFLASH reuses the DSpark machinery: the planner, the budget objective, top-k allocation, ragged packing, compact graphs, cutoff accept, the confidence relay, and the cost table with its profiler. The new piece is where the confidence comes from; the other changes below are fixes and gates that DSPARK shares (listed at the end of this section).

• Confidence. OnlineAcceptanceEstimator ([Feat] spec: online acceptance estimator for drafts without a confidence head #46187) is a per-position logistic model, refit online from verify outcomes.
  • Its feature is logit(q_max) of the selector row each draft token was drawn from, at the request's temperature (temperature 1 for greedy rows). It scores the draft distribution, never the drawn token, so trimming stays lossless.
  • The draft CUDA graph computes the feature and the confidence next to the selector walk, in one kernel.
  • After each verify, the estimator observes the verified and accepted draft counts. None of this syncs with the host.
• Wiring. The DFLASH worker builds the DSpark verify planner (taking the worker's confidence) and the target-verify executor, provides the overlap budget-prepare hook, and publishes its confidence into the existing relay. The DSPARK-only gates now cover the DFLASH family: supports_ragged_verify, the confidence relay, and the scheduler's record and pinned-budget controls.
• Cutoff accept. Every DFLASH accept path stops at the planned verify length.
  • Selector sampling passes the cutoff to DSpark's sampling kernel.
  • Greedy accept caps through AcceptGreedy, which now also takes precomputed argmax ids.
  • The bonus comes from the last verified row, as in DSPARK.
• Ragged window. The anchor is read at the block's real row stride. DFLASH blocks are gamma + 1 wide; DSpark's are gamma wide.
• mRoPE. Spec-verify mRoPE positions no longer view the packed tokens as [bs, T / bs], which failed for ragged layouts on Qwen3.5/3.8 targets. Multimodal deltas follow the per-request verify lengths.
• TP.
  • As with a DSPARK confidence head, every rank computes the same confidence from the same draft: the estimator's updates are deterministic and observe the broadcast acceptance.
  • The relay used to skip the lagged copy when it had not finished yet. Readiness is rank-local, so the relay now waits for it, and the next scatter waits (device side) for the previous ring copy. This is the source change of [Fix] dspark: Keep TP ranks planning the same verify budget under overlap scheduling #42717 (without its new test).
  • Every rank therefore plans the same budget and graph tier. The lengths are broadcast from rank 0, as before.
• Small batches. A batch whose shortest plan already replays the full block's compact graph prices no budget and verifies every draft; when the batch's full block is itself a captured tier (every batch of up to 8 requests here), it also skips packing and scattering. This removes the per-step planning and packing cost where trimming cannot help: always for a single request, often for small batches.
• Cost table. sglang.benchmark.dspark_sps_profiler accepts DFLASH servers.
  • A pinned profiler budget (--fracs) is now scheduled even without a table, so the 2-D compact table can be measured from scratch.
  • The scheduler refuses to pin a budget when SGLANG_SIMULATE_ACC_LEN is above 0 and not 1.0, because a simulated acceptance could run past the trimmed lengths.
• Guards.
  • Ragged modes need a selector (DFlash2) draft and are refused under attention DP.
  • compact is refused at startup when the target attention backend cannot verify packed ragged tokens (supports_ragged_verify_graph). A GDN hybrid target (Qwen3.8) needs [Feat] gdn: run target verify on packed ragged tokens (ReplaySSM) #46189.
• Other.
  • Compact verify scatters rows to the block layout row by row, which also works on a vocabulary shard, so DFLASH keeps its deferred target-verify vocabulary gather in every mode.
  • The default static mode is unchanged, for DFLASH and DSPARK.

What DSPARK sees: in both ragged modes, the relay waits for the lagged copy instead of skipping it, and the scatter waits for the previous ring copy; the scheduler refuses to pin a budget under a simulated acceptance length other than 1.0 (cap-accept already refuses that setting at startup). In compact mode only: a pinned budget is scheduled without a table (cap-accept already did this); small batches skip budget pricing, and full-block plans take the cached uniform layout, without packing when the batch fills a captured tier; on mRoPE targets, spec-verify positions are indexed per request instead of failing the [bs, T / bs] view. These were not measured on a DSPARK draft.

Accuracy Tests

Qwen3.8-27B target, DFlash2 block-8 draft, GB300. Client: SPEED-Bench first turns.

• Unit tests (CPU and GPU): 116 passed before the estimator's last commit in [Feat] spec: online acceptance estimator for drafts without a confidence head #46187; the same files pass on [Feat] spec: DFlash adaptive verification integration (do not merge) #46182, which has it. They cover:

  • greedy and selector-sampling cutoff accept (accept length, bonus row, trimmed counts);
  • the selector-row feature and the estimator observations;
  • the planner with the worker's confidence, and the gates;
  • the pinned budget and its SGLANG_SIMULATE_ACC_LEN guard;
  • the startup refusal;
  • the relay resolving an in-flight copy;
  • mRoPE ragged positions;
  • the kernel parity sweep (DFLASH anchor stride, argmax ids).

  The new GPU cases fail with the cutoff or the relay change reverted.

• No regression at full budget. T=0, c1, cap-accept with the budget pinned to the full block: the outputs of 12 prompts (up to 768 tokens each) are identical to static verification, token for token.

• Trimming. T=0, c1, cap-accept with the budget pinned to 50%.

  • Two runs give identical outputs (deterministic).
  • Against static verification, 4 of 12 outputs are identical. The rest diverge at near-tie tokens after 118-1156 characters, for example "think through requirements. User" vs "think through requirements.\n\nUser".
  • That is the same kind of divergence static speculative decoding itself shows against plain target decoding (3 of 12 identical).

• Overlap scheduling. cap-accept with a cost table plans the same verify lengths with overlap on and with --disable-overlap-schedule: 7.53 vs 7.51 verify tokens per request in steps of 1-16 requests, 3.84 vs 3.80 in steps of 17-64 requests.

• TP4 consistency. In test builds, each rank logged its budget and verify-token tier on every decode step. They were identical on all 4 ranks in every step:

  • cap-accept, overlap on: 8502 steps;
  • cap-accept, overlap off: 6203 steps;
  • compact on GDN: 7253 and 7461 steps;
  • without broadcasting the confidence (as this PR now does): 5729 steps.

  TP4 compact serving runs without hangs or errors from c1 to c512 at T=1, and at T=0 at c16 and c64 (acceptance length 4.02 and 3.87).

Speed Tests and Profiling

T=1 (top_p 0.95, top_k 20), up to 4096 output tokens, bf16 mamba state. Decode tok/s = per-request decode rate (output tokens after the first / time after the first token), times concurrency; c1 is the single-request decode rate. Static and compact were run on the same node in the same window.

The compact column needs packed GDN verify for Qwen3.8, so it runs this PR merged with #46189: ReplaySSM, packed conv and GDN verify, including SubSir's deduplicated conv window tile store. It uses the 2-D cost table the profiler measured on that stack. On this PR alone, compact on a GDN target is refused at startup.

TP1, --max-running-requests 128, decode tok/s:

concurrency	static	compact (this PR + GDN packed verify)	change
1	330	329	-0.3%
4	1233	1146	-7.0%
8	2249	2084	-7.3%
16	3649	3542	-2.9%
32	5597	5398	-3.6%
64	6940	6956	+0.2%
128	8141	9622	+18.2%

The c64 and c128 static numbers are the mean of two earlier runs on the same stack.

TP4, --max-running-requests 512, decode tok/s:

concurrency	static	compact (this PR + GDN packed verify)	change
1	654	627	-4.1%
4	2381	2243	-5.8%
8	4176	3965	-5.1%
16	6754	6167	-8.7%
32	9889	9119	-7.8%
64	12759	12392	-2.9%
512	18982	22055	+16.2%

With the budget pinned to the full block (every draft verified; no packing when the batch fills a captured tier), TP4 compact decodes at 4018 / 6372 / 9222 tok/s at c8 / c16 / c32: 1-3% above the planned budget and still 4-7% below static. So most of the gap below c64 comes from the compact graph path itself, not from trimming; see "Known limitation".

compact pays off at high concurrency: +18% at TP1 c128 and +16% at TP4 c512. This tree does not include #46186 or #46188, and at TP1 c128 and TP4 c512 the running cap equals the concurrency, which reads higher, slightly more for compact. With all seven PRs merged and a cap above the concurrency, #46182 measures +14.4% at TP1 c128 and +12.5% at TP4 c512 (T=1).

Use the 2-D table (--fracs in compact mode). The 1-D diagonal table from a static-mode run overprices the savings at large batches and over-trims.

Known limitation

Below c64, compact decodes slower than static: 3-7% at TP1 c4-c32 and 4-9% at TP4 c1-c32; at c64 the two are within 3%. Compact CUDA graphs are captured per token tier, and a tier of T tokens carries min(T, max_bs) request slots, so a small batch replays a graph sized for many more requests (up to 8x the real batch when every block is full). Steps that verify the full block are not routed back to the batch-size-keyed verify graphs that static mode uses. Static mode, the default, is unaffected. The fix, routing full-block steps to the batch-size graphs (or capturing compact graphs per request bucket and token tier), is a graph-runner change left for a follow-up. Until then, keep static below full load.

Related upstream work

• [Fix] dspark: Keep TP ranks planning the same verify budget under overlap scheduling #42717 (same change). The relay commit (resolve relayed verify confidence identically on every TP rank) is the same code change in overlap_utils.py. Drop that commit if [Fix] dspark: Keep TP ranks planning the same verify budget under overlap scheduling #42717 lands first.
• [Fix] dspark: Produce correct SPS cost tables when profiling a server without a table #42711 (same fix, theirs is stricter). It schedules a pinned profiler budget without a cost table and refuses pins under simulated acceptance, as the profiler commit here does. Its guard is stricter: it also refuses SGLANG_SIMULATE_ACC_LEN=1.0 unless SGLANG_SIMULATE_ACC_METHOD=match-expected, because multinomial sampling around 1.0 can accept two or more tokens. Prefer its version: if it lands first, the profiler commit keeps only the DFLASH enablement of the profiler and the scheduler controls; if this PR goes first, it should adopt that stricter guard.
• [Fix] Honor DSpark SPS profiler budget pins and handle ragged mRoPE positions #38897 (overlaps three commits). It carries the same pinned-budget fix; a ragged mRoPE fix that maps each packed token to its request by a search over the per-request verify offsets, as this PR and [Fix][GDN] Route DFlash/DSpark commits through ReplaySSM spec fold #35725 do (it sets padding positions to 0, where those two give padding a zero delta); and a full-budget shortcut to the cached uniform layout in schedule_layout. The small-batch commit here does more: it skips budget pricing when even the shortest plan replays the full-block graph, decides by graph tier rather than by budget, and skips packing and scattering for full-block layouts. Both edit the same lines of schedule_layout, so whichever lands second rebases onto the other; the mRoPE commit here can be dropped if either of those lands first. Its GDN part overlaps [Feat] gdn: run target verify on packed ragged tokens (ReplaySSM) #46189.
• [Feat] speculative: Online acceptance estimator for DSpark confidence-scheduled verification #44866 adds an online estimator for DSPARK under the same module and class name as [Feat] spec: online acceptance estimator for drafts without a confidence head #46187, the base of this PR. It also adds an online_confidence planner flag on the same DSparkVerifyPlanner lines as this PR's external_confidence (constructor, confidence-head refusal, carries_confidence), and changes the padding fill of the compact scatter in run_compact, which the small-batch commit here wraps. If it lands first, this PR moves onto it and uses its flag (see [Feat] spec: online acceptance estimator for drafts without a confidence head #46187).
• [Speculative] Derive the DSpark verify-budget cost table at CUDA graph capture #35413 derives the cost table at CUDA graph capture (opt-in). It also adds a full-window shortcut (_budget_covers_full_window) to the verify-all branch of schedule_layout, on the same lines as the small-batch commit here, so whichever lands second rebases. This PR uses the existing offline profiler and does not wire the startup derivation into the DFLASH worker.
• [Fix] dspark: Skip top-p renormalization on compact verify padding rows #43374 changes how compact padding rows go through top-p in the DSpark accept and verify paths that DFLASH now also uses; whichever lands second rebases.

---

CI States

Latest PR Test (Base): ❌ Run #40044049685
Latest PR Test (Extra): ❌ Run #40044049597
Latest PR Test (AMD ROCm 10): ❌ Run #40044049553