[weireweire]

Motivation

DFlashDraftInputV2.prepare_for_decode allocates KV pages only for the requests whose committed length plus the 2-block reserve crosses their allocated length. With page size 256 and accept length ~3.5, each request crosses a page about every 73 decode steps. In a DEP32 c5768 nsys profile (~36 running per rank), a rank allocated on ~35% of its steps, and each allocation step cost ~0.78 ms of host time, mostly get_last_loc (95 us), paged alloc_extend (578 us) and the req_to_token assign (56 us).

Under attention DP this sits before the DP agreement. With 32 ranks allocating independently, some rank pays it on nearly every step, and the whole step waits for that rank.

Modifications

SGLANG_OPT_DFLASH_DECODE_TOPUP_PAGES=K (default 2; 0 turns it off; capped at 8 pages) changes what happens on a step that must allocate anyway:

• Every request below K pages of headroom is topped up to K pages beyond its page-aligned need, in the same alloc_for_spec_decode call.
• The top-up is bounded by the req_to_token row width.
• Already-running requests then next need pages about K pages of decode later. A rank's allocations are batched into shared allocation steps instead of one step per request page crossing.
• The top-up is skipped unless the allocator's free tokens cover the paged-extend eviction demand (needed + top-up + bs x page_size, the demand take_extend passes to make_room) plus 3x the top-up. When it is skipped, the step allocates exactly as before. So on the full/SWA token pools the top-up does not trigger a prefix-cache eviction that the step would not have done anyway. (The NPU DSV4 C128 sidecar pool is not part of this check; this path was not run on NPU.)
• The top-up is skipped under HiSparse (with a one-time warning only when the env is set explicitly). HiSparse's spec-decode path always evicts, and it sizes its host pool by kv_allocated_len.

Admission becomes slightly more conservative. Decode admission subtracts a per-running-request decode reservation from the free size, and the free size already excludes the topped-up pages. So up to K pages per running request are counted twice. On a PD decode worker the reservation is --num-reserved-decode-tokens (512), so at page size 256 and K=2 the worst case doubles it (512 more tokens per running request; 98k tokens at 192 running). This is self-limiting: the guard caps a top-up at a quarter of the free tokens beyond needed + bs x page_size, so as free tokens approach the reservation the top-ups shrink and stop, and decode uses up the pages already held.

Behavior change: the top-up is on by default (K=2) for every DFlash-family decode with a page size above 1. K=2 is the only value measured; it holds at most 2 extra pages per running request, and the free-space guard above makes it fall back to exact allocation when memory is tight.

The change is independent of #47676, #47677 and #47678 (the other DSV4.1 decode host-path changes). It was measured stacked on them.

Accuracy Tests

All runs: GB300, DSV4.1-Flash, K=2, on a 2026-10-30 nightly.

• Consistency check (diagnostic overlay, not part of this PR): on every allocation step, the check verifies that each newly allocated [cur, nxt) range of req_to_token is nonzero, is not shared with any other request's new range, and that kv_allocated_len equals nxt. Result: 870 checks, 0 bad.
• GSM8K, 1P1D DEP16 with real DSpark acceptance, same run: 96.82% (control without the top-up, with the other three changes: 96.82%).

Speed Tests and Profiling

DEP32 c5768 (10P1D). This comparison isolates the top-up: both arms have #47676-#47678, and the arm adds SGLANG_OPT_DFLASH_DECODE_TOPUP_PAGES=2.

	without top-up	with top-up
decode step p50, 32-47 running	13.9 ms	13.7 ms
steady-window per-GPU throughput (prompt + completion tokens, mean of minutes 8-20, x0.96 both arms)	660.6k tok/s/GPU	687.8k tok/s/GPU
p90 interactivity	219.8 tok/s/user	223.4 tok/s/user

The throughput gain (+4.1%) comes from a single pair of separate runs, and each run has one dip bin (control at minutes 10-12, arm at minutes 14-16). Run-to-run drift on this shape can reach ~3%, so the throughput delta is not conclusive on its own. The step-time reduction (-0.2 ms p50) is the signal.

9P1D DEP16 bs192 c7480 checks a high-batch, KV-heavy shape. It compares all four host changes against none, so it is not an isolated measurement of the top-up:

• decode step p50 is 0.2-0.26 ms lower in every max-running bucket (33.64 -> 33.38 ms at 192 running);
• throughput is neutral: 1127.2k vs 1127.2k tok/s/GPU at p90 100.0 vs 101.2 tok/s/user;
• KV headroom: full KV usage max 0.55 -> 0.54; SWA usage mean 0.063 -> 0.112 (max 0.16 -> 0.23); 0 retracted requests in both.

At these loads the low-free skip path is not exercised. In the GSM8K run the consistency check also logged the free size: on every allocation step, free tokens were at least 15.8x the skip bound. The GPU runs used the guard before the bs x page_size term was added; the term adds 10k tokens at 40 running and 49k at 192.

Checklist

• Format with pre-commit
• GPU validation on cluster (above)
• Existing DFlash / decode-prep unit tests (test_decode_bookkeeping_ownership.py, test_dflash_overlap_hostsync.py, test_dflash_draft_graph_eligibility.py, test_pp_cp_rank_offsets.py, test_schedule_batch_prepare_for_decode.py) pass on CPU with the flag off and with SGLANG_OPT_DFLASH_DECODE_TOPUP_PAGES=2

---

CI States

Latest PR Test (Base): ❌ Run #41405346973
Latest PR Test (Extra): ❌ Run #41405346992
Latest PR Test (AMD ROCm 10): ❌ Run #41405346896

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

Required checks failed on 82b3dd2; this PR cannot be merged until they pass.

Lint (run)
Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

[nvpohanh]

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

[nvpohanh]

@Qiaolin-Yu could you review this? thanks