[brandonfzhang]

Motivation

EAGLE target verify gives every request exactly num_tokens_per_req (= draft_token_num) query rows: EagleVerifyInput builds qo_indptr as a constant-stride arange. FlashInfer's FA2 CUDA-graph planner cannot see that from the ragged CSR alone, so it sizes its query tile for the worst case (total_rows - bs + 1 rows per request). For a typical verify batch (32 q-heads, TP8, 4 draft tokens) that means running a 64-row tile on a 16-packed-row problem.

flashinfer-ai/flashinfer#6144 added a uniform_q_len keyword to BatchPrefillWithPagedKVCacheWrapper.plan(): a host-side promise that every request has exactly that many query rows, letting the planner size the tile for uniform_q_len * gqa_group_size packed rows. The same hint is the dispatch key for FlashInfer's specialized SM90 fp8-KV verify kernel from that PR.

Modifications

One wiring change in flashinfer_backend.py: call_begin_forward adds uniform_q_len=num_tokens_per_req to the paged plan kwargs for EAGLE verify batches (SpecInputType.EAGLE_VERIFY; DFLASH / NGRAM verify and draft extends keep the stock plan).

Capture fixes the verify kernel's query tile from the hint, so every plan through a captured verify wrapper must carry the same hint: the hint is passed on both plan paths, the real plan() and the SGLANG_OPT_FLASHINFER_VERIFY_HOST_PLAN sync-free replay (fast_prefill_plan, which previously pinned the argument to 0).

Requires the FlashInfer >= 0.7.2.post1 pin (#46926), the first release with the plan(uniform_q_len=...) keyword; this PR should land after it. An earlier revision carried an inspect.signature capability probe to tolerate older FlashInfer; per review it is dropped.

Adds a CPU unit test for the gating (only EAGLE verify batches are hinted, unset num_tokens_per_req is not a promise).

Benchmarks

Qwen3.5-397B-A17B-FP8, TP8/EP8 on 8x H200, EAGLE (steps 3 / topk 1 / draft 4), 256k-context agentic workload at 10 concurrent sessions, 1 h steady state per arm. Reported metric: normalized interactivity (output tokens/s per request over each request's own decode, so it isolates decode speed from TTFT and queueing). Each arm is compared against a production run of the same day on the same hardware; cross-day control drift was 1.8% at p50 and 3.7% at p90.

configuration	p50	p90
baseline (no hint, same-day control per arm)	113.5 / 115.5 tok/s	63.6 / 65.9 tok/s
uniform_q_len hint (planner tile only)	130.5 tok/s (+15.0%)	71.0 tok/s (+11.6%)
hint + FlashInfer's SM90 fp8-KV verify kernel (dispatches behind the same hint)	136.1 tok/s (+17.8%)	73.6 tok/s (+11.7%)

Corresponding ITL p50: −15.0% and −19.4%. Output tok/s +6%, accept length unchanged (3.30 vs 3.28), output lengths unchanged, GSM8K accuracy at parity (197 vs 196 / 200 correct, paired).

Checklist

• Format your code: pre-commit run --all-files
• Add unit tests
• Update documentation / benchmarks as needed

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #41449065916
Latest PR Test (Extra): ❌ Run #41449065830
Latest PR Test (AMD ROCm 10): ❌ Run #41449065807

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[YAMY1234]

Overall the code logic LGTM. Maybe we can wait for the FlashInfer version bump, then retest the performance and run CI.

[nvpohanh]

[by Claude Code] Severity: style | Confidence: High

Issue

The Benchmarks section gives results but no commands, so others cannot rerun them. This PR has no effect with the pinned FlashInfer 0.7.1, so the FlashInfer build also matters.

Fix

Please add:

• The server launch command for Qwen3.5-397B-A17B-FP8 on 8x H200 (TP8/EP8, EAGLE 3/1/4), and which FlashInfer build you used (one that includes flashinfer-ai/flashinfer#6144). For the "hint only" row, say how the SM90 kernel was turned off, for example FLASHINFER_DISABLE_EAGLE_VERIFY_FP8KV_SM90=1.
• The benchmark command for the 256k-context agentic workload, with the trace or dataset and its length and session settings. If the trace is private, a public long-context run such as python -m sglang.bench_serving against the same server is enough.
• The GSM8K command behind "197 vs 196 / 200".

[nvpohanh]

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

[nvpohanh]

[by Claude Code] Pushed b06b088 to address the latest review comments.

This PR still depends on the FlashInfer upgrade to 0.7.2.post1 in #46926. The currently pinned FlashInfer 0.7.2 has no uniform_q_len keyword in BatchPrefillWithPagedKVCacheWrapper.plan(), so this PR must merge after #46926.

[github-actions]

Required checks failed on b06b088; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.