[eppaneamd]

Motivation

With --cuda-graph-backend-prefill breakable, the prefill runner refuses graph replay for every Qwen3.8-Flash-Next (Qwen4-Exp) batch that carries multimodal inputs, so image prefills always run eager.

Scope: this enables graph replay for image prefills in deployments without speculative decoding. With speculative decoding enabled (any algorithm, including NGRAM), behavior is unchanged.

• The refusal came with [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) sgl-project/sglang#41729, which added breakable prefill graphs for this model with text-only scope.
• Multimodal input_embeds already replay under breakable graphs (Enable Kimi multimodal breakable prefill CUDA graph replay sgl-project/sglang#31391): the eager outer forward builds the merged text and vision embeddings, and replay copies them into the captured static slot, as for text batches.
• With speculative decoding, the draft runner still refuses image batches, and the target must refuse with it: under DP attention the replay vote consults only the target runner and the draft follows it, so a draft-only refusal could split ranks between replay and eager.

This PR keys the refusal on whether speculative decoding is enabled, a launch-time property identical on every rank. Without speculative decoding, image batches become replay-eligible; with any speculative algorithm (including NGRAM), behavior is unchanged. No new flag, environment variable or API.

Modifications

• python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py (+8/-4): replace _qwen_bcg_mtp_draft (no other reader) with _qwen_bcg_multimodal_eager = qwen_bcg and model_runner.spec_algorithm.is_speculative(), and gate multimodal batches on it.
• test/registered/unit/model_executor/test_prefill_cuda_graph_runner.py: flag derivation over backend, architecture, worker role and speculative algorithm.
• test/registered/unit/managers/scheduler_components/test_dp_attn.py: DP vote with one image rank and one text rank, refusal on and off.
• test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py: forward-time gate for image and text batches.
• test/registered/unit/model_executor/test_prefill_cuda_graph_runner_helpers.py: drop fixture lines for the removed flag.

The breakable prefill backend is the default on CUDA, so there this applies by default to Qwen3.8-Flash-Next without speculative decoding. On ROCm it is opt-in.

Accuracy Tests

Mean probability of the gold answer over 805 MMMU validation items at concurrency 1, prefill graph off vs on: +0.20 pp (95% CI -0.35 to +0.76). Setup as below.

Speed Tests and Profiling

Setup

• Image: lmsysorg/sglang-rocm:v0.5.22-rocm724-mi35x-20261021 (SGLang source tree at a2e4ffbcea, AITER 285579b6, torch 2.11.0+rocm7.2).
• Hardware: 2x MI355X (gfx950).
• Model: amd/Qwen3.8-Flash-Next-Quark-MXFP4 (revision b2091c38), MTP off.
• Server (both arms): TP2 + EP2, --attention-backend aiter --linear-attn-backend triton --context-length 8192 --max-running-requests 128 --disable-radix-cache --mm-enable-dp-encoder.
• Baseline: no prefill CUDA graph (the ROCm default).
• Patched: this gate change plus

  --cuda-graph-backend-prefill breakable
  --cuda-graph-bs-prefill 256 512 768 1024 1280 1408 2560 2752 3840 4096 5376
  

  The bucket list was derived from this workload's prefill sizes; the default list has not been measured yet.
• Benchmark: sglang.bench_serving, one 512x512 image per request, 1024 input / 128 output tokens (range ratio 0.8), prompts = 10x concurrency; 2 server launches x 2 runs per row, mirrored order.
• Revision: measured with an equivalent gate (refusing multimodal batches only on the MTP draft), which admits the same batches as this commit when speculative decoding is off. A re-measure on this commit, including the default bucket list, is in progress.

Concurrency	Output tok/s base	Patched	Change	SE (pp)	TTFT	TPOT
4	246.9	268.0	+8.5%	0.97	-31.9%	-5.4%
16	564.6	689.8	+22.2%	0.77	-30.7%	-17.3%
32	804.9	1000.3	+24.3%	0.83	-24.1%	-19.3%
64	1028.1	1244.9	+21.1%	0.81	-18.3%	-17.5%
128	1342.9	1538.7	+14.6%	0.10	-10.2%	-13.1%

SE: standard error of the change in percentage points, estimated from baseline-vs-baseline (A/A) runs with the same procedure; a change beyond about 2 SE is outside run-to-run noise. One stalled baseline run at c16 is excluded.

Replay coverage: 3179 of 3408 prefill steps (93.3%) replayed; the rest exceeded the largest bucket or the 2x padding cap, none was refused for multimodal input.

Checklist

• Format your code: ruff check and ruff format --check clean on the five files.
• Add unit tests: the four suites above, 65 passed (53 subtests) on CPU (torch 2.14.1+cpu).
• Update documentation: not needed (no new flag or API).
• Provide accuracy and speed benchmark results: above; the re-measure on this commit is in progress.
• Follow the SGLang code style guidance.

Notes

• Builds on [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) sgl-project/sglang#41729 (text-only breakable prefill graphs for this model, multimodal batches refused), Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in allowlist sgl-project/sglang#30620 (multimodal allowlist) and Enable Kimi multimodal breakable prefill CUDA graph replay sgl-project/sglang#31391 (multimodal input_embeds replay). [Perf] CP: QSA Context Parallel Optimization sgl-project/sglang#43792 moved QSA prefill host lengths into attention-metadata construction between the measured revision and this base.
• Adjacent open work, none of which edits this gate: [Fix] bcg: preserve Qwen3-VL DeepStack inputs during replay sgl-project/sglang#33726, [Spec] Fix OOB in EAGLE3 draft-extend prefill BCG on multimodal models sgl-project/sglang#38330, fix(qwen4): CPU-offload correctness, breakable graphs, mmap PLE table sgl-project/sglang#37793, [Fix] dp-attention: Align prefill graph buckets and decode padding across DP ranks sgl-project/sglang#46138, [Perf] [NPU] attention: 🚧 Split mixed chunked-prefill FIA into prefill/decode calls on Ascend 950 sgl-project/sglang#43740 (trivial textual conflict). Tracking issue: [ROCm][AMD] Qwen3.8-Flash-Next gfx950 / MI355X Performance Optimization sgl-project/sglang#44304.
• This change was developed with AI assistance (Claude); I reviewed every line and ran the tests listed.

🤖 Generated with Claude Code

[eppaneamd]

Superseded by sgl-project#46329.