[eppaneamd]

Motivation

With --cuda-graph-backend-prefill breakable, the prefill runner refuses graph replay for every Qwen3.8-Flash-Next (Qwen4-Exp) batch that carries multimodal inputs, so image prefills always run eager.

Scope: this enables graph replay for image prefills in deployments without speculative decoding. With speculative decoding enabled (any algorithm, including NGRAM), behavior is unchanged.

• The refusal came with [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) #41729, which added breakable prefill graphs for this model with text-only scope.
• Multimodal input_embeds already replay under breakable graphs (Enable Kimi multimodal breakable prefill CUDA graph replay #31391): the eager outer forward builds the merged text and vision embeddings, and replay copies them into the captured static slot, as for text batches.
• With speculative decoding, the draft runner still refuses image batches, and the target must refuse with it: under DP attention the replay vote consults only the target runner and the draft follows it, so a draft-only refusal could split ranks between replay and eager.

This PR keys the refusal on whether speculative decoding is enabled, a launch-time property identical on every rank. Without speculative decoding, image batches become replay-eligible; with any speculative algorithm (including NGRAM), behavior is unchanged. No new flag, environment variable or API.

Modifications

• python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py (+8/-4): replace _qwen_bcg_mtp_draft (no other reader) with _qwen_bcg_multimodal_eager = qwen_bcg and model_runner.spec_algorithm.is_speculative(), and gate multimodal batches on it.
• test/registered/unit/model_executor/test_prefill_cuda_graph_runner.py: flag derivation for no speculative decoding, EAGLE and NGRAM.
• test/registered/unit/managers/scheduler_components/test_dp_attn.py: DP vote with one image rank and one text rank, refusal on and off.
• test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py: forward-time gate for image and text batches.
• test/registered/unit/model_executor/test_prefill_cuda_graph_runner_helpers.py: drop fixture lines for the removed flag.

The breakable prefill backend is the default on CUDA, so there this applies by default to Qwen3.8-Flash-Next without speculative decoding. On ROCm it is opt-in.

Accuracy Tests

Mean probability of the gold answer over 805 MMMU validation items, unpatched main vs this commit with breakable prefill graphs, 2 server launches per arm:

• explicit bucket list: +0.37 pp (95% CI -0.22 to +0.95);
• default bucket list: -0.10 pp (95% CI -0.66 to +0.47).

A greedy-output probe (16 requests) completed in every server launch; its outputs are not used as an accuracy comparison.

Speed Tests and Profiling

Setup

• Image: lmsysorg/sglang-rocm:v0.5.22-rocm724-mi35x-20261024, SGLang tree 13b7ef75c5 with this commit applied (prefill_cuda_graph_runner.py is unchanged between that tree and this PR's base), torch 2.11.0+rocm7.2.
• Hardware: 2x MI355X (gfx950).
• Model: amd/Qwen3.8-Flash-Next-Quark-MXFP4 (revision b2091c38), MTP off.
• Server (all arms): TP2 + EP2, --attention-backend aiter --linear-attn-backend triton --context-length 8192 --max-running-requests 128 --disable-radix-cache --mm-enable-dp-encoder.
• Baseline: unpatched main, no prefill CUDA graph (the ROCm default).
• Patched, explicit list: this commit plus

  --cuda-graph-backend-prefill breakable
  --cuda-graph-bs-prefill 256 512 768 1024 1280 1408 2560 2752 3840 4096 5376
  

  The bucket list was derived from this workload's prefill sizes.
• Patched, default list: this commit plus --cuda-graph-backend-prefill breakable.
• Benchmark: sglang.bench_serving, one 512x512 image per request, 1024 input / 128 output tokens (range ratio 0.8), prompts = 10x concurrency; 2 server launches per arm (order A, explicit, default, default, explicit, A), 2 measured runs per row after a warm-up pass.

Concurrency	Output tok/s base	Explicit list	Change	Default list	Change	TTFT (explicit / default)	TPOT (explicit / default)
4	250.0	273.1	+9.2%	273.6	+9.4%	-31.6% / -30.8%	-6.3% / -6.6%
16	579.5	694.7	+19.9%	693.0	+19.6%	-28.0% / -29.4%	-15.8% / -15.4%
32	822.2	1008.5	+22.7%	1010.6	+22.9%	-23.4% / -28.7%	-18.3% / -17.9%
64	1066.0	1280.3	+20.1%	1284.2	+20.5%	-18.3% / -25.3%	-16.8% / -16.4%
128	1382.9	1570.2	+13.6%	1590.8	+15.0%	-10.1% / -20.4%	-12.3% / -12.5%

Mean change over concurrency 16-64: +20.9% (explicit), +21.0% (default), SE 0.55 pp. SE: standard error of the change in percentage points, estimated from baseline-vs-baseline (A/A) runs with the same procedure; a change beyond about 2 SE is outside run-to-run noise.

Checklist

• Format your code: ruff check and ruff format --check clean on the five files.
• Add unit tests: the four suites above, 65 passed (50 subtests) on CPU (torch 2.14.1+cpu).
• Update documentation: not needed (no new flag or API).
• Provide accuracy and speed benchmark results: above, measured on this commit.
• Follow the SGLang code style guidance.

Notes

• Builds on [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) #41729 (text-only breakable prefill graphs for this model, multimodal batches refused), Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in allowlist #30620 (multimodal allowlist) and Enable Kimi multimodal breakable prefill CUDA graph replay #31391 (multimodal input_embeds replay).
• Adjacent open work, none of which edits this gate: [Fix] bcg: preserve Qwen3-VL DeepStack inputs during replay #33726, [Spec] Fix OOB in EAGLE3 draft-extend prefill BCG on multimodal models #38330, fix(qwen4): CPU-offload correctness, breakable graphs, mmap PLE table #37793, [Fix] dp-attention: Align prefill graph buckets and decode padding across DP ranks #46138, [Perf] [NPU] attention: 🚧 Split mixed chunked-prefill FIA into prefill/decode calls on Ascend 950 #43740 (trivial textual conflict). Tracking issue: [ROCm][AMD] Qwen3.8-Flash-Next gfx950 / MI355X Performance Optimization #44304.
• This change was developed with AI assistance (Claude); I reviewed every line and ran the tests listed.

---

CI States

Latest PR Test (Base): ❌ Run #41000722234
Latest PR Test (Extra): ❌ Run #41000722037
Latest PR Test (AMD ROCm 10): ✅ Run #41000722040

[rkarhila-amd]

I don't see any issues with this code.
Go ahead and open for code owner review.

[eppaneamd]

Required checks failed on ae416e9; this PR cannot be merged until they pass.

Lint (run) Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

Passing.