Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in allowlist by YAMY1234 · Pull Request #30620 · sgl-project/sglang

YAMY1234 · 2026-07-09T08:28:04Z

Motivation

Since #29458, breakable CUDA graph (BCG) is the default prefill CUDA graph backend on CUDA, but the generic "multimodal model" rule in _disable_breakable_cudagraph_if_incompatible turns it off for every arch in multimodal_model_archs. Qwen3.5 (Qwen3_5[Moe]ForConditionalGeneration) is caught by this rule even for text-only serving, although its GDN/GQA prefill path fully supports BCG capture/replay.

The failure mode the rule protects against ("multimodal prefill replay faults") is real: the captured graph embeds from input_ids only, while multimodal batches merge mm embeddings in the outer model wrapper, which capture bypasses — replaying them silently drops image inputs. The existing can_run_graph guard only covered input_embeds / replace_embeds; this PR extends it to batches carrying mm_inputs (caught by test_vlm_tp4.py MMMU dropping to 0.50 on the first CI run of this PR; multimodal batches now fall back to eager prefill, which is today's behavior).

Modifications

Mirror the existing tc_piecewise opt-in pattern (multimodal_piecewise_cuda_graph_supported_model_archs) with a BCG-specific allowlist:

model_config.py: add multimodal_breakable_cuda_graph_supported_model_archs (Qwen3.5 dense + MoE) and the corresponding ModelConfig field.
server_args.py: the BCG "multimodal model" rule becomes is_multimodal and not is_multimodal_breakable_cuda_graph_supported.
prefill_cuda_graph_runner.py: can_run_graph rejects batches carrying mm_inputs, so multimodal batches run eager prefill while text-only batches replay the graph. mm_inputs persists on the request until it finishes, so every prefill chunk of a multimodal request is covered.

No behavior change for any arch not on the allowlist.

Accuracy Tests

Speed Tests and Profiling

Qwen3.5-397B-A17B-FP8, single node TP4 (GB200), 1k1k random dataset, closed-loop client (range-ratio 0.8, num-prompts 10×conc, request-rate inf). Server args follow the InferenceX/InferenceMAX qwen3.5 fp8 configs (trtllm_mha attention, flashinfer_trtllm MoE, fp8_e4m3 KV cache, --enable-symm-mem, --disable-radix-cache). "BCG off" = same build with --disable-prefill-cuda-graph, which is the effective status quo for Qwen3.5 today.

Config	Concurrency	BCG on: total tok/s / p50 TTFT	BCG off: total tok/s / p50 TTFT	Δ throughput	Δ TTFT
TP4	4	1241.3 / 208.6 ms	1139.6 / 361.0 ms	+8.9%	−42.2%
TP4 + EAGLE MTP	16	4955.4 / 464.6 ms	4382.8 / 540.6 ms	+13.1%	−14.1%
TP4	64	7369.1 / 466.8 ms	6758.5 / 532.9 ms	+9.0%	−12.4%

EAGLE accept length is unchanged between the two arms (2.956 vs 2.931). One-time capture cost: ~41 s at startup, 1.83 GB per GPU (chunked-prefill 16384 bucket set).

Note: the numbers above were measured with #27918 (restores the fused Triton MRoPE path under BCG) applied on top; that fix is complementary and tracked separately.

Unit test

test/registered/unit/server_args/test_server_args.py::TestBreakableCudaGraphMultimodalAllowlist covers the new rule (allowlisted arch keeps breakable, non-allowlisted multimodal arch resolves to disabled) and allowlist membership. Verified together with the neighboring TestCudaGraphDisaggregationRoles: 7 passed.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ✅ Run #29230286789
Latest PR Test (Extra): ❌ Run #29230286551

gemini-code-assist

Code Review

This pull request introduces support for breakable CUDA graphs in specific multimodal model architectures, specifically Qwen 3.5 and Qwen 3.5 MoE. It adds a configuration property and helper function to identify supported architectures and updates the server arguments to allow breakable CUDA graphs for these allowlisted multimodal models. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Oasis-Git

Could you add a CI test?

…allowlist The generic "multimodal model" rule in _disable_breakable_cudagraph_if_incompatible disables prefill BCG for every arch in multimodal_model_archs. Qwen3.5 (Qwen3_5[Moe]ForConditionalGeneration) is caught by it even for text-only serving, although its GDN/GQA prefill path fully supports BCG capture/replay. Mirror the existing tc_piecewise opt-in pattern (multimodal_piecewise_cuda_graph_supported_model_archs) with a BCG-specific allowlist. The replay-time guard is unchanged: batches carrying input_embeds/replace_embeds are rejected by can_run_graph and fall back to eager, so vision requests never replay the text-only graph. Add unit tests covering the rule (allowlisted arch keeps breakable, non-allowlisted multimodal arch is disabled) and allowlist membership.

YAMY1234 · 2026-07-09T22:29:31Z

Could you add a CI test?

Sure — added a unit test for the gate logic: TestBreakableCudaGraphMultimodalAllowlist

sglang/test/registered/unit/server_args/test_server_args.py

Line 1327 in 3d3cf25

class TestBreakableCudaGraphMultimodalAllowlist(CustomTestCase):

(allowlisted archs keep breakable; other multimodal archs still resolve to disabled).

E2E is already covered by existing CI — these run per-commit on B200 (base-c) with a gsm8k >= 0.95 gate, and neither sets any prefill CUDA-graph flag, so with this PR they run under BCG by default:

sglang/test/registered/models_e2e/test_qwen35_fp4_flashinfer.py

Line 13 in 61602b9

register_cuda_ci(est_time=720, stage="base-c", runner_config="4-gpu-b200")

sglang/test/registered/models_e2e/test_qwen35_fp4_mtp.py

Line 17 in 61602b9

register_cuda_ci(est_time=740, stage="base-c", runner_config="4-gpu-b200")

The nightly FP8 suite picks it up as well:

sglang/test/registered/8-gpu-models/test_qwen35.py

Line 10 in 61602b9

register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True)

YAMY1234 · 2026-07-09T22:29:41Z

/tag-and-rerun-ci

The captured prefill graph embeds from input_ids only; multimodal batches merge mm embeddings in the outer model wrapper, which graph capture bypasses, so replaying them silently drops image inputs (caught by test_vlm_tp4.py MMMU dropping to 0.50 vs the 0.65 threshold once the allowlist enabled BCG for Qwen3.5). can_run_graph already rejects input_embeds/replace_embeds batches; add the same rejection for batches carrying mm_inputs so they fall back to eager prefill, while text-only batches keep the captured graph. mm_inputs persists on the request until it finishes, so every prefill chunk of a multimodal request is covered.

YAMY1234 · 2026-07-10T22:12:22Z

/rerun-failed-ci

YAMY1234 · 2026-07-13T06:38:20Z

/rerun-test test/registered/vlm/test_vlm_tp4.py test/registered/models_e2e/test_qwen35_fp4_flashinfer.py test/registered/models_e2e/test_qwen35_fp4_mtp.py test/registered/unit/server_args/test_server_args.py

github-actions · 2026-07-13T06:38:44Z

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit cbcbef6.

Re-run was not dispatched. What to do:

Rebase your branch onto the latest main and push again
Follow issue #21065 for context
CI-fix PRs may request the bypass-maintenance label to skip this check

YAMY1234 · 2026-07-13T07:07:07Z

/rerun-test test/registered/vlm/test_vlm_tp4.py test/registered/models_e2e/test_qwen35_fp4_flashinfer.py test/registered/models_e2e/test_qwen35_fp4_mtp.py test/registered/unit/server_args/test_server_args.py

github-actions · 2026-07-13T07:07:30Z

Results for /rerun-test test/registered/vlm/test_vlm_tp4.py test/registered/models_e2e/test_qwen35_fp4_flashinfer.py test/registered/models_e2e/test_qwen35_fp4_mtp.py test/registered/unit/server_args/test_server_args.py:

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/vlm/test_vlm_tp4.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_qwen35_fp4_flashinfer.py
cd test/ && python3 registered/models_e2e/test_qwen35_fp4_mtp.py

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/server_args/test_server_args.py

…allowlist (sgl-project#30620)

gemini-code-assist Bot reviewed Jul 9, 2026

View reviewed changes

YAMY1234 marked this pull request as ready for review July 9, 2026 08:30

Oasis-Git approved these changes Jul 9, 2026

View reviewed changes

YAMY1234 force-pushed the bcg-multimodal-allowlist-qwen35 branch from 46d66a5 to 3d3cf25 Compare July 9, 2026 21:27

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Jul 9, 2026

YAMY1234 requested review from Fridge003, Ying1123, hnyls2002, ispobock and merrymercy as code owners July 10, 2026 07:09

amd-bot mentioned this pull request Jul 11, 2026

[CI Monitor] Daily Report - 2026-07-11 bingxche/sglang-ci-bot#132

Open

Merge branch 'main' into bcg-multimodal-allowlist-qwen35

b1eeaad

YAMY1234 added the bypass-fastfail label Jul 12, 2026

Merge branch 'main' into bcg-multimodal-allowlist-qwen35

e2abb04

Fridge003 approved these changes Jul 14, 2026

View reviewed changes

Fridge003 merged commit 41ad0d9 into sgl-project:main Jul 14, 2026
148 of 167 checks passed

amd-bot mentioned this pull request Jul 15, 2026

[CI Monitor] Daily Report - 2026-07-15 bingxche/sglang-ci-bot#136

Open

b8zhong mentioned this pull request Jul 17, 2026

Remove stale non-Standard-GQA gate for Nemotron BCG #31099

Open

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in …

109be43

…allowlist (sgl-project#30620)

YAMY1234 deleted the bcg-multimodal-allowlist-qwen35 branch August 25, 2026 16:15

This was referenced Oct 24, 2026

[Perf] Qwen3.8: Allow breakable prefill CUDA graphs for image batches eppaneamd/sglang#2

Closed

[Perf] Qwen3.8: Allow breakable prefill CUDA graphs for image batches #46329

Open

Navigation Menu

Repository navigation

Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in allowlist - #30620

Allow prefill breakable CUDA graph for Qwen3.5 via multimodal opt-in allowlist#30620
Fridge003 merged 4 commits into
sgl-project:mainfrom
YAMY1234:bcg-multimodal-allowlist-qwen35

YAMY1234 commented Jul 9, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Oasis-Git left a comment

Uh oh!

YAMY1234 commented Jul 9, 2026

Uh oh!

YAMY1234 commented Jul 9, 2026

Uh oh!

YAMY1234 commented Jul 10, 2026

Uh oh!

YAMY1234 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

YAMY1234 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

YAMY1234 commented Jul 9, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Unit test

Checklist

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Oasis-Git left a comment

Choose a reason for hiding this comment

Uh oh!

YAMY1234 commented Jul 9, 2026

Uh oh!

YAMY1234 commented Jul 9, 2026

Uh oh!

YAMY1234 commented Jul 10, 2026

Uh oh!

YAMY1234 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

⚠️ Rebase Required Before Re-run

Uh oh!

YAMY1234 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YAMY1234 commented Jul 9, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading