[diffusion] Enable breakable CUDA graph (BCG) for diffusion DiTs by BBuf · Pull Request #27436 · sgl-project/sglang

BBuf · 2026-06-06T10:18:29Z

Summary

Add Breakable CUDA Graph for selected diffusion DiT forward passes: graph-safe spans are captured, while dynamic attention stays eager.

This PR currently enables diffusion BCG only for:

Qwen/Qwen-Image
Qwen/Qwen-Image-2512
Tongyi-MAI/Z-Image
Tongyi-MAI/Z-Image-Turbo
zai-org/GLM-Image
Comfy-Org/Ideogram-4

Other diffusion models are intentionally left unchanged from main.

Why it work?

Qwen-Image @512x512

main:

pr:

Torch profiler

B200 profiler evidence for the supported model groups with captured traces. Eager and BCG are shown side by side for each model so the profiler differences are easier to compare. Profiler timings include profiler overhead and are separate from the benchmark table.

Qwen/Qwen-Image @ 512x512

Eager	BCG

Profiler commands

Eager profiler command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 SGLANG_TORCH_PROFILER_DIR=/tmp/pr27436_clean_benchmark/traces/qwen_image_eager_profile sglang generate --backend=sglang --model-path Qwen/Qwen-Image --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --profile --num-profiled-timesteps=5 --no-save-output'

BCG profiler command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 SGLANG_TORCH_PROFILER_DIR=/tmp/pr27436_clean_benchmark/traces/qwen_image_bcg_profile sglang generate --backend=sglang --model-path Qwen/Qwen-Image --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --enable-breakable-cuda-graph --profile --num-profiled-timesteps=5 --no-save-output'

Tongyi-MAI/Z-Image @ 256x256

Eager	BCG

Profiler commands

Eager profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_final_zimage_v4/zimage_base_256/eager_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 256 --height 256 --seed 42 --performance-mode speed --warmup --warmup-resolutions 256x256 --no-save-output --profile --num-profiled-timesteps 5

BCG profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_final_zimage_v4/zimage_base_256/bcg_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 256 --height 256 --seed 42 --performance-mode speed --warmup --warmup-resolutions 256x256 --no-save-output --enable-breakable-cuda-graph --profile --num-profiled-timesteps 5

Tongyi-MAI/Z-Image-Turbo @ 512x512

Eager	BCG

Profiler commands

Eager profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/zimage_turbo_512/eager_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image-Turbo --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 9 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --guidance-scale 0.0 --profile --num-profiled-timesteps 5

BCG profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/zimage_turbo_512/bcg_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image-Turbo --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 9 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --guidance-scale 0.0 --enable-breakable-cuda-graph --profile --num-profiled-timesteps 5

zai-org/GLM-Image @ 512x512

Eager	BCG

Profiler commands

Eager profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/glm_image_512/eager_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_glm HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path zai-org/GLM-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 30 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --profile --num-profiled-timesteps 5

BCG profiler command:

env SGLANG_TORCH_PROFILER_DIR=/mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/glm_image_512/bcg_trace PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_glm HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path zai-org/GLM-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 30 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --enable-breakable-cuda-graph --profile --num-profiled-timesteps 5

Comfy-Org/Ideogram-4 @ 512x512

Eager	BCG

Profiler commands

Eager profiler command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_ideogram_desc && CUDA_VISIBLE_DEVICES=6 PYTHONPATH=/tmp/sglang_pr27436_ideogram_desc/python FLASHINFER_DISABLE_VERSION_CHECK=1 SGLANG_TORCH_PROFILER_DIR=/tmp/pr27436_ideogram_desc/profiles/eager_trace sglang generate --backend=sglang --model-path Comfy-Org/Ideogram-4 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --profile --num-profiled-timesteps=5 --no-save-output'

BCG profiler command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_ideogram_desc && CUDA_VISIBLE_DEVICES=6 PYTHONPATH=/tmp/sglang_pr27436_ideogram_desc/python FLASHINFER_DISABLE_VERSION_CHECK=1 SGLANG_TORCH_PROFILER_DIR=/tmp/pr27436_ideogram_desc/profiles/bcg_trace sglang generate --backend=sglang --model-path Comfy-Org/Ideogram-4 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --enable-breakable-cuda-graph --profile --num-profiled-timesteps=5 --no-save-output'

Correctness: eager vs BCG serving req2

The BCG images below are generated by serving with --enable-breakable-cuda-graph: after server warmup, I sent req1 with the short prompt and then req2 with the benchmark long prompt. The eager images use the same serving flow, same req2 prompt, same resolution, and seed=42, with only BCG disabled. Logs were checked to ensure the native SGLang backend was used and no diffusers fallback occurred.
Req1 prompt: A small red cube on a white table

Req2 prompt: A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail

Qwen/Qwen-Image @ 512x512

Eager req2	BCG serving req2

Qwen/Qwen-Image-2512 @ 512x512

Eager req2	BCG serving req2

Tongyi-MAI/Z-Image @ 256x256

Eager req2	BCG serving req2

Tongyi-MAI/Z-Image-Turbo @ 512x512

Eager req2	BCG serving req2

zai-org/GLM-Image @ 512x512

GLM-Image has a sampled AR prior before the denoise span (do_sample=True). The older serving req2 pair below was generated before this PR fixed AR prior seeding, so the final images can differ because the two runs sampled different prior tokens. The exact AR prior/RNG from that older over-exposed BCG response was not saved, so the same-prior check below replays a newly saved BCG prior through eager. The BCG-covered denoise replay itself was checked separately with the same prior/input and matched eager exactly.

Eager req2	BCG serving req2

Same-prior replay check: BCG serving sampled and saved the GLM AR prior; eager then loaded that exact prior with BCG disabled. The two output images are an exact pixel match (diff mean=[0,0,0], rms=[0,0,0], all-channel extrema (0,0)).

Comfy-Org/Ideogram-4 @ 512x512

Eager and BCG req2 are an exact pixel match (diff mean=[0,0,0], rms=[0,0,0], all-channel extrema (0,0), diff_bbox=None).

Eager req2	BCG serving req2

Supported Models

B200 benchmarks with the native SGLang backend. Both BCG and eager runs use explicit warmup (--warmup --warmup-resolutions ...). BCG uses the PR default text buckets 64 128 256 512 1024 (--bcg-text-buckets omitted); eager removes only --enable-breakable-cuda-graph. The latency columns report non-profiled runs after warmup. Commands are folded below the table to keep the support matrix readable.

Model / shape	Capture after warmup -> req1 -> req2	Eager denoise	BCG denoise	Denoise speedup	Eager total	BCG total	Peak mem (Eager -> BCG)
`Qwen/Qwen-Image @ 512x512`	`5 -> 5 -> 5`	6.48s	2.45s	2.64x	6.48s	2.45s	55.038GB -> 58.098GB
`Qwen/Qwen-Image-2512 @ 512x512`	`5 -> 5 -> 5`	6.21s	2.44s	2.55x	6.21s	2.44s	55.038GB -> 58.098GB
`Tongyi-MAI/Z-Image @ 256x256`	`5 -> 5 -> 5`	1.205s	0.634s	1.90x	1.231s	0.662s	19.611GB -> 20.635GB
`Tongyi-MAI/Z-Image-Turbo @ 512x512`	`5 -> 5 -> 5`	0.113s	0.026s	4.26x	0.138s	0.097s	20.400GB -> 22.682GB
`zai-org/GLM-Image @ 512x512`	`1 -> 1 -> 1`	1.100s	0.878s	1.25x	8.056s	8.082s	35.855GB -> 36.324GB
`Comfy-Org/Ideogram-4 @ 512x512`	`6 -> 6 -> 6`	1.564s	0.916s	1.71x	1.625s	0.991s	23.068GB -> 29.418GB

GLM-Image total latency is dominated by the sampled AR prior/text path; the measured BCG benefit is on the capture-stable denoise span. This PR also seeds the GLM AR prior from the request seed so eager/BCG correctness comparisons are reproducible.

Benchmark commands

Qwen/Qwen-Image @ 512x512

Eager benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Qwen/Qwen-Image --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --no-save-output'

BCG benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Qwen/Qwen-Image --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --enable-breakable-cuda-graph --no-save-output'

Qwen/Qwen-Image-2512 @ 512x512

Eager benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image_2512 HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image_2512/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Qwen/Qwen-Image-2512 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --no-save-output'

BCG benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_clean_b200 && CUDA_VISIBLE_DEVICES=4 HF_HOME=/tmp/pr27436_clean_hf_qwen_image_2512 HF_HUB_CACHE=/tmp/pr27436_clean_hf_qwen_image_2512/hub PYTHONPATH=/tmp/sglang_pr27436_clean_b200/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Qwen/Qwen-Image-2512 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --enable-breakable-cuda-graph --no-save-output'

Tongyi-MAI/Z-Image @ 256x256

Eager benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 256 --height 256 --seed 42 --performance-mode speed --warmup --warmup-resolutions 256x256 --no-save-output --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_final_zimage_v4/zimage_base_256/eager_perf.json

BCG benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_final_zimage_base/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 256 --height 256 --seed 42 --performance-mode speed --warmup --warmup-resolutions 256x256 --no-save-output --enable-breakable-cuda-graph --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_final_zimage_v4/zimage_base_256/bcg_perf.json

Tongyi-MAI/Z-Image-Turbo @ 512x512

Eager benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image-Turbo --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 9 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --guidance-scale 0.0 --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/zimage_turbo_512/eager_perf.json

BCG benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_zimage_turbo/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path Tongyi-MAI/Z-Image-Turbo --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 9 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --guidance-scale 0.0 --enable-breakable-cuda-graph --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/zimage_turbo_512/bcg_perf.json

zai-org/GLM-Image @ 512x512

Eager benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_glm HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path zai-org/GLM-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 30 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/glm_image_512/eager_perf.json

BCG benchmark command:

env PYTHONPATH=/mnt/local_disk/bbuf/sglang_pr27436_image_only/python FLASHINFER_DISABLE_VERSION_CHECK=1 HF_HOME=/mnt/local_disk/bbuf/pr27436_hf_bench_glm HF_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub HUGGINGFACE_HUB_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/hub TRANSFORMERS_CACHE=/mnt/local_disk/bbuf/pr27436_hf_bench_glm/transformers CUDA_VISIBLE_DEVICES=0 sglang generate --backend sglang --model-path zai-org/GLM-Image --prompt A\ small\ red\ cube\ on\ a\ white\ table A\ detailed\ cinematic\ scene\ of\ a\ glass\ observatory\ above\ a\ quiet\ lake\ at\ sunrise\,\ with\ soft\ mist\,\ warm\ reflections\,\ and\ crisp\ architectural\ detail --width 512 --height 512 --seed 42 --num-inference-steps 30 --performance-mode speed --warmup --warmup-resolutions 512x512 --no-save-output --enable-breakable-cuda-graph --perf-dump-path /mnt/local_disk/bbuf/pr27436_bcg_bench_profile_v1/glm_image_512/bcg_perf.json

Comfy-Org/Ideogram-4 @ 512x512

Eager benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_ideogram_desc && CUDA_VISIBLE_DEVICES=6 PYTHONPATH=/tmp/sglang_pr27436_ideogram_desc/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Comfy-Org/Ideogram-4 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --no-save-output --perf-dump-path /tmp/pr27436_ideogram_desc/bench/eager_perf.json'

BCG benchmark command:

sudo docker exec sglang_bbuf bash -lc 'cd /tmp/sglang_pr27436_ideogram_desc && CUDA_VISIBLE_DEVICES=6 PYTHONPATH=/tmp/sglang_pr27436_ideogram_desc/python FLASHINFER_DISABLE_VERSION_CHECK=1 sglang generate --backend=sglang --model-path Comfy-Org/Ideogram-4 --prompt "A small red cube on a white table" "A detailed cinematic scene of a glass observatory above a quiet lake at sunrise, with soft mist, warm reflections, and crisp architectural detail" --width=512 --height=512 --seed=42 --performance-mode=speed --warmup --warmup-resolutions 512x512 --no-save-output --enable-breakable-cuda-graph --perf-dump-path /tmp/pr27436_ideogram_desc/bench/bcg_perf.json'

Notes

Ideogram-4 warmup captures five positive text-bucket signatures plus the unconditional 1024-token signature (6 signatures, 210 total graph segments); req1/req2 reused the warmed captures without new captures.
Serving capture validation sends two real prompts with different prompt lengths after warmup; the capture count remains unchanged for every supported model above.
Native backend gate passed for the extra models: no Falling back to diffusers backend, Using diffusers backend, Loaded diffusers pipeline, capture failed, Traceback, RuntimeError, or CUDA error appeared in the benchmark/profile/serving logs.
The temporary HF caches used by the extra model runs were deleted after each completed model group.
GLM-Image AR prior is now seeded from the request seed; the same-prior debug replay matched BCG and eager outputs pixel-for-pixel.
BCG requires explicit --warmup-resolutions because diffusion graphs are resolution-specific.
BCG is disabled with --enable-torch-compile and Cache-DiT.

CI States

Latest PR Test (Base): ⏳ Run #28916751820
Latest PR Test (Extra): ⏳ Run #28916751679

gemini-code-assist

Code Review

This pull request introduces a model-agnostic Breakable CUDA Graph (BCG) runner for diffusion DiT transformers, moving BCG primitives to a shared package (sglang.srt.breakable_cuda_graph) to be utilized by both LLM and diffusion runtimes. The feedback highlights three important improvements: addressing a critical stream synchronization race condition during graph replay to prevent reading or cloning stale data, adding support for dictionary and ModelOutput types in _clone_output to avoid overwriting static output buffers, and extending _weak_ref_if_tensor to support dictionaries so that intermediate tensors can be properly reclaimed by the shared mempool.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

mickqian · 2026-06-15T07:25:42Z

perhaps try ideogram-4?

Oasis-Git

Some high level comments:

the diffusion runner should inherent from base runner and implement capture replay api. The capture should happen during the class init. As for the resolution, maybe we can set a server args to let the user enter it when engine is up instead of runtime capture.
instead of directly injection on the transformer.forward(), we can implement an eager runner which only does replay() -> transformer.forward() with other functions directly pass
maybe we can put the model specific helper functions into their own specific stages file and gather other helper functions in a utils.py

Oasis-Git

The code structure is clean now. However I think the introduction of real ref of captured output will increase the memory usage thus it may be unacceptable.

…uda-graph # Conflicts: # test/registered/unit/test_legacy_global_ratchet.py

BBuf · 2026-07-08T03:35:04Z

https://github.com/sgl-project/sglang/actions/runs/28744060199/job/85581949068?pr=27436

The srt breakable CUDA graph core returns byte-identical to its state before sgl-project#27436 (back at srt/model_executor/runner_backend_utils/breakable_cuda_graph, dedup support and capture-failure logging restored, top-level srt/breakable_cuda_graph package removed). The diffusion BCG feature is kept and adapted to the pre-PR core: - replay-token machinery moves to multimodal_gen/runtime/breakable_cuda_graph/replay_token.py; the mm runner stamps a token around each graph.replay() - tuple-returning attention forwards (UlyssesAttention, used by Z-Image) are boxed into an attribute object for the break point, since the pre-PR _copy_output copies objects-with-tensor-attrs in place but ignores tuples - layer.py imports repoint accordingly Verified: core files byte-identical to ead1e49; all consumers import; GPU capture/replay of a boxed tuple break point produces fresh outputs on every replay; 16/16 BCG unit tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…-project#27436) Co-authored-by: BBuf <xiaoyu.zhang@radixark.net> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: BBuf <bbuf@sglang.local>

github-actions Bot added the diffusion SGLang Diffusion label Jun 6, 2026

gemini-code-assist Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread python/sglang/multimodal_gen/runtime/breakable_cuda_graph_runner.py Outdated

Comment thread python/sglang/multimodal_gen/runtime/breakable_cuda_graph_runner.py Outdated

Comment thread .../sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py

This was referenced Jun 7, 2026

[CI Monitor] Daily Report - 2026-06-07 bingxche/sglang-ci-bot#96

Open

[CI Monitor] Daily Report - 2026-06-10 bingxche/sglang-ci-bot#99

Open

[CI Monitor] Daily Report - 2026-06-12 bingxche/sglang-ci-bot#102

Open

BBuf marked this pull request as ready for review June 14, 2026 09:27

BBuf requested review from Fridge003, HaiShaw, Ying1123, hnyls2002, ispobock, merrymercy, mickqian, ping1jing2, yhyang201, yichiche and yingluosanqian as code owners June 14, 2026 09:27

BBuf added run-ci run-ci-extra labels Jun 15, 2026

BBuf force-pushed the diffusion/breakable-cuda-graph branch from b9e2bef to 51b03ff Compare June 15, 2026 01:45

BBuf force-pushed the diffusion/breakable-cuda-graph branch from 6b0e87c to 59ace4f Compare June 15, 2026 12:42

Oasis-Git reviewed Jun 16, 2026

View reviewed changes

Oasis-Git self-assigned this Jun 16, 2026

amd-bot mentioned this pull request Jun 17, 2026

[CI Monitor] Daily Report - 2026-06-17 bingxche/sglang-ci-bot#107

Open

BBuf requested a review from AgainstEntropy as a code owner June 17, 2026 07:24

mickqian reviewed Jun 19, 2026

View reviewed changes

Oasis-Git reviewed Jun 21, 2026

View reviewed changes

Comment thread .../sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py Outdated

BBuf force-pushed the diffusion/breakable-cuda-graph branch from ff06e95 to 25d2266 Compare June 27, 2026 13:49

AgainstEntropy mentioned this pull request Jul 1, 2026

feat(diffusion): add OmniDreams autoregressive video world model #27442

Open

BBuf added 2 commits July 5, 2026 17:47

Relax NPU W4A4 throughput guard again

4ff2c07

Stabilize AMD diffusion update-weights CI

dacb13c

BBuf requested review from Kangyan-Zhou and bingxche as code owners July 5, 2026 10:52

github-actions Bot added the amd label Jul 5, 2026

BBuf added 2 commits July 5, 2026 22:19

Merge remote-tracking branch 'origin/main' into diffusion/breakable-c…

19d8735

…uda-graph # Conflicts: # test/registered/unit/test_legacy_global_ratchet.py

Revert unrelated CI and non-diffusion test tweaks

bad8cd8

Oasis-Git added the bypass-fastfail label Jul 6, 2026

amd-bot mentioned this pull request Jul 6, 2026

[CI Monitor] Daily Report - 2026-07-06 bingxche/sglang-ci-bot#127

Open

Oasis-Git approved these changes Jul 8, 2026

View reviewed changes

BBuf added 2 commits July 8, 2026 11:44

Merge origin/main into diffusion/breakable-cuda-graph

b2a57ab

Use Z-Image caption mask for BCG padding

28c1c28

BBuf merged commit 33c3dfd into sgl-project:main Jul 8, 2026
127 of 182 checks passed

amd-bot mentioned this pull request Jul 9, 2026

[CI Monitor] Daily Report - 2026-07-09 bingxche/sglang-ci-bot#130

Open

BBuf mentioned this pull request Jul 9, 2026

Fix diffusion BCG lifetime and add Z-Image-Turbo CI #30584

Merged

Oasis-Git mentioned this pull request Jul 9, 2026

Move breakable CUDA graph back into model_executor/runner_backend_utils #30586

Merged

5 tasks

michaelzhang-ai mentioned this pull request Jul 10, 2026

[AMD] Cherry-pick AMD fixes into release/v0.5.15 #30722

Closed

3 tasks

This was referenced Jul 10, 2026

[CI Monitor] Daily Report - 2026-07-10 bingxche/sglang-ci-bot#131

Open

[CI Monitor] Daily Report - 2026-07-12 bingxche/sglang-ci-bot#133

Open

This was referenced Jul 19, 2026

[CI Monitor] Daily Report - 2026-07-19 bingxche/sglang-ci-bot#140

Open

[CI Monitor] Daily Report - 2026-07-25 bingxche/sglang-ci-bot#146

Open

BBuf mentioned this pull request Aug 7, 2026

[diffusion] Enable breakable CUDA graph for SANA (H200 1024px e2e -26%, bit-exact) #33989

Merged

hnyls2002 mentioned this pull request Aug 10, 2026

[Diffusion] add cuda graph support for Qwen-Image #19516

Closed

5 tasks

Navigation Menu

[diffusion] Enable breakable CUDA graph (BCG) for diffusion DiTs - #27436

[diffusion] Enable breakable CUDA graph (BCG) for diffusion DiTs#27436
BBuf merged 86 commits into
sgl-project:mainfrom
BBuf:diffusion/breakable-cuda-graph

BBuf commented Jun 6, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mickqian commented Jun 15, 2026

Uh oh!

Oasis-Git left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oasis-Git left a comment

Uh oh!

Uh oh!

BBuf commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BBuf commented Jun 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why it work?

Qwen-Image @512x512

Torch profiler

Qwen/Qwen-Image @ 512x512

Tongyi-MAI/Z-Image @ 256x256

Tongyi-MAI/Z-Image-Turbo @ 512x512

zai-org/GLM-Image @ 512x512

Comfy-Org/Ideogram-4 @ 512x512

Correctness: eager vs BCG serving req2

Qwen/Qwen-Image @ 512x512

Qwen/Qwen-Image-2512 @ 512x512

Tongyi-MAI/Z-Image @ 256x256

Tongyi-MAI/Z-Image-Turbo @ 512x512

zai-org/GLM-Image @ 512x512

Comfy-Org/Ideogram-4 @ 512x512

Supported Models

Benchmark commands

Notes

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mickqian commented Jun 15, 2026

Uh oh!

Oasis-Git left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oasis-Git left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BBuf commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BBuf commented Jun 6, 2026 •

edited by github-actions Bot

Loading

Oasis-Git left a comment •

edited

Loading