[Feat] comfyui: Add SGLDCacheOptions for per-run Cache-DiT by OnePunchMonk · Pull Request #43696 · sgl-project/sglang

OnePunchMonk · 2026-10-11T14:26:38Z

Adds an SGLDCacheOptions node that turns Cache-DiT on or off for a single run from the graph. It sets the options on a cloned MODEL, so toggling it never restarts the worker. The executor reads them from transformer_options and maps them to the per-request enable_cache_dit and cache_dit_params. The loader option stays as the fallback, and a per-run on/off wins over it.

Only MiniMax-H3 supports it. Other models send one independent step per request, so there is no step history to reuse, and the node rejects them with an error. TeaCache is not exposed because its state resets on timestep 0.

This is meant to line up with slimming SGLDOptions down to load-time options. If #43504 lands first I will rebase onto its model_options path.

Tested on CPU only (test_comfyui_adapters.py, test_comfyui_generator.py, 44 passed). GPU unit run is in the comments.

CI States

Latest PR Test (Base): ❌ Run #41124542308
Latest PR Test (Extra): ❌ Run #41124542543
Latest PR Test (AMD ROCm 10): ❌ Run #41124542339

Adds a node that sets Cache-DiT per run on a cloned MODEL, so toggling it never restarts the worker. The executor reads the options from transformer_options and maps them to the per-request enable_cache_dit and cache_dit_params fields. The loader option enable_cache_dit stays as the fallback, and a per-run on/off wins over it. Only MiniMax-H3 runs a multi-step session on the worker. Other models send one independent step per request, so Cache-DiT has no history to reuse; the node and the executor reject it for them instead of silently doing nothing. TeaCache is not exposed: its state resets on timestep 0, and every ComfyUI request is timestep 0.

OnePunchMonk · 2026-10-11T15:53:10Z

GPU run on a Modal L4 (CUDA 13, torch 2.13 with the released sglang 0.5.21 deps, this branch's code on top, ComfyUI master checked out). Ran all test_comfyui_*.py plus test_ipc_cuda.py: 123 passed, 1 skipped, 3 xfailed. Plain main on the same box is 117 passed.

This does not exercise Cache-DiT on a real H3. The full H3 checkpoint is about 498 GB, so I did not run it. Treat the per-run on/off behavior as unit-tested only, and it would be good to have someone with H3 weights try it. The only skip is a test that needs a tiny H3 checkpoint.

OnePunchMonk · 2026-10-11T15:55:06Z

Full details of the GPU run, so it can be reproduced.

Environment

Modal, 1x NVIDIA L4 (sm_89)
Image: nvidia/cuda:13.0.1-devel-ubuntu22.04, Python 3.11
pip install 'sglang[diffusion]' (0.5.21) for the dependencies, which brought in torch 2.13. This branch is cloned and put first on PYTHONPATH, so the PR code is what runs.
ComfyUI master cloned to /comfy and added to PYTHONPATH, so the tests that need comfy run instead of skipping.

Command (from python/)

PYTHONPATH=/comfy:. python -m pytest \
  sglang/multimodal_gen/test/unit/test_comfyui_*.py \
  sglang/multimodal_gen/test/unit/test_ipc_cuda.py -q -rfs

Baseline: plain sgl-project/sglang main on the same image gives 117 passed, 1 skipped, 3 xfailed.

Result on this branch: 123 passed, 1 skipped, 3 xfailed. The one skip needs a tiny H3 checkpoint.

What this does and does not show

The rebased branch passes everything main passes, plus the Cache-DiT option tests, with ComfyUI installed and a CUDA device present.
No failures were found on the GPU box for this branch.
Cache-DiT itself was not run on a real H3. The full H3 checkpoint is about 498 GB, so I did not download it. The per-run on/off path is covered by unit tests with a fake executor, not by a real denoising run.
Other models still reject the node by design, since they send one independent step per request.

niehen6174 · 2026-10-12T12:00:27Z

Thanks! Can this be generic rather than H3-only? Also, please add an e2e run and share speed and quality numbers with it on vs off.

ComfyUI sends one request per DiT call, so the worker keeps one Cache-DiT context per sampler run, sized from the run's sample_sigmas.

OnePunchMonk · 2026-10-12T15:48:46Z

E2E numbers. H100, ComfyUI v0.39.2, Flux.1-dev, 1024x1024, 25 steps, euler/simple, cfg 1, same seed, default DBCache settings (Fn=1 Bn=0 W=4 R=0.24 MC=3). The worker log confirms one context for the whole run: cache-dit enabled on transformer (steps=25, ...).

	prompt time	PSNR	SSIM
native ComfyUI	4.62s
SGLD cache off	3.64s	25.37 dB vs native	0.903 vs native
SGLD cache on	2.01s	18.08 dB vs cache off	0.616 vs cache off

That is 1.81x vs cache off. With these defaults the output drifts noticeably (layout shifts a bit, still a clean image), so the default threshold is on the aggressive side for 25 step Flux. The node exposes the DBCache knobs for a more conservative setting.

cache-dit 1.5.1 marks the FakeDiffusionPipeline class as cached and never clears it on disable_cache(transformer), so a knob change between requests asserted on remount.

OnePunchMonk · 2026-10-12T16:29:54Z

Follow-up with gentler DBCache settings, same H100 setup (Flux.1-dev, 1024x1024, 25 steps, same seed), compared against cache off:

W8 R0.08: 1.37x, 22.5 dB PSNR / 0.81 SSIM
W8 R0.12: 1.45x, 20.7 dB / 0.74
W4 R0.12: 1.52x, 19.2 dB / 0.69
Default W4 R0.24 (from above): 1.81x, 18.1 dB / 0.62

Every setting keeps a clean image, with the layout shifting more as the threshold goes up. To make sure the drift is Cache-DiT itself and not the per-run wiring, I checked that W25 (no caching possible) is bit-identical to cache off, and that a fresh mount and a later run with the same seed are bit-identical too.

The sweep also hit a crash when the knobs change between runs: cache-dit 1.5.1 leaves a class-level flag set after disable_cache on a bare transformer, so the remount asserted. Fixed in 7585fd5 with a regression test against the real cache-dit. This affects server requests that change cache_dit_params as well, not just ComfyUI.

ComfyUI calls the model once per CFG branch, so each request carries one row and no CFG of its own. Instead of rejecting uncond calls, the executor reads the guider in the sampler wrapper (cfg != 1 with a negative cond) and sends comfyui_separate_cfg; the worker then mounts Cache-DiT with separate CFG so alternating cond / uncond calls count as one step. Registered adapters (Z-Image, Qwen-Image, FLUX) ignore has_separate_cfg and default to their diffusers pipeline, which batches CFG, so pass enable_separate_cfg explicitly for ComfyUI runs only. Z-Image Turbo, 30 steps, H200, warm: cfg 4 6.2-6.8 s -> 4.1 s; cfg 1 4.0 s -> 2.0 s; switching cfg and cache on one worker remounts cleanly.

niehen6174 · 2026-10-13T13:38:59Z

Hi @OnePunchMonk, I pushed one commit on top of your branch (814256e) so CFG > 1 runs are cached instead of rejected:

Detect CFG from the guider: the sampler wrapper checks cfg != 1 and a negative cond, and the executor sends comfyui_separate_cfg (per-call cond_or_uncond is only [0] or [1]).
Mount with separate CFG: the worker treats the alternating cond / uncond requests as one step. Registered adapters (Z-Image / Qwen-Image / FLUX) ignore has_separate_cfg, so enable_separate_cfg=True is passed explicitly, for ComfyUI runs only.
Results (Z-Image Turbo, 30 steps, H200, warm): cfg 4: 6.5 s -> 4.1 s; cfg 1: 4.0 s -> 2.0 s. Switching cfg and cache on one worker remounts cleanly.
Scope: I dropped Cache-DiT from [diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight #44058 so this PR owns it; enable_cache_dit there stays the loader default you fall back to.

Feel free to rework or drop the commit.

Resolve test_comfyui_generator.py: keep both appended tests; the reuse test's last_options gains runtime_model_path, which main's gather_options now holds.

niehen6174 · 2026-10-16T02:09:31Z

Could you merge the latest main into this branch, resolve the conflicts, and update the docs and default parameter guidance? Please clarify the quality trade-off and consider a more conservative preset. I also reproduced an issue where switching to off/default after tuning the cache knobs raises a ValueError; users should be able to disable caching without resetting every knob.

…s, add conservative preset

…models in preflight, fix cache docs

comfyui_cache_dit_run detected the first call of a sampler run from the shared run-state slot, which Qwen-Image-2.1's condition stage fills before denoising, so first_call was never true and the Cache-DiT context was not refreshed after the first run. Track it in its own dict, cleared with the run state.

…, keep knobs on default

OnePunchMonk · 2026-10-20T15:54:33Z

Does the 3N fallback hold for dpm_adaptive? It's a wrapper closure, so I couldn't confirm its per-step call count.

github-actions Bot added documentation Improvements or additions to documentation diffusion SGLang Diffusion labels Oct 11, 2026

OnePunchMonk marked this pull request as ready for review October 11, 2026 15:53

OnePunchMonk requested review from AgainstEntropy, BBuf, HaiShaw, kevin-mii, mickqian, niehen6174, ping1jing2 and yichiche as code owners October 11, 2026 15:53

style: ruff format

c301cb4

[diffusion] comfyui: per-run Cache-DiT for every integrated model

f16b1bd

ComfyUI sends one request per DiT call, so the worker keeps one Cache-DiT context per sampler run, sized from the run's sample_sigmas.

[diffusion] cache-dit: allow remount after disabling a bare transformer

7585fd5

cache-dit 1.5.1 marks the FakeDiffusionPipeline class as cached and never clears it on disable_cache(transformer), so a knob change between requests asserted on remount.

This was referenced Oct 13, 2026

[Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin #43536

Merged

[diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight #44058

Merged

Merge branch 'main' into feat/comfyui-cache-options

6035ef6

Resolve test_comfyui_generator.py: keep both appended tests; the reuse test's last_options gains runtime_model_path, which main's gather_options now holds.

This was referenced Oct 13, 2026

[Bug] [diffusion] ComfyUI plugin: integrated-mode bugs in GGUF detection, import guard and VRAM accounting #43874

Closed

ComfyUI ideas OnePunchMonk/sglang#4

Open

Merge main into feat/comfyui-cache-options

a252f8f

Merge main into feat/comfyui-cache-options; allow off with tuned knob…

fb0c815

…s, add conservative preset

This comment has been minimized.

Sign in to view

lint: ruff format nodes.py

22c746a

OnePunchMonk changed the title ~~[diffusion] comfyui: add SGLDCacheOptions for per-run Cache-DiT~~ [Feat] comfyui: Add SGLDCacheOptions for per-run Cache-DiT Oct 16, 2026

This comment has been minimized.

Sign in to view

OnePunchMonk added 5 commits October 18, 2026 14:25

comfyui: measured conservative cache preset, allow cache_dit for all …

10c0add

…models in preflight, fix cache docs

[diffusion] comfyui: fix Cache-DiT separate cfg at cfg=1, reject Fn=0…

56913dc

…, keep knobs on default

Guard FakeDiffusionPipeline import, omit enable_separate_cfg when unset

a33d1a2

[Fix] comfyui: derive Cache-DiT step count from sampler model calls

01a41f8

niehen6174 and others added 5 commits October 22, 2026 15:02

Merge branch 'main' into feat/comfyui-cache-options

2958418

[Fix] comfyui: preserve cache compatibility after main merge

a4c5845

[Test] comfyui: fold Cache-DiT test sweeps into existing tests

e489bf6

Merge branch 'main' into feat/comfyui-cache-options

ed6ec53

Merge main into feat/comfyui-cache-options

d6a6c15

Navigation Menu

Repository navigation

[Feat] comfyui: Add SGLDCacheOptions for per-run Cache-DiT - #43696

[Feat] comfyui: Add SGLDCacheOptions for per-run Cache-DiT#43696
OnePunchMonk wants to merge 19 commits into
sgl-project:mainfrom
OnePunchMonk:feat/comfyui-cache-options

OnePunchMonk commented Oct 11, 2026 •

edited by github-actions Bot

Loading

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

niehen6174 commented Oct 12, 2026

Uh oh!

OnePunchMonk commented Oct 12, 2026

Uh oh!

OnePunchMonk commented Oct 12, 2026

Uh oh!

niehen6174 commented Oct 13, 2026

Uh oh!

niehen6174 commented Oct 16, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

OnePunchMonk commented Oct 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

OnePunchMonk commented Oct 11, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI States

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

niehen6174 commented Oct 12, 2026

Uh oh!

OnePunchMonk commented Oct 12, 2026

Uh oh!

OnePunchMonk commented Oct 12, 2026

Uh oh!

niehen6174 commented Oct 13, 2026

Uh oh!

niehen6174 commented Oct 16, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

OnePunchMonk commented Oct 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

OnePunchMonk commented Oct 11, 2026 •

edited by github-actions Bot

Loading