[Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin by OnePunchMonk · Pull Request #43536 · sgl-project/sglang

OnePunchMonk · 2026-10-10T09:32:39Z

Summary

Integrated-mode fixes for python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/, narrowed per review. Bug numbers match the issue.

Bug 1: GGUF unets could not be detected or loaded. Detection now reads names and logical shapes from the GGUF header with read_gguf_tensor_meta() and runs ComfyUI's detection on them, like safetensors, so no BF16 companion file is needed. A GGUF picked in SGLDUNETLoader is loaded as the DiT. The GGUF loader maps tensor names through the spec's param mapping for both quant lookup and weight loading: Z-Image qkv -> to_qkv, w1/w3 packed shards are concatenated into the fused w13, and packed tensors whose layer builds without GGUF (adaLN ReplicatedLinear) are dequantized on load.
Bug 2: .gguf leaked into ComfyUI's global diffusion_models registry. SGLDUNETLoader now builds its own list with ComfyUI's recursive_search / filter_files_extensions, so the stock loader no longer offers GGUF files.
Bug 3: detection failure returned a bare None ("cannot unpack non-iterable NoneType"). It now raises ValueError with the model path.
Bug 4: a missing runtime made every node vanish silently. executors/minimax_h3.py guards its runtime imports like base.py, and __init__.py logs why registration failed.

Dropped from this PR: Cache-DiT (#43696), worker sleep/wake (#44086), server-mode items (#44082), and VRAM accounting (moved to #44086).

Tests

Regression tests live in the existing unit files and reuse their fixtures (test_comfyui_checkpoint_spec.py, test_comfyui_generator.py, test_comfyui_lora_node.py); no new test file. Targeted CPU run: 52 passed.

ComfyUI v0.39.2 headless, 1x L40S, Z-Image Turbo, 1024x1024, 8 steps, seed 42:

Path	Time	GPU mem used	vs native BF16
Native ComfyUI BF16	2.87 s	-	-
SGLD BF16	2.79 s	23.7 GB	24.2 dB PSNR / 0.879 SSIM
SGLD `z-image-turbo-Q4_K_M.gguf`	3.09 s	14.7 GB	14.2 dB PSNR / 0.431 SSIM

The GGUF image keeps the same composition; the lower PSNR is Q4 drift in fine detail against a BF16 reference.

H3 GGUF: checked the real minimax_h3_fl2va_pruned-Q4_K.gguf header against the new name mapping; 0 of 532 tensor names change, so the H3 path loads as before.

TODO: H3 GGUF end-to-end ComfyUI run still pending.

Known gap: Flux ComfyUI GGUF likely still needs its _convert_weights reshapes on the GGUF path; left as a follow-up.

CI States

Latest PR Test (Base): ✅ Run #39834374029
Latest PR Test (Extra): ❌ Run #39834374072
Latest PR Test (AMD ROCm 10): ❌ Run #39834374024

OnePunchMonk · 2026-10-10T18:29:18Z

hi @niehen6174,

Planning to add support for a few minor features.

Video-to-video: Add a VIDEO input and a denoise-strength control to the generic video node. /v1/videos already accepts video_path / video_reference, so this is for restyling and extending clips.
Generic audio-video node: One node instead of one per model (H3, LTX-2, MOVA, Kandinsky-6). It takes an optional AUDIO input for lip-sync or music-driven video. It returns an AUDIO output that ComfyUI's audio nodes can use.
Prompt enhancement: The server supports enhance_prompt. Return the rewritten prompt as a STRING so users can see, edit and reuse it.
Checkpoint sweep: Uses /update_weights_from_disk to swap weights in place with no restart. It runs the same prompt and seed across step-500 to step-5000 and lays the results out as a grid.
LoRA browser and stacking: Fill a dropdown from /v1/list_loras. Support several LoRAs with per-LoRA strength and merge mode. The current Set LoRA node takes one free-text path and has no strength control.
Workflow to API recipe: Turns an SGLang-backed graph into a ready-to-run curl request, Python snippet, or sglang generate command with all parameters filled in.

Wanted to get your view on this.

Thank you!

OnePunchMonk · 2026-10-11T12:45:49Z

built these as part of #43673

niehen6174 · 2026-10-13T09:33:57Z

Thanks for the audit, there are real fixes here. To keep it reviewable, could you narrow this PR to the integrated-mode fixes (#1, #2, #5, #8, and #12 if you want)?

Please drop Json Decode && Mutl-Turns #4, since [Feat] comfyui: Add SGLDCacheOptions for per-run Cache-DiT #43696 handles Cache-DiT per run.
enable an installation without CUDA_HOME? #13 sleep/wake is a feature rather than a fix, so let's leave it out for now.
Please remove the server-mode items (Add install with pip #3, Fix test cases #6, fix radix cache match #7, Gemini Backend #9, Improve Readme #10, Update Readme #11, Colab? #14) or move them to a separate PR. They're lower priority for us right now.

…l nodes silently executors/minimax_h3.py imported sglang.multimodal_gen unguarded at module scope, breaking the "keep nodes importable without the runtime" pattern base.py/flux.py/zimage.py already follow. core/generator.py imports every executor unconditionally, so this took Flux/Z-Image down with it whenever the diffusion extras weren't installed. __init__.py's except branch then swallowed the resulting ImportError with no logging, so every node just vanished from ComfyUI with no indication why. Guard the import the same way base.py does, raise clearly on first use instead, and log the failure in __init__.py.

…chitecture detect _h3_detect_companion always looked for a MiniMax-H3 BF16 companion regardless of what GGUF was actually being loaded. A Flux/Qwen/Z-Image GGUF either failed with an H3-specific error, or, if an H3 BF16 file happened to exist in diffusion_models, silently built the H3 model around the wrong weights. Infer the real architecture from an explicit SGLDOptions.model_type hint or the GGUF's filename, look up a companion from a per-architecture table, and raise a clear error when the architecture can't be determined instead of defaulting to H3.

… diffusion_models registry SGLDUNETLoader enabled .gguf by mutating folder_paths.folder_names_and_paths["diffusion_models"] at import time, a global registry every other node reads too. This made ComfyUI's own stock "Load Diffusion Model" node offer GGUF files it cannot load. Build SGLDUNETLoader's own file list locally (stock get_filename_list plus a local .gguf scan of the same folders) instead of mutating shared state.

…onfusing TypeError get_comfyui_model returned a bare None on two model-config-build failure paths, but load_model always unpacks the result into comfyui_model, model_config, model_type. A user hitting either failure saw "cannot unpack non-iterable NoneType object" instead of the real error. Raise ValueError with the model path and detected type instead.

SGLDModelPatcher.model_size() returned a hardcoded guess for flux (27GB) and lumina2 (8GB) and 0 for everything else (qwen_image, minimax_h3), so ComfyUI freed VRAM for some SGLD models and not others even though the weights live in the worker process, not in this one. Compute the real byte size from the checkpoint header at load time (parameter count * dtype size) and pass it into the patcher, falling back to the old table only when a caller builds a patcher without that. For a GGUF load, size from the actual (much smaller, quantized) GGUF file rather than the BF16 companion used only for ComfyUI's architecture detect.

# Conflicts: # python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/minimax_h3.py

… merging main main added VDNH3Executor, whose class body computes its sigma grid from the runtime at import time, which broke the import guard. Only compute it when the runtime imported. Also update the new H3 generator test stub for get_comfyui_model's four-value return.

niehen6174 · 2026-10-14T09:47:16Z

Thanks for narrowing it down. Bugs 2, 3 and 4 look good. Two changes:

Bug 1: instead of filename hints and a BF16 companion table, could you read the GGUF header with read_gguf_tensor_meta() (runtime/loader/gguf_weights.py) and run ComfyUI's detection on those names and logical shapes, the same way we already do for safetensors? Then no companion file is needed.
Bug 5: please move it to [Feat] comfyui: Free the SGLD worker's GPU memory when ComfyUI unloads the model #44086. On its own it makes things worse: loaded_size() stays 0, so ComfyUI would try to free the full model size on every sampling run.

…model types" This reverts commit 7f5e9f3.

Read tensor names and logical shapes with read_gguf_tensor_meta() and run ComfyUI's detection on them, as is done for safetensors. Drops the filename hints and the BF16 companion table.

OnePunchMonk · 2026-10-14T18:46:43Z

hi @niehen6174 , made the changes.

…ents

… the VRAM revert

… loader With header-based detection the .gguf itself is the worker's model path, but is_comfyui_single_file only accepted .safetensors, so the worker skipped load_comfyui_transformer. Accept .gguf there, use model_path as the GGUF file when no transformer_weights_path is set, and read the Flux guidance tensor name from the GGUF header instead of opening it as safetensors.

niehen6174 · 2026-10-21T05:22:53Z

Tested the latest head on RTX 5090 with real ComfyUI workflows. BF16 and H3 GGUF both work, but Z-Image GGUF (z-image-turbo-Q4_K_M.gguf) still fails during loading:

Linear layer 'noise_refiner.0.attention.to_qkv' has no weight in the GGUF checkpoint

The checkpoint contains noise_refiner.0.attention.qkv.weight, while the quantized layer looks up to_qkv.weight. It looks like the raw GGUF tensor metadata isn't aligned with the runtime layer names.

Could you normalize the names consistently for both GGUF metadata lookup and weight loading, including packed QKV handling? Please also add a small regression test for this mapping and rerun Z-Image GGUF end to end, while keeping the working BF16 and H3 GGUF paths intact.

Map GGUF tensor names through the spec's param mapping for both the quant lookup and weight loading: qkv -> to_qkv, and w1/w3 shards are concatenated into the fused w13. Packed tensors whose layer builds without GGUF (adaLN ReplicatedLinear) are dequantized on load.

Drop test_sgld_bugfixes.py and its ComfyUI stub harness; move each case into the existing unit file that already has the fixture. List GGUF DiTs with ComfyUI's recursive_search/filter_files_extensions, and rely on the GGUF reader's own error for missing files.

OnePunchMonk · 2026-10-21T17:29:44Z

hi @niehen6174, thanks for testing. Fixed in the latest push.

The GGUF loader now maps tensor names through the spec mapping for both quant lookup and weight loading. For Z-Image, qkv maps to to_qkv, the packed w1/w3 shards are concatenated into w13, and packed adaLN weights (plain ReplicatedLinear) are dequantized on load. Added a regression test for the mapping in test_comfyui_checkpoint_spec.py.

ComfyUI v0.39.2, 1x L40S, 1024x1024, 8 steps, seed 42:

Path	Time	vs native BF16
Native BF16	2.87 s	-
SGLD BF16	2.79 s	24.2 dB / 0.879 SSIM
SGLD `z-image-turbo-Q4_K_M.gguf`	3.09 s	14.2 dB / 0.431 SSIM

The GGUF image keeps the same composition; the gap is Q4 detail drift.

H3 GGUF: the new mapping leaves all 532 tensor names of minimax_h3_fl2va_pruned-Q4_K.gguf unchanged, so that path should load as before. TODO: H3 GGUF end-to-end run still pending on my side, would be great if you can recheck it on your setup.

# Conflicts: # python/sglang/multimodal_gen/test/unit/test_comfyui_checkpoint_spec.py

niehen6174 · 2026-10-22T09:27:20Z

/tag-and-rerun-ci

OnePunchMonk requested review from AgainstEntropy, BBuf, HaiShaw, kevin-mii, mickqian, niehen6174, ping1jing2 and yichiche as code owners October 10, 2026 09:32

github-actions Bot added the diffusion SGLang Diffusion label Oct 10, 2026

OnePunchMonk changed the title ~~[diffusion] comfyui: fix 12 bugs in the SGLDiffusion ComfyUI plugin~~ [diffusion] comfyui: fix 14 bugs in the SGLDiffusion ComfyUI plugin Oct 10, 2026

OnePunchMonk mentioned this pull request Oct 10, 2026

ComfyUI ideas OnePunchMonk/sglang#4

Open

11 of 14 tasks

OnePunchMonk mentioned this pull request Oct 12, 2026

[Bug] [diffusion] ComfyUI plugin: integrated-mode bugs in GGUF detection, import guard and VRAM accounting #43874

Closed

4 tasks done

OnePunchMonk added 6 commits October 13, 2026 20:27

[diffusion] lint: apply ruff format

a0030b7

OnePunchMonk force-pushed the fix/comfyui-sgld-bugs branch from 3dcefc9 to a0030b7 Compare October 13, 2026 15:00

OnePunchMonk changed the title ~~[diffusion] comfyui: fix 14 bugs in the SGLDiffusion ComfyUI plugin~~ [diffusion] comfyui: fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin Oct 13, 2026

OnePunchMonk added 2 commits October 13, 2026 20:49

Merge remote-tracking branch 'origin/main' into narrow/comfyui-sgld-bugs

552c4fb

# Conflicts: # python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/minimax_h3.py

bingxche mentioned this pull request Oct 14, 2026

[CI Monitor] Daily Report - 2026-10-14 bingxche/sglang-ci-bot#225

Open

OnePunchMonk added 2 commits October 14, 2026 15:20

Revert "[diffusion] comfyui: fix inconsistent VRAM accounting across …

bb1977a

…model types" This reverts commit 7f5e9f3.

[diffusion] comfyui: detect GGUF architecture from the GGUF header

c767d11

Read tensor names and logical shapes with read_gguf_tensor_meta() and run ComfyUI's detection on them, as is done for safetensors. Drops the filename hints and the BF16 companion table.

niehen6174 and others added 3 commits October 15, 2026 03:18

[diffusion] comfyui: describe current behavior in the plugin-fix comm…

e72a24d

…ents

[diffusion] comfyui: restore the 3-tuple get_comfyui_model stub after…

3a2dd88

… the VRAM revert

Merge origin/main into fix/comfyui-sgld-bugs

26e0b9e

OnePunchMonk changed the title ~~[diffusion] comfyui: fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin~~ [Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin Oct 16, 2026

OnePunchMonk added 2 commits October 18, 2026 16:01

[Fix] comfyui: list GGUF DiTs in subfolders, drop dead test setup

3494019

OnePunchMonk added 2 commits October 21, 2026 22:41

github-actions Bot added the lora label Oct 21, 2026

[Fix] comfyui: fold GGUF routing check into the GGUF load test

37e843a

Merge remote-tracking branch 'origin/main' into fix/comfyui-sgld-bugs

baa36be

# Conflicts: # python/sglang/multimodal_gen/test/unit/test_comfyui_checkpoint_spec.py

niehen6174 approved these changes Oct 22, 2026

View reviewed changes

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 22, 2026

niehen6174 and others added 2 commits October 22, 2026 15:02

[Test] comfyui: isolate adapter registry in import failure tests

82f5915

Merge branch 'main' into fix/comfyui-sgld-bugs

f4244fb

niehen6174 merged commit fa628c5 into sgl-project:main Oct 23, 2026
146 of 153 checks passed

Navigation Menu

Repository navigation

[Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin - #43536

[Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin#43536
niehen6174 merged 21 commits into
sgl-project:mainfrom
OnePunchMonk:fix/comfyui-sgld-bugs

OnePunchMonk commented Oct 10, 2026 •

edited by github-actions Bot

Loading

Uh oh!

OnePunchMonk commented Oct 10, 2026

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

niehen6174 commented Oct 13, 2026

Uh oh!

niehen6174 commented Oct 14, 2026

Uh oh!

OnePunchMonk commented Oct 14, 2026

Uh oh!

niehen6174 commented Oct 21, 2026

Uh oh!

OnePunchMonk commented Oct 21, 2026

Uh oh!

niehen6174 commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

OnePunchMonk commented Oct 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

CI States

Uh oh!

OnePunchMonk commented Oct 10, 2026

Uh oh!

OnePunchMonk commented Oct 11, 2026

Uh oh!

niehen6174 commented Oct 13, 2026

Uh oh!

niehen6174 commented Oct 14, 2026

Uh oh!

OnePunchMonk commented Oct 14, 2026

Uh oh!

niehen6174 commented Oct 21, 2026

Uh oh!

OnePunchMonk commented Oct 21, 2026

Uh oh!

niehen6174 commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

OnePunchMonk commented Oct 10, 2026 •

edited by github-actions Bot

Loading