Skip to content

[diffusion] Add MiniMax-H3 to ComfyUI integrated mode - #1

Open
niehen6174 wants to merge 1510 commits into
feat/comfyui-plugin-refactorfrom
feat/comfyui-h3
Open

niehen6174 wants to merge 1510 commits into
feat/comfyui-plugin-refactorfrom
feat/comfyui-h3

Conversation

@niehen6174

@niehen6174 niehen6174 commented Aug 22, 2026 •

Copy link
Copy Markdown
Owner

Motivation

sgl-project#35960 turns ComfyUI integrated mode into a DiT-only profile on the native pipeline (adapter + checkpoint spec). Flux / Z-Image / Qwen already sit on that path. MiniMax-H3 does not: it is a packed [video, audio] DiT with T2V, I2VA, FL2VA, and R2V layouts that ComfyUI already ships as community graphs.

This PR adds H3 to that same contract so ComfyUI keeps CLIP / VAE / the sampler loop, and SGLang runs one packed DiT forward per apply_model.

24GB skip-step speed-ups are not in this stack. They land in #35858 (Cache-DiT + layerwise) and #35684 (Spectrum). After those merge, a follow-up will expose the recommend recipes on SGLDOptions.

Modifications

  • H3 adapter + worker step — pack/unpack [video, audio], translate ComfyUI layout / RoPE / denoise mask, match the official audio sigma-shift ODE.
  • Checkpoint spec — register MiniMax-H3; GGUF overlay; read AdaLN curve from pruned BF16 so the DiT is not built as the unpruned MLP.
  • Session / generator — cache scoped to one sampler.sample(); one SGLD generator per process (two-pass upscaler and back-to-back H3 graphs).
  • TP — shard fused MergedColumn per logical matrix.
  • Shipped graphs — T2V, R2V, two-pass latent upscaler.

Workflows

ComfyUI R2V UI:

comfyui_sgld_ref2v

Shipped under apps/ComfyUI_SGLDiffusion/workflows/:

Graph Task
minimax_h3_t2v_sgld.json T2V / fl2va DiT
minimax_h3_r2v_sgld.json Reference-to-video
minimax_h3_t2v_sgld_upscaler.json Low-res then 3D ×2 refine

Also tested on this branch (H3 only):

Task Result
I2VA (first frame) / FL2VA (first+last) OK
R2V image / R2V video+audio OK (needs the ref2va checkpoint)
kitchen_int8 + layerwise OK
pruned BF16 + layerwise OK
GGUF Q4 overlay OK
sol_attn / sage_attn OK
Cache-DiT OK (24GB + layerwise coexist waits on sgl-project#35858)
lightx2v turbo LoRA on BF16 OK
Same UNET sampled twice OK
GGUF × LoRA Rejected (expected)

GGUF × LoRA is fail-closed. kitchen-int8 × LoRA is not. H3 SP needs ulysses_degree=2.

Tests

Unit: test_comfyui_h3.py (layout vs ComfyUI, adapter, step kwargs, profile).

4090 D 24GB · ComfyUI 0.30.0 · CLIP on CPU. Wall is prompt e2e.

Smoke (432×240 · 25f · 4 step) unless noted:

H3 path Wall
kitchen_int8 + sol + layerwise 96 s
pruned BF16 + layerwise 104 s
R2V ref2va + kitchen 145 s
Two-pass upscaler 432→864 217 s

768P quality (1344×768 · 5s · 20 step · kitchen_int8 + FA + layerwise):

Task Wall
I2VA 418 s
FL2VA 426 s
R2V image 609 s
R2V video+audio 1433 s

Test Video:

i2va_ramen_first.mp4
ref2va_girl.mp4
ref2va_video.mp4

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

mmangkad and others added 13 commits September 21, 2026 18:33
…oject#39524)

Co-authored-by: mmangkad <mohammad.angkad@radixark.ai>
…roject#40507)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
…_all (sgl-project#40590)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…page-table transform (sgl-project#39340)

Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ject#35504)

Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…gl-project#38546)

Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
rwang5203 and others added 18 commits September 25, 2026 12:59
…oject#28135)

Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
…bound (sgl-project#41201)

Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
…oject#41067)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
CjhHa1 and others added 5 commits September 25, 2026 17:27
… the residency manager (sgl-project#34417)

Co-authored-by: mickqian <mickqian@users.noreply.github.com>
CI does not install ComfyUI. Keep Qwen executors and the model patcher off the unit-test import path.
load_modules checks that flag, and this SimpleNamespace omitted it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.