[feat] Extend disaggregated MiniMax-H3 stage with VAE encoders by asukaqaq-s · Pull Request #6939 · vllm-project/vllm-omni

asukaqaq-s · 2026-09-02T06:06:14Z

Purpose

Extend the existing disaggregated MiniMax-H3 text encoder stage with video VAE and audio WVAE encoders.

VAE separation: Move video/audio VAE encoding into Stage 0, while Stage 1 keeps decode-only VAE modules.Transfer visual/audio latents and shape metadata through the stage payload.
Shared world size, different parallelism: The new encoders reuse Stage 0's existing tensor-parallel world size and scheduler. Their execution strategies remain role-specific: text encoder uses tp, video VAE uses patch or leader, and audio WVAE uses leader.
Reuse existing text path: Build MiniMaxH3Encoder on top of MiniMaxH3TextEncoderBackbone, preserving the existing Qwen3-VL preprocessing, text embeddings, token tags, and stage interfaces.

Test Plan

MiniMax-H3 encoder payload, contract, and stage-input processor tests.
Validate standard and Turbo disaggregated deployment configurations.
Run Ruff checks on changed Python files.

vLLM Version: <fill in test environment>

vLLM-Omni Commit: 10ab4704

Test Result

ruff check: passed
ruff format: 17 files unchanged
Runtime tests: not run in this environment; require MiniMax-H3 model weights and GPU
setup.

@hsliuustc0106 @Gaohan123 @yuanwu2017 @yJader

chatgpt-codex-connector · 2026-09-02T06:06:19Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

vllm-omni-review-bot · 2026-09-02T06:09:36Z

This PR appears to be related to model: MinimaxH3.

Model owners: @david6666666

@asukaqaq-s, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

hsliuustc0106 · 2026-09-02T09:04:49Z

this should be able to deploy across nodes @yuanwu2017

hsliuustc0106 · 2026-09-02T22:10:15Z

+            additional_information.get("text_encoder_output") if isinstance(additional_information, Mapping) else None
+        )
+        if not isinstance(payload, Mapping) or not payload:
+            raise OmniClientError("MiniMax H3 diffusion stage requires encoder conditioning from the encoder stage")


[P1] This breaks the standard single-stage deployment. recipes/MiniMaxAI/MiniMax-H3.md (unchanged by this PR) serves vllm serve MiniMaxAI/MiniMax-H3 --omni, which routes here with the model_loaded["text_encoder"] default of True and no stage-0 encoder — forward/prepare_encode now raise "requires encoder conditioning from the encoder stage" on every request; at base that path was served by the deleted load_text_encoder/encode_prompt branch. Please restore the in-pipeline fallback or retire single-stage explicitly (update MiniMax-H3.md, note the break in the PR body); the new line in MiniMax-H3-Disaggregated.md ("standard MiniMax H3 recipes remain single-stage and continue to run all components inside the diffusion pipeline") is now false.

Fixed: the mode dispatch at pipeline_minimax_h3.py:1718-1729 restores the in-pipeline path for the default text_encoder=true, vae_encoder=true, the Qwen encoder import is back, both recipes now match the behavior, and test_minimax_h3_execution_modes.py covers all three modes. Resolved.

hsliuustc0106 · 2026-09-02T22:10:23Z

+                audio_lengths.append(int(length))
+        if audio_lengths:
+            if any(length < 80 or length > 600 for length in audio_lengths):
+                raise ValueError("MiniMax H3 audio references must each be between 2 and 15 seconds")


[P2] Only standalone audios get the [2,15]s OmniClientError check at prepare time (validate_reference_audio_waveforms(standalone_audios) in stage_input_processors/minimax_h3.py); embedded video audios are first caught here with a bare ValueError inside make_omni_output, and gpu_ar_model_runner has no per-request catch around it — a ref2va video whose audio stream is <2s or >15s kills the whole stage-0 engine instead of 4xx-ing that request (the old DiT-side check was OmniClientError). Smallest fix: validate the non-None video_audio_inputs in prepare_encoder_prompt too — waveform and sample rate are already in hand.

Fixed: prepare_encoder_inputs validates the combined effective audio list (embedded video audios + standalone) at prepare time with OmniClientError (encoder_processing.py:451-457), and _encode_local_media converts any residual ValueError to OmniClientError and propagates it rank-consistently via all_gather (pipeline_minimax_h3.py:1643-1657). Resolved.

hsliuustc0106 · 2026-09-02T22:10:29Z

+    ) -> AbstractContextManager:
+        parallel_size = int(getattr(self, "parallel_size", 1))
+        if parallel_size <= 1:
+            return nullcontext()


[P2] The old encode_image unconditionally forced self.model.parallel_tiling = False; with parallel_size == 1 (video_vae "leader" mode — set_parallel_size is never called) this nullcontext now runs image/keyframe encode with the constructor value int(config["vae_parallel_tiling"]) (vae.py:122), so encode becomes tiled whenever the gated checkpoint config is truthy — silent drift vs the reference whole-frame keyframe encode. Can't verify the checkpoint config in-repo: either restore the forced-False for parallel_size == 1 or add a base-vs-head A/B showing tiled image encode matches reference keyframe latents.

Fixed: encode_image forces parallel_tiling = False whenever parallel_size <= 1, with save/restore in the finally block (vae.py:430-456); the multi-rank path additionally falls back to rank-local tiling with a warning on tile shortage. Resolved.

hsliuustc0106 · 2026-09-02T22:10:36Z

    minimax_h3_prepare_denoise_rows,
    minimax_h3_publish_denoise_progress,
 )
-from .encoder import MiniMaxH3Qwen3VLEncoder


[P3] With this import removed, MiniMaxH3Qwen3VLEncoder (~1500 lines in diffusion/models/minimax_h3/encoder.py) has no production caller left — only tests (test_minimax_h3_contract.py, test_minimax_h3_encoder_sdpa.py, test_minimax_h3_dlo_lifecycle.py) still exercise it, so CI stays green on a dead module. Delete the module and its tests here, or note why they're kept.

Resolved by the single-stage restore — the pipeline imports MiniMaxH3Qwen3VLEncoder again (pipeline_minimax_h3.py:99), so the module has its production caller back.

hsliuustc0106

Requesting changes primarily for the single-stage break: MiniMaxH3Pipeline now requires stage-0 encoder conditioning unconditionally, but recipes/MiniMaxAI/MiniMax-H3.md (unchanged) still documents vllm serve MiniMaxAI/MiniMax-H3 --omni, which was served by the deleted in-pipeline encoder — every request on the standard deployment fails, and the new MiniMax-H3-Disaggregated.md line claiming standard recipes remain single-stage is now false. Please restore the in-pipeline fallback or retire single-stage explicitly (recipe + PR body).

Also worth addressing before merge: the embedded-video-audio [2,15]s check surfaces as a bare ValueError inside stage-0 make_omni_output (engine-fatal, vs the old DiT-side OmniClientError 4xx), and encode_image no longer forces parallel_tiling = False on the single-rank path. Details in the inline comments.

Signed-off-by: Jader <yjader@foxmail.com>

- Reject weights for disabled encoder components with a clear ValueError. - Enforce the shared 15-second budget across embedded and standalone audio. - Update regression tests for the unified audio budget contract. Signed-off-by: Jader <yjader@foxmail.com>

Signed-off-by: Jader <yjader@foxmail.com>

lishunyang12 · 2026-09-15T16:58:34Z

+    # Embedded soundtracks and standalone references share one request-level
+    # duration budget.  Validate the combined list so a mixed request cannot
+    # bypass the 15-second total limit by passing each category separately.
+    validate_reference_audio_waveforms(audio_inputs)


Keep embedded and standalone audio budgets separate, matching encode_media.

This logic and corresponding tests have been fixed in aabd895. Thanks

lishunyang12 · 2026-09-15T16:58:34Z

                    "latent_t": context["latent_t"],
                    "latent_h": context["latent_h"],
                    "latent_w": context["latent_w"],
                    "audio_t": context["audio_t"],


Save preencode_mp4, preencode_batch_frames, and video_codec_options in the step state.

fixed in bf61833

lishunyang12 · 2026-09-15T16:58:35Z

    from vllm_omni.diffusion.models.minimax_h3.pipeline_minimax_h3 import MiniMaxH3Pipeline

    pipeline = object.__new__(MiniMaxH3Pipeline)
+    pipeline.load_text_encoder = False


Set load_vae_encoder=False in this fixture; prepare_encode now reads it.

Signed-off-by: Jader <yjader@foxmail.com>

Re-port the MiniMax H3 timeline-guide feature onto main's split encoder / diffusion conditioning refactor (vllm-project#6939). Conflict resolution: - pipeline_minimax_h3.py: drop the request-preparation code main moved into ``encoder_processing`` (``_resolve_shape``, the old ``encode_prompt``, the ``_encode_*_conditions`` family, ``_validate_reference_image``) and re-attach the guide pipeline to ``_prepare_encoder_conditioning_inputs`` so guides work from either locally or remotely produced ``MiniMaxH3EncoderConditioning``. Reference/target row budgets are now exact instead of pre-decode estimates, and the distributed-error helpers come from ``distributed_errors`` rather than main's inline copies. - Guided requests need a local VAE encoder in the diffusion stage, so a ``vae_encoder=false`` diffusion stage now rejects guides with a 400. - encoder_processing.py: route the canvas through the shared ``resolve_minimax_h3_target_canvas`` and apply the guided down-only Ref2VA reference policy while preparing encoder images, keeping the Qwen presentation and the VAE reference in agreement. - stage_input_processors/minimax_h3.py: keep the pre-Qwen guide rejections on main's renamed ``prepare_encoder_prompt``. - tests: rebuild the timeline-guide suites on the encoder-conditioning entry point, keep the ``distributed_errors`` unit coverage, and drop cases whose subjects main deleted. Signed-off-by: shykoe <shykoe@gmail.com>

Rebase the unique vllm-project#6264 delta onto main a769d93 after merged vllm-project#6939 (d554e6b). Preserve upstream VAE encoder, multi-GPU/offload and request-start callback changes. Retain full-payload routing, TP fanout, inline cleanup and legacy text compatibility. Reconcile tests without replaying dependency commits. Signed-off-by: yuanwu <yuan.wu@intel.com>

…project#6939) Signed-off-by: asukaqaq-s <1311722138@qq.com> Signed-off-by: Jader <yjader@foxmail.com> Co-authored-by: Jader <yjader@foxmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: Matthieu Laneuville <matthieu.laneuville@surf.nl>

…project#6939) Signed-off-by: asukaqaq-s <1311722138@qq.com> Signed-off-by: Jader <yjader@foxmail.com> Co-authored-by: Jader <yjader@foxmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

按 §3 表的 H3 推理契约一行补 §3.1，写明： - 机制：`prepare_encoder_inputs` 的 `reference_image_shape` 回调只有 diffusion pipeline 传，分档时 bridge 不传，落回上游的"保持分辨率 + 32 对齐"。 - 实测数字：256x256 参考图下，单档编到 2048（视觉条件 64x64+1、latent 128x128），分档保持 256（8x8+1、latent 16x16）。 - 性质：并行的两条设计，非我方缺陷（上游没这个回调，两条路径都走上游几何）；已登记为有意偏离。 - 分档启动时若契约几何与上游不同会打 warning，不拒绝启动（上游的两 stage 档会用）。 - 现网 11 份档全是单 stage，因此走契约几何那条路。 - "让分档也应用契约几何" 单独立项：回调要跨 stage 边界，或把 resize 挪到 stage1 —— 但 stage0 在 vllm-project#6939 之后已把参考图编成 latent，stage1 无法再 resize。 Signed-off-by: reputationly <reputationly@gmail.com> Signed-off-by: reputationly <197039020@qq.com>

asukaqaq-s requested review from Gaohan123, NickCao, Sy0307, alex-jw-brooks, amy-why-3459, david6666666, gcanlin, hsliuustc0106, linyueqian, lishunyang12, tzhouam, wtomin and yenuo26 as code owners September 2, 2026 06:06

hsliuustc0106 reviewed Sep 2, 2026

View reviewed changes

hsliuustc0106 added enhancement New feature or request diffusion codes related to diffusion models labels Sep 2, 2026

hsliuustc0106 requested changes Sep 3, 2026

View reviewed changes

yJader mentioned this pull request Sep 8, 2026

[MiniMax-H3] Restore single-stage compatibility and refine VAE encoder disaggregation asukaqaq-s/vllm-omni#1

Merged

yJader requested review from Isotr0py, RuixiangMa, SamitHuang, ZJY0516, princepride and xuechendi as code owners September 9, 2026 17:04

yJader added 5 commits September 14, 2026 03:05

refactor: rename encoder output key to encoder_output

5a5cf9e

Signed-off-by: Jader <yjader@foxmail.com>

fix: add gpu_memory_utilization to disaggregated_turbo deploy config

9e89f34

Signed-off-by: Jader <yjader@foxmail.com>

fix: add validate for video_audios

fcd885a

Signed-off-by: Jader <yjader@foxmail.com>

refactor(minimax-h3): rename audio wvae role to audio vae

ad7e0d9

Signed-off-by: Jader <yjader@foxmail.com>

test(minimax-h3): initialize VAE encoder mode in contracts

0e3377a

Signed-off-by: Jader <yjader@foxmail.com>

yJader force-pushed the pr/vae-encoder-disaggregate branch from 7abb255 to 0e3377a Compare September 14, 2026 03:05

MrlixiangWE mentioned this pull request Sep 15, 2026

[Perf] Onboard omni encoder stages to the upstream multimodal encoder CUDA Graph framework #7452

Open

6 tasks

yJader added 3 commits September 15, 2026 13:20

Merge upstream/main into pr-6939-upstream-merge

86a458d

style: apply ruff

686b00a

Signed-off-by: Jader <yjader@foxmail.com>

lishunyang12 reviewed Sep 15, 2026

View reviewed changes

yJader and others added 4 commits September 15, 2026 17:17

fix: step execution path

bf61833

Signed-off-by: Jader <yjader@foxmail.com>

fix: independent calculate embeded & standalone audio len

aabd895

Signed-off-by: Jader <yjader@foxmail.com>

fix: restore text-conditioning contract

bdfed3c

Signed-off-by: Jader <yjader@foxmail.com>

Merge branch 'main' into pr/vae-encoder-disaggregate

d8384b4

hsliuustc0106 added the ready label to trigger buildkite CI label Sep 16, 2026

yJader and others added 3 commits September 16, 2026 03:40

fix: fix tests for updated MiniMax H3 pipeline config

a64afe5

Signed-off-by: Jader <yjader@foxmail.com>

Merge branch 'main' into pr/vae-encoder-disaggregate

d505ff9

Merge branch 'main' into pr/vae-encoder-disaggregate

e199c1e

hsliuustc0106 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Sep 16, 2026

hsliuustc0106 merged commit d554e6b into vllm-project:main Sep 16, 2026
5 of 9 checks passed

david6666666 mentioned this pull request Sep 17, 2026

[Bugfix][MiniMax-H3] Use Hopper-safe modulation precision #7693

Merged

avicii-forever mentioned this pull request Sep 17, 2026

[Frontend] MiniMax-H3: Add latent-mask editing to the ComfyUI extension #7575

Merged

Dong1017 mentioned this pull request Sep 28, 2026

[Performance]: Track MiniMax-H3 16-die deployment and Ref2VA host latency #8228

Closed

11 of 20 tasks

Conversation

asukaqaq-s commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

chatgpt-codex-connector Bot commented Sep 2, 2026

Uh oh!

vllm-omni-review-bot commented Sep 2, 2026

Uh oh!

hsliuustc0106 commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yJader Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

asukaqaq-s commented Sep 2, 2026 •

edited

Loading

yJader Sep 15, 2026 •

edited

Loading