[Dong1017]

Purpose

Move Ref2VA CPU random-noise preparation off the serial preparation path while preserving the existing random draws. A request-owned single-worker executor prepares initial video/audio noise and visual-reference noise during text/media encoding. The first output consumes the prepared tensors; continuation, external conditioning and subsequent outputs retain their existing behavior.

Visual-condition mixing keeps NPU input on NPU and preserves CPU FP32 coefficient rounding and operation order. Video-shape prediction reuses the checkpoint processor's actual temporal/spatial alignment methods and shared padding metadata; the pipeline checks predicted dimensions against encoded dimensions before consuming the prepared noise.

Related to #8228. This PR contains three H3 production files, two CPU test files and a reproducible eight-rank resident-VAE YAML. The YAML uses existing configuration options: TP8 text encoder, USP8 DiT, patch8/tile VAE, offload dit,text_encoder, and resident video/audio VAE. It includes the measured rank-local text-encoder weight transfer. Allocator telemetry and KV distribution are separate PRs; #8129 and #8229 contain the media optimizations.

Test Plan

vLLM Version: 0.29.0 for device measurements; 0.30.0 for CPU regression.

vLLM-Omni Commit: device measurements use 0e6791e9c4925c3539352cff2c2738bd48a1a65a plus recorded candidate changes. The publication branch is based on main d5a3380103df2fc827f095c60dbfb6e1c5655fd3.

Hardware/workload: 8×Ascend 910B1, 64 GB/device; PyTorch 2.10.0+cpu, torch-npu 2.10.0.post4. Fixed 15-second reference video with audio, seed 2101, requested 50 inference steps, flow shifts 12/3, short edge 768, aspect ratio 16:9, fps 24. Output: 1344×768, 362 frames with audio.

pytest -q tests/diffusion/models/minimax_h3/test_ref2va_initial_noise_preparation.py \
  tests/diffusion/models/minimax_h3/test_minimax_h3_packing.py

vllm serve "$MODEL" --omni \
  --config recipes/MiniMaxAI/minimax-h3-ref2va-8-rank-resident.yaml

The device experiment ran equivalent CLI parameters. The new YAML parses successfully; it does not add a new residency mechanism.

Test Result

• 23 focused CPU tests passed. Same-live-input correctness comparison on all eight ranks passed 114 checks each, including 30 bitwise tensor comparisons: all nine returned fields, branch attributes, static arguments and initial-noise tensors match the frozen reference. Shapes, dtypes, devices, layouts, strides and values match; CPU global RNG state is unchanged. NPU visual mixing matches the CPU reference bit for bit. This is preparation-input equivalence.
• An isolated profile for the visual-noise increment reduces common eight-rank idle inside the visual-mix context from 408.332 to 8.538 ms, saving 399.794 ms (97.9%). Three unprofiled E2E pairs for this increment have essentially unchanged medians, 2248.162351 → 2248.137201 s; peak allocated/reserved memory matches on all ranks.
• In the full-combination profile, initial/visual CPU-noise work overlaps text encoding by 0.804 s, and input-builder wall time falls 1.394456 → 0.091986 s. VAE load/offload calls disappear on all ranks with the resident configuration. Media-encoding and decode spans fall 29.461511 → 19.081425 s and 15.230415 → 9.802328 s; these spans also include allocator/configuration effects.

Complete-combination acceptance

Each service completed a fixed-input 50-step warmup on all eight ranks before three alternating unprofiled measurement pairs. Combining this PR, #8578, #8579, #8129, #8229 and resident VAE configuration gives client E2E medians 2253.664743 → 2239.304066 s, saving 14.360677 s (0.637%). Each pair improves: 15.435865 / 18.683454 / 14.360677 s.

A separate profiled pair covers all eight device event databases per arm. Common time with no device task on any rank falls 6.631556 → 3.039535 s (−54.17%); the preparation window falls 3.670694 → 0.941927 s (−74.34%). These are complete-combination results, with the CPU preparation mechanisms identified above; they are not assigned wholly to this PR.

Publication-baseline CPU validation (main d5a3380103 plus the three scoped candidates, vLLM 0.30.0 / PyTorch 2.13.0+cu130, accelerator devices hidden): 160 passed, 1 deselected, 17 warnings in 129.78 s, normal process exit. This covers the 23 preparation tests, 18 allocator/lifecycle/revert tests, and 119 KV cases. Tests run in isolation from the full repository plugin fixture, which requires vLLM 0.31. The one deselected existing example-CLI test imports that 0.31 engine interface and fails in the installed 0.30 runtime; its error is retained separately. Required cache-dit and pytest-mock packages are installed in task-local directories; the shared environment is unchanged.

[Dong1017]

Author self-review completed using the repository's precheck-pr, full mode, against main d5a3380103.

• Scope: three H3 production files, two focused CPU test files and one resident-VAE YAML. No media-conversion, allocator, KV or shutdown changes are included.
• Correctness: independent CPU generators preserve seeds and full draw shapes; prediction shares real VAE padding metadata and is checked against encoded shapes. The request owns the executor; exceptions propagate, context-manager exit joins work, and prepared tensors are consumed only for the first output. Continuation and external conditioning retain the original path.
• Numerical evidence: all-eight-rank preparation-input comparison passes 114 checks/rank, including 30 bitwise tensor comparisons. NPU mixing preserves CPU FP32 coefficient rounding and arithmetic order.
• Full diff-scoped code-quality, examples-policy and simplification checks found no blocking issue. The VAE processor metadata helper retains the existing Any-valued metadata contract; no new general abstraction or dependency is introduced.
• All applicable local pre-commit gates pass, including Ruff, formatting, typos, Python 3.10 mypy, test marks, SPDX, forbidden imports and new accelerator-call checks. No hooks were bypassed.
• Performance attribution distinguishes the isolated visual-mix idle reduction, overlapped CPU work, VAE configuration effects and the complete-combination E2E result.

Ready for maintainer review.

Publication-baseline CPU validation (main d5a3380103 plus the three scoped candidates, vLLM 0.30.0 / PyTorch 2.13.0+cu130, accelerator devices hidden): 160 passed, 1 deselected, 17 warnings in 129.78 s, normal process exit. This covers the 23 preparation tests, 18 allocator/lifecycle/revert tests, and 119 KV cases. Tests run in isolation from the full repository plugin fixture, which requires vLLM 0.31. The one deselected existing example-CLI test imports that 0.31 engine interface and fails in the installed 0.30 runtime; its error is retained separately. Required cache-dit and pytest-mock packages are installed in task-local directories; the shared environment is unchanged.

@vllm-omni-review-bot Please review this PR.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/offloader.md, docs/design/module/diffusion/diffusion_model_integration.md, docs/design/module/diffusion/index.md.

Module owners: @david6666666 @fhfuih @Bounty-hunter

Routing: @david6666666 via module of the changed files, semantic router, model owner; @fhfuih via module of the changed files, CODEOWNERS; @Bounty-hunter via module of the changed files

@Dong1017, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 4c2a91ef4d3a produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[Dong1017]

@vllm-omni-review-bot Please review this PR.

@david6666666 Could you review this change as the MiniMax-H3 model owner, especially its model semantics, correctness evidence and performance attribution?

Scope: CPU initial/visual-noise preparation overlapped with encoding, exact preparation-input checks, and a resident-VAE YAML using existing options.

The five changes have separate scopes: #8129, #8229, #8577, #8578 and #8579. The shared eight-rank Ref2VA evidence and per-change attribution are summarized in the Issue #8228 update. The existing Draft/Ready status is retained.

[hsliuustc0106]

Reviewed head 702410456965eda951be08e68e31d8353b677449.

[P2] Update the mixed-reference test's VAE mock — vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py:2632.

The existing test_mixed_reference_conditioning_matches_stage_zero_handoff configures encode_video() but leaves the new predict_video_latent_shape() unconfigured. Static tracing shows both inline and full_payload cases receive a Mock shape and fail the prediction-versus-encoding check. Set pipeline.video_vae.predict_video_latent_shape.return_value = (1, 2, 2) alongside the existing encoder return value and rerun both cases.

No production correctness defect identified. Reported CI, Python syntax checks, and diff hygiene pass. Pytest and NPU benchmarks were not rerun because an isolated sandbox for the fork could not start; the finding above is based on source tracing and an independent standard-library Mock behavior check.

[Dong1017]

@hsliuustc0106 Fixed in 6c2fdea.

The existing VAE mock now returns (1, 2, 2) from predict_video_latent_shape(). The test also distinguishes precomputed noise in the local path from deferred noise in the stage-zero handoff path, then checks that the actual _build_denoise_inputs() outputs (video_rows, audio_rows, cond_anchor, and audio_anchor) match exactly with torch.equal.

Both inline and full_payload cases pass on CPU: 2 passed. This fix changes only the test; production code is unchanged.

[david6666666]

These parameters keep both CPU noise buffers alive throughout denoising, even after _build_denoise_inputs() has moved the inputs to the device. context.pop() only removes the caller's references. Could we release the consumed buffers here to avoid retaining them on every rank?

[Dong1017]

Fixed in beb6c38.

Deleting the callee's parameters alone was insufficient: the caller's temporary keyword dictionary retained the CPU tensors until diffuse returned. diffuse now consumes the precomputed noise directly from the existing request context while building denoise inputs, so neither the context nor the caller's keyword dictionary retains the consumed buffers throughout denoising. The step-execution path still builds its inputs directly.

A regression test uses real CPU noise, actual denoise-input construction with CPU-to-meta transfers, and weak references to assert that every consumed host noise tensor is released before selecting the DiT transformer. The existing bitwise denoise-input equivalence tests remain enabled. 107 targeted CPU tests passed, and all applicable pre-commit checks passed. This fix addresses buffer ownership without introducing a new performance claim.

[david6666666]

If text or media encoding fails, this with still waits for the noise worker and runs any queued visual-noise job. Could we cancel pending futures on failure and add an encoder-failure regression test? The running task can still be joined for cleanup.

[Dong1017]

Fixed in beb6c38.

The request-owned executor now uses shutdown(wait=True, cancel_futures=True) in finally. A text- or media-encoder exception cancels pending initial/visual-noise jobs while still joining any running work. The original exception propagates to the caller.

Two regression cases inject failures at the text and media encoder boundaries, hold a real executor worker, and check that both pending noise jobs are cancelled and the running worker has finished. Together with the noise equivalence and lifetime cases, 107 targeted CPU tests passed. All applicable pre-commit hooks passed; the previous head's reported checks were successful, and the follow-up needs its own CI results.

[Dong1017]

@david6666666 Fixed the PR-owned NVIDIA/AMD CPU failures in 2e77dfd295.

Both failures were test_mixed_reference_conditioning_matches_stage_zero_handoff[inline/full_payload]. The existing diffuse mock did not consume the request-owned noise, leaving the test to compare unequal context dictionaries containing tensors. The update also converts the diffuse-only context argument into the consumed noise arguments expected by the input builder.

The test now consumes those buffers at the mocked diffuse boundary, matching the real method's contract. It still calls the real denoise-input builder and retains exact equality checks for video rows, audio rows and both condition anchors, plus the encoder-bypass and reference-layout assertions. One test file changed; no production code or numerical thresholds changed.

Matched CPU reproduction on refreshed head f1114bdf (vLLM 0.31.0 / PyTorch 2.13.0+cu130, devices hidden):

Arm	Result
Before this fix	2 failed, 140 passed; same two failures as CI
After this fix	142 passed

Coverage includes execution modes, initial/visual noise preparation and lifetime, packing, and H3 stage-input processing. Tests used the existing isolated CPU harness (--noconftest); no dependency API compatibility shims were added. All applicable pre-commit and commit checks passed, including DCO.

Latest failed checks on the previous head are now separated by cause:

Check	Attribution
NVIDIA 16944	The two PR-owned H3 handoff assertions above; fixed here.
AMD 13534	The same two assertions on shards 2/5 and 3/5; fixed here.
Intel 9815	Overall 30-minute job budget exhausted. The last active case was Qwen3-TTS test_text_to_audio_001[no_cuda_graph], during engine/model startup; this was not a H3 assertion or execution of the modified noise-preparation path. It needs a fresh job run.

NPU 8670 passed on f1114bdf; the new head needs its own CI results.

[Dong1017]

The remaining NVIDIA failure in build 16968 was a worker-RPC timeout, before either accuracy assertion could run.

Both 50-step requests were still advancing when the 600-second executor budget expired: I2VA reached step 36/50 at approximately 18 s/step; Ref2VA reached step 23/50 at approximately 28 s/step. The guard covers the whole execute_model request, so a normally progressing generation can exceed it. The tests already allowed 3600 seconds at the HTTP/client layer.

Fix: 4c2a91ef4d3ad3396e1313585bbde53745ef96fd adds VLLM_OMNI_DLO_DP_WAVE_TIMEOUT=str(REQUEST_TIMEOUT_SECONDS) to both test servers. This is a three-line change in one test file. Production defaults, model/sampling settings, and all SSIM/PSNR and metadata assertions are unchanged.

Validation: all applicable pre-commit and commit checks passed; importing the real executor with the test environment confirmed a 3600-second worker budget; the existing timeout-related CPU tests passed (2 passed). The full NVIDIA 50-step accuracy rerun remains the final check.

On the preceding head 2e77dfd295, AMD, Intel and NPU CI all passed; NVIDIA's only failing job was the two accuracy requests above.

@david6666666 Can you add ready label again when you have a chance? Thanks

[Dong1017]

CI check for head 4c2a91ef4d3ad3396e1313585bbde53745ef96fd:

• NVIDIA 16997, Intel 9862 and NPU 8724 passed, as did pre-commit, wheels, DCO and docs.
• The NVIDIA 50-step I2VA and Ref2VA accuracy job passed both tests. I2VA: SSIM 0.976997, PSNR 35.822602 dB. Ref2VA: SSIM 0.980586, PSNR 42.648519 dB. The original SSIM ≥ 0.97 / PSNR ≥ 34 dB thresholds were retained. This closes the preceding worker-RPC timeout failure.

The remaining failure is AMD 13583, Simple · Other Test:
tests/host_weight_runtime/test_runtime_resolution.py::test_domain_lock_timeout_and_fresh_runtime_recovery[required-failed].

All assertions about the expected domain-lock timeout report passed. The failure is at process.join(5); assert process.exitcode == 0: the child was still alive, so its exit code was None. Job summary: 1 failed, 5039 passed, 68 skipped.

This is outside the H3 CPU-preparation path. The test directly exercises Host Weight Runtime; the test file is byte-identical to current main (blob cd2de28165d3fd3dba4ce4603cb8262506291b86), and this PR changes neither the host-weight runtime nor its test. The log establishes a child-exit deadline failure; it does not establish the underlying reason for the delayed exit. A focused rerun of this test on the AMD worker is the useful next check. No existing issue matching this exact assertion was found. No code or assertion changes were made for this failure.

[david6666666]

LGTM