Repository navigation
[Bugfix][MiniMax-H3] Release decoded outputs before the next seed - #6997
Closed
heyuanliu-intel wants to merge 1 commit into
Closed
heyuanliu-intel wants to merge 1 commit into
heyuanliu-intel wants to merge 1 commit into
Conversation
## Purpose **Problem:** with model-level CPU offload, vllm-project#6072 evicts the DiT before VAE decode, but the decoded output stays on the device until the request returns. For `num_outputs_per_prompt > 1` the appended output of seed N is still resident when seed N+1 reloads the DiT, so the peak the eviction removed comes back; with a large output even the first request can OOM in decode. **Fix:** on the executor reply rank, move each decoded output to the host inside the same seed iteration while active model-level offload is installed; non-reply ranks keep a metadata placeholder. Video is released after `_prepare_minimax_h3_video_output` quantizes to `uint8`, so vllm-project#6824's transfer optimisation is kept. Other offload modes and the no-offload path are unchanged; the design doc records the exception. ## Test Plan **Reproduce:** 8 XPU cards, TP4xSP2, ref2va at 1344x768 / 124 frames / 50 steps, Cache-DiT and model-level CPU offload for encoders and VAE. The first request finishes denoising (49/49) and then fails in `klvae.py:621 write_part -> torch.empty` with `XPU out of memory. Tried to allocate 1.43 GiB ... 1.42 GiB is free`, HTTP 500. ```bash pytest -q tests/diffusion/models/minimax_h3/test_minimax_h3_offload.py ``` **vLLM Version:** as pinned by `requirements/`. **vLLM-Omni Commit:** rebased onto current `main`. ## Test Result CPU tests pass (release only under active model-level offload; reply rank gets a host tensor, non-reply ranks a placeholder; video copied after quantization; no-offload path unchanged). On the 8-card run with this change: 0 OOM, 3/3 requests HTTP 200, 24 `Evicted 1 DiT module(s) (15.43 GiB)` lines. The device run used one output per prompt; the multi-seed overlap is covered by the CPU tests. Signed-off-by: heyuanliu-intel <heyuan.liu@intel.com>
heyuanliu-intel
requested review from
Gaohan123,
NickCao,
david6666666,
hsliuustc0106,
wtomin and
yenuo26
as code owners
September 3, 2026 08:48
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
This PR appears to be related to model: MinimaxH3. Model owners: @david6666666 @heyuanliu-intel, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
hsliuustc0106
added
high priority
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Problem: with model-level CPU offload, #6072 evicts the DiT before VAE decode, but the decoded output stays on the device until the request returns. For
num_outputs_per_prompt > 1the appended output of seed N is still resident when seed N+1 reloads the DiT, so the peak the eviction removed comes back; with a large output even the first request can OOM in decode.Fix: on the executor reply rank, move each decoded output to the host inside the same seed iteration while active model-level offload is installed; non-reply ranks keep a metadata placeholder. Video is released after
_prepare_minimax_h3_video_outputquantizes touint8, so #6824's transfer optimisation is kept. Other offload modes and the no-offload path are unchanged; the design doc records the exception.Test Plan
Reproduce: 8 XPU cards, TP4xSP2, ref2va at 1344x768 / 124 frames / 50 steps, Cache-DiT and model-level CPU offload for encoders and VAE. The first request finishes denoising (49/49) and then fails in
klvae.py:621 write_part -> torch.emptywithXPU out of memory. Tried to allocate 1.43 GiB ... 1.42 GiB is free, HTTP 500.vLLM Version: as pinned by
requirements/. vLLM-Omni Commit: rebased onto currentmain.Test Result
CPU tests pass (release only under active model-level offload; reply rank gets a host tensor, non-reply ranks a placeholder; video copied after quantization; no-offload path unchanged). On the 8-card run with this change: 0 OOM, 3/3 requests HTTP 200, 24
Evicted 1 DiT module(s) (15.43 GiB)lines. The device run used one output per prompt; the multi-seed overlap is covered by the CPU tests.