Repository navigation
[Bugfix] Avoid enlarging MiniMax H3 reference images and videos - #8253
Merged
lishunyang12 merged 5 commits intoSep 28, 2026
Merged
lishunyang12 merged 5 commits into
lishunyang12 merged 5 commits into
Conversation
lishunyang12
requested review from
Gaohan123,
NickCao,
Sy0307,
amy-why-3459,
david6666666,
gcanlin,
hsliuustc0106,
linyueqian,
tzhouam and
yenuo26
as code owners
September 28, 2026 16:17
|
This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md. Module owners: @tzhouam @fake0fan @Gaohan123 Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files @lishunyang12, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Omni ReviewBot triage noteResolved as of |
Signed-off-by: lishunyang12 <lishunyang12@users.noreply.github.com>
lishunyang12
force-pushed
the
fix/minimax-h3-reference-image-resolution
branch
from
September 28, 2026 16:30
eaf4861 to
a11c6e8
Compare
lishunyang12
changed the title
ywang96
approved these changes
Sep 28, 2026
Collaborator
Author
|
Let's just remove upscaling option and stick to original image size and video size for vae encoder instead. |
Signed-off-by: lishunyang12 <lishunyang12@users.noreply.github.com>
lishunyang12
changed the title
lishunyang12
added
ready
Signed-off-by: lishunyang12 <lishunyang12@users.noreply.github.com>
lishunyang12
changed the title
lishunyang12
added
ready
lishunyang12
changed the title
lishunyang12
enabled auto-merge (squash)
September 28, 2026 19:44
Signed-off-by: lishunyang12 <lishunyang12@users.noreply.github.com>
lishunyang12
force-pushed
the
fix/minimax-h3-reference-image-resolution
branch
from
September 28, 2026 19:46
c6b383f to
9d2589b
Compare
lishunyang12
added
ready
1 task done
devin-ai-integration Bot
added a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 12, 2026
- §5.3 新增 VDN 命名隔离：我方 vdn.VDNCheckpoint 与上游 vdnh3.VDNCheckpoint 同名、 self._vdn 同名，上游一侧改用 UpstreamVDNCheckpoint / self._vdn_upstream； vdnh3_attn.py 依赖 vllm-project#7379 新增的 capabilities.py，禁止单挑 vllm-project#8439 - §5.3 参考图：vllm-project#8253 改的是 preprocessing 兜底路径，现网 DIT 路径经 pipeline 的 reference_image_shape 回调走我方几何；裁决改为两边都保留，更正上一版 "只在 guided 路径生效"的错误表述 Signed-off-by: reputationly <reputationly@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
reputationly
pushed a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 12, 2026
表述更正： - §5.0 A4 / §6.3：撤销"本轮唯一一处会改变现网 H3 数值路径"的说法。生产镜像 基于 origin/main，10-04 带进来的 85 个提交对现网全是新代码，至少还有： flash_attn.py(+300)/utils/fa.py(+25)/attention/layer.py(+93)（现网 VDN_WINDOW_ATTN 直接依赖前两者）、pipeline_minimax_h3.py(+498)、 reference_video.py(vllm-project#8253)、vllm-project#7355 视频传输。md5 不一致时的排查顺序改为 solver -> flash_attn/fa -> 其余上游改动 -> 基座算子 新增 P2（参考视频，vllm-project#8253 的第二处改动）： - vllm-project#8253 同时改了 reference_video.py:_reference_video_shape：参考视频面积小于 画布时不再放大到画布，只对齐 32 倍数 - 已核实现网会走到：reference_video.py:518 被 prepare_reference_videos 调用； pipeline:2338 按 reference_video_decode_mode 分派；现网是 legacy_h264_intermediate（strategy.py:314 默认值，已从生产容器日志确认）； reference_video.py 不在 50 个冲突里，改动会自动进树；参考图的 reference_image_shape 回调管不到视频路径 - 裁决：补一个与 VLLM_OMNI_H3_REF_IMAGE_NO_UPSCALE 同形的开关，默认关闭 （保持现网"放大到画布"）。理由是原则 1 优先、且参考图在现网是保持放大的， 直接采用上游会造成"图放大、视频不放大"的内部不一致 命令兼容性： - §4 加写法 B（临时 worktree 里 merge --no-commit + --abort），不依赖 git 版本；写法 A（merge-tree --write-tree）标注需要 git >= 2.38 §6.3 ref2va 用例新增：用比画布小的参考视频，对比合并前后送进模型的分辨率 Signed-off-by: reputationly <reputationly@gmail.com>
reputationly
pushed a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 12, 2026
P2 补全（原来只写"补一个开关"，执行者无法照做）： - 开关名 VLLM_OMNI_H3_REF_VIDEO_NO_UPSCALE，默认关闭 = 保持放大到画布 - 改哪里：reference_video.py 的 _reference_video_shape；开关关闭时跳过 vllm-project#8253 新增的"面积小于画布则不放大"分支 - 怎么提交：vllm-project#8253 已在 10-04 里、该文件本轮不冲突，所以开关作为合并后补的 独立 commit，与 Q21 的 A100 补丁同样便于单独回退 - 单测：512×288 的参考视频，关闭时断言返回画布尺寸、打开时断言返回输入对齐 32 - §3 保留清单加一行，让 §6.1 L3 符号审计覆盖该开关 行号修正： - P2 的行号原取自 origin/main，改为起点分支（reference_video.py 调用点 523、 prepare_reference_videos 483、pipeline _prepare_reference_videos 2377、 分派 2388），并保留一行 origin/main 的对照 - VDN 命名隔离那节：我方 pipeline 行号从 194/1730 改为起点分支的 207/1756 - 头部新增"行号约定"：一律取自起点分支，引用上游/10-04/现网现状时分别取 upstream/main、merge/upstream-20261004、origin/main，并逐处标明 Signed-off-by: reputationly <reputationly@gmail.com>
reputationly
pushed a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 12, 2026
1. 上游测试冲突（不写清楚，§6.2 的上游测试必红）： - vllm-project#8253 在 test_minimax_h3_contract.py 加了 test_reference_video_shape_caps_large_inputs_without_enlarging_small_inputs （起点分支 1773-1792），其中 (640,384)->(640,384)、 (384,640)->(384,640) 断言"不放大"，我们默认放大必然失败 - 处理：该测试改为在 monkeypatch.setenv("VLLM_OMNI_H3_REF_VIDEO_NO_UPSCALE","1") 下跑，原有断言不动；另新增开关关闭时的用例（512x288 被放大到画布） - test_minimax_h3_official_contract.py:61-72 也直接调 _reference_video_shape， 两种开关状态都要跑 - 改动登记为"有意偏离上游"；§6.2 同步加了这条例外说明 2. 读取方式修正：原写"参照 vdn_hybrid.py"，但该文件 os.environ/getenv 为 0 处， 引错了。改为与参考图开关同形：常量 MINIMAX_H3_REFERENCE_VIDEO_NO_UPSCALE_ENV 放 reference_video.py，解析复用 reference_image_geometry.parse_reference_image_no_upscale（108 行），在 _reference_video_shape 调用时现读（参照 pipeline_minimax_h3.py:295 与 strategy.py:550）。明确不要用模块级常量缓存（否则 monkeypatch 不生效）， 且不改函数签名 (width, height) 3. 单测位置：改到 tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py， 紧挨上游用例；stage_input_processors 那份不测几何 另：理由里引用了 pipeline_minimax_h3.py:288-294 我方早先的注释（"参考视频的 768 同样会放大小视频……需要画质 A/B 才能决定是否翻默认值"），支持本轮保持默认 Signed-off-by: reputationly <reputationly@gmail.com>
reputationly
pushed a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 12, 2026
手册 §5.0 P2 的落地。上游 vllm-project#8253 除了参考图，还改了 reference_video.py:_reference_video_shape：输入面积小于画布时不再放大。该文件 本轮不冲突、改动自动进树；已核实这条路径现网会走到 （reference_video.py:523 调用、prepare_reference_videos 483、 pipeline _prepare_reference_videos 2377 按 reference_video_decode_mode 分派， 现网为 legacy_h264_intermediate）。参考图的 reference_image_shape 回调管不到 视频路径。 做法（与参考图开关同形）： - 常量 MINIMAX_H3_REFERENCE_VIDEO_NO_UPSCALE_ENV 放在 reference_video.py - 解析复用 reference_image_geometry.parse_reference_image_no_upscale - 在 _reference_video_shape 内调用时现读 os.environ，不用模块级缓存 （否则 monkeypatch.setenv 不生效） - 不改函数签名 (width, height) - 默认关闭 = 保持现网的"放大到画布"；打开才走 vllm-project#8253 的不放大 测试： - 上游 test_reference_video_shape_caps_large_inputs_without_enlarging_small_inputs 改为在 monkeypatch.setenv(...,"1") 下跑，原有断言一律不动 - 新增 test_reference_video_no_upscale_switch_defaults_to_enlarging： 512x288 关闭->(1344,768)、打开->(512,288)，另含 640x384 与三个大尺寸 - test_minimax_h3_official_contract.py 的 test_reference_video_canvas_matches_official 参数化为两种开关状态都跑 - 逐条验证：上游用例开关打开 9/9 通过；我方用例两态 5/5 通过 这是对上游的有意偏离，已登记进 merge 正文。 Signed-off-by: reputationly <reputationly@gmail.com>
reputationly
added a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 14, 2026
原 §6.3 H3 通过标准在同一句里既把「solver 切换」列为"已知会改 H3 数值路径"， 又规定"差异来自 solver 切换算不通过"。若前者成立，标准在构造上不可满足—— 而 solver 切换不是本轮解的冲突，是 §5.0 A4 主动采纳的（denoise_loop.py 在 继承清单里），即"规定必须这么做，又规定这么做就验收失败"。§5.0 A4 正文其实 写对了（sigma_curr/sigma_next 照旧传进 minimax_h3_euler_eta0_step），是汇总句 过度概括。 2026-10-14 真机复核把三类"已知会改数值"全部排除： - solver：euler 路径逐位重构。6 条 schedule（含两份现网档 vdn8）× 全部 step × 3 种 shape，torch.equal 全真；只有 res_multistep 改数值，而现网进不去 （vdnh3.py 拒绝非 euler sampler；pipeline 对 res_multistep + 固定蒸馏 schedule 直接报错）。 - flash_attn/fa：A100 上不改数值。utils/fa.py 纯新增，非 Blackwell 的 CUDA provider 链未动；backends/flash_attn.py 的 forward 改走 _normalize_flash_attention_metadata，12 组 metadata 的路由判定逐条一致， 唯一发散条件 extra["attention_mask_mode"] 全仓无 producer；FA4 分支在 A100 上 IS_FLASH_ATTN_4 恒为 False。VDN_WINDOW_ATTN 窗口分组本就绕开 forward。 - vllm-project#8253 P2：开关默认关闭，关闭时参考视频几何逐位复现。 据此改写： - §5.0 A4 结论句改为"euler 路径上是逐位重构，不改数值"，0-3 条按实测重写； 补上 A4 第 2 条的严格校验已被 6 条 schedule 实测通过。 - §6.3 H3 通过标准：三类"已知会改数值"改为"已全部排除"；排查顺序只剩 其余上游改动 → 基座算子；接受侧补入"attention 内核实现"，不合格侧收敛到 "我方的契约代码"（点名 §3 表那一行与 §5.3 第 39/61/325 行）。 - §6.2 单测条目标注已由 53a81c2 补上。 Signed-off-by: reputationly <reputationly@gmail.com> Signed-off-by: reputationly <197039020@qq.com>
reputationly
added a commit
to reputationly/vllm-omni
that referenced
this pull request
Oct 14, 2026
并入 2026-10-14 复核的后续结论： 1. §6.3 三类"已知会改数值"全部实测排除 - solver 切换：euler 路径逐位重构（见 aa72a0d 的同类证据） - flash_attn/fa：utils/fa.py 纯新增、非 Blackwell provider 链未动；backends/flash_attn.py 的 forward 路由判定 12 组 metadata 逐条一致，唯一发散条件 extra["attention_mask_mode"] 全仓无 producer；FA4 分支在 A100 上 IS_FLASH_ATTN_4 恒为 False；VDN_WINDOW_ATTN 的窗口 分组本就绕开 forward - vllm-project#8253 P2：开关默认关闭，关闭时几何逐位复现 会改数值的 res_multistep 现网进不去（vdnh3.py 拒绝非 euler sampler；pipeline 对 res_multistep + 固定蒸馏 schedule 报错）。手册已同步更正（main 的 9e66c4e）。 2. H3 全目录 A/B 第二期（补齐 workspace 后）：pristine 与 fixed 均为 93 failed / 1529 passed / 44 skipped，失败集合对称差为空——修复无回归。第一期的 96/1526 是我少拷 deploy-configs/ 与 benchmarks/ 造成的假失败，已在文档中更正。 3. 93 个失败收敛到 5 个根因，全部既存： - 导入路径写错 vllm_omni.diffusion.models.minimax_h3.reference_video，约 56 个 - 从 api_server 导入不存在的私有 helper（_order_with_separate_audio 实际在 video/generation/helpers.py），约 20 个 - 契约卫生类测试探测范围未覆盖 model_executor/ 下的 reader，2 个 - 测试侧陈旧与需联网，约 15 个 判据是失败所依赖的导入/声明/读取处/hop 清单在 HEAD 与 origin/main 逐字节相同。 Signed-off-by: reputationly <reputationly@gmail.com> Signed-off-by: reputationly <197039020@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref2VA keeps reference images at their input resolution and avoids enlarging small reference videos. Large reference videos retain the existing size limit: 1920×1152 becomes 1280×768, while 640×384 stays unchanged. Dimensions are aligned to 32 pixels. Existing API and CLI calls need no new settings.
8×SM120 (73,415 MiB/device), native BF16, eager TP2/SP4, Qwen TP8, VAE tiling, cuDNN dense attention. Output target: 1344×768, 15 seconds. Images: 1344×768 each. Every case uses the same 15-second audio reference and seed. PyTorch 2.13.0+cu132; vLLM 0.30.0.
Only original step 0 of the 50-step schedule runs. Times are medians of 3 warmed requests, taking the maximum CUDA-event elapsed time across all 8 ranks per request. Encoder and decode time are excluded. Q includes padding; KV is the valid attention prefix.