Skip to content

fix(e2e): re-point HunyuanImage3 offline test to text_to_image.py - #4

Open
chethanuk wants to merge 108 commits into
mainfrom
fix/issue-7495-bug-nightly-ci-failed-testse2eaccuracytesthunyuanimage3pixelaccuracypytesthunyuanimage3pixelaccuracyoffline-cant
Open

chethanuk wants to merge 108 commits into
mainfrom
fix/issue-7495-bug-nightly-ci-failed-testse2eaccuracytesthunyuanimage3pixelaccuracypytesthunyuanimage3pixelaccuracyoffline-cant

Conversation

@chethanuk

Copy link
Copy Markdown
Owner

Purpose

Fixes nightly/CI failure tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py::test_hunyuan_image3_pixel_accuracy_offline — can't open file '.../hunyuan_image3/end2end.py': [Errno 2].

Root cause: commit a7ab91fb8 (vllm-project#5559) deleted examples/offline_inference/hunyuan_image3/end2end.py and migrated HunyuanImage-3.0 to the shared examples/offline_inference/text_to_image/text_to_image.py, but tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py:76 still invoked the deleted path, exiting with status 2 (/usr/bin/python3: can't open file ...).

Fix (one file, tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py):

  • _OFFLINE_SCRIPT re-pointed to examples/offline_inference/text_to_image/text_to_image.py.
  • _run_vllm_omni_hunyuan_image3_offline flag remap: --modality dropped, --prompts → --prompt, --steps → --num-inference-steps, --bot-task none + --sys-type en_unified → --extra-body '{"bot_task": null}' + --use-system-prompt en_unified + --trust-remote-code, and --output now a direct file path (no output_*.png glob/re-save). See text_to_image.py:106,128,140,323,364,383.

Fixes vllm-project#7495

Test Plan

vLLM Version: n/a

vLLM-Omni Commit: 0a857ad (this branch, 1 commit vs main)

Local (non-GPU):

  • python3 -m py_compile tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py — OK
  • git grep -n "hunyuan_image3.*end2end" — only tests/buildkite/test_vllm_omni_package_discovery.py:59 (end2end_probe.py fixture) remains
  • Flag existence check: --prompt :106, --output :128, --num-inference-steps :140, --extra-body :323, --use-system-prompt :364 (en_unified in choices), --trust-remote-code :383 in text_to_image.py

GPU (exact CI commands, per plan):

  • CUDA 4×H100 (.buildkite/cuda/test-nightly.yml:286): pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py -m full_model --run-level full_model (add -k offline for local)
  • NPU 4×A3 (.buildkite/npu/test-npu-nightly.yml:179): -m "full_model and A3" --run-level full_model

Note: old script let pipeline_hunyuan_image3.py:1235 build the generator, new script seeds via torch.Generator(device).manual_seed(seed) (text_to_image.py:489); seeding triage: if offline SSIM/PSNR regresses while ..._online passes, check noise seeding first before touching thresholds.

Test Result

  • py_compile OK, worktree/primary clean, single-file diff (10+, 14-).
  • No runtime on 4×H100/A3 in this env; full pytest -m full_model to be confirmed on CI.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.

Commit a7ab91f removed examples/offline_inference/hunyuan_image3/end2end.py
and moved HunyuanImage-3.0 to the shared text_to_image/text_to_image.py;
tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py still invoked
the deleted path, causing test_hunyuan_image3_pixel_accuracy_offline to
fail with No such file / exit 2.

Re-point _OFFLINE_SCRIPT and remap CLI flags per the shared script:
--modality dropped, --prompts->--prompt, --steps->--num-inference-steps,
--bot-task/--sys-type->--extra-body '{"bot_task": null}' /
--use-system-prompt en_unified plus --trust-remote-code, and write
directly to the file path instead of globbing output_*.png.

Fixes vllm-project#7495

Signed-off-by: ChethanUK <chethanuk@outlook.com>
@codeant-ai

codeant-ai Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Reviewed your PR 0a857ad Sep 14, 2026 · 14:27 14:31

@codeant-ai

codeant-ai Bot commented Sep 14, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@coderabbitai

coderabbitai Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 47ba8658-bac8-43d6-8e23-91a17dd6ba42

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 68631ebd-ef48-458c-a2ce-775bf06be22f

📥 Commits

Reviewing files that changed from the base of the PR and between 58adeec and 0a857ad.

📒 Files selected for processing (1)
  • tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The HunyuanImage-3 offline pixel-accuracy test now invokes the generic text-to-image example with updated CLI options. It passes system-prompt and request-body values, enables remote code, and validates the image at the specified output path.

Changes

HunyuanImage-3 accuracy test

Layer / File(s) Summary
Generic text-to-image runner integration
tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py
The test selects the generic offline text-to-image script and uses --prompt and --num-inference-steps.
Request options and output validation
tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py
The test replaces the previous bot-task and system-type flags, adds --trust-remote-code, and loads the image from the exact output path after checking that the path exists.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: yenuo26

Merge Risk: ⚪ Minimal · up to 0a857

The accuracy test now uses the supported shared runner interface and reads the exact image path it produces.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the HunyuanImage-3 offline test fix and specifies that it now uses text_to_image.py.
Description check ✅ Passed The description is directly related to the changeset. It explains the deleted script failure, the replacement script, the CLI updates, and the planned validation.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codeant-ai codeant-ai Bot added the size:S This PR changes 10-29 lines, ignoring generated files label Sep 14, 2026
@codeant-ai

codeant-ai Bot commented Sep 14, 2026

Copy link
Copy Markdown

User description

Purpose

Fixes nightly/CI failure tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py::test_hunyuan_image3_pixel_accuracy_offline — can't open file '.../hunyuan_image3/end2end.py': [Errno 2].

Root cause: commit a7ab91fb8 (vllm-project#5559) deleted examples/offline_inference/hunyuan_image3/end2end.py and migrated HunyuanImage-3.0 to the shared examples/offline_inference/text_to_image/text_to_image.py, but tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py:76 still invoked the deleted path, exiting with status 2 (/usr/bin/python3: can't open file ...).

Fix (one file, tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py):

  • _OFFLINE_SCRIPT re-pointed to examples/offline_inference/text_to_image/text_to_image.py.
  • _run_vllm_omni_hunyuan_image3_offline flag remap: --modality dropped, --prompts → --prompt, --steps → --num-inference-steps, --bot-task none + --sys-type en_unified → --extra-body '{"bot_task": null}' + --use-system-prompt en_unified + --trust-remote-code, and --output now a direct file path (no output_*.png glob/re-save). See text_to_image.py:106,128,140,323,364,383.

Fixes vllm-project#7495

Test Plan

vLLM Version: n/a

vLLM-Omni Commit: 0a857ad (this branch, 1 commit vs main)

Local (non-GPU):

  • python3 -m py_compile tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py — OK
  • git grep -n "hunyuan_image3.*end2end" — only tests/buildkite/test_vllm_omni_package_discovery.py:59 (end2end_probe.py fixture) remains
  • Flag existence check: --prompt :106, --output :128, --num-inference-steps :140, --extra-body :323, --use-system-prompt :364 (en_unified in choices), --trust-remote-code :383 in text_to_image.py

GPU (exact CI commands, per plan):

  • CUDA 4×H100 (.buildkite/cuda/test-nightly.yml:286): pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py -m full_model --run-level full_model (add -k offline for local)
  • NPU 4×A3 (.buildkite/npu/test-npu-nightly.yml:179): -m "full_model and A3" --run-level full_model

Note: old script let pipeline_hunyuan_image3.py:1235 build the generator, new script seeds via torch.Generator(device).manual_seed(seed) (text_to_image.py:489); seeding triage: if offline SSIM/PSNR regresses while ..._online passes, check noise seeding first before touching thresholds.

Test Result

  • py_compile OK, worktree/primary clean, single-file diff (10+, 14-).
  • No runtime on 4×H100/A3 in this env; full pytest -m full_model to be confirmed on CI.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.


CodeAnt-AI Description

Restore HunyuanImage-3 offline pixel-accuracy testing

What Changed

  • The offline accuracy test now uses the shared text-to-image example instead of the removed HunyuanImage-3 script.
  • Updated generation options to match the shared example, including prompt, inference-step, system-prompt, and model configuration.
  • Generated images are saved to and verified at the expected file path, allowing the test to compare them with the baseline image.

Impact

✅ Fewer nightly CI failures
✅ Restored HunyuanImage-3 offline accuracy coverage
✅ Reliable generated-image validation

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

andyluo7 and others added 23 commits September 15, 2026 13:24
…t#7472)

Signed-off-by: xuezhihao <89690814+xuexueligao@users.noreply.github.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: dongbo910220 <1275604947@qq.com>
…oject#7434)

Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
…7499)

Signed-off-by: Mengjie Zhao <zmj0129@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Yuliya Zhautouskaya <yzhautouskay@nvidia.com>
Signed-off-by: dengchangxiong <dengchangxiong1@huawei.com>
Co-authored-by: dengchangxiong <dengchangxiong1@huawei.com>
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
Signed-off-by: liuyao0322 <yaoliu548926@gmail.com>
Signed-off-by: Clodagh Walsh <clodaghwalsh17@gmail.com>
Signed-off-by: HAAZZZEEEE <wang-taicheng@qq.com>
…uts (vllm-project#7608)

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
…lete (vllm-project#6759)

Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: wuli666 <421774554@qq.com>
Co-authored-by: wuli666 <421774554@qq.com>
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
david6666666 and others added 19 commits September 18, 2026 15:16
…eights straight into host memory under DLOFix/mxfp online offload after quant (vllm-project#7009)

Signed-off-by: brandneway <gyuan4892@gmail.com>
…when weights are on CPU (vllm-project#7590)

Signed-off-by: holykie <116775684+holykie@users.noreply.github.com>
Signed-off-by: chickeyton <ngton2014@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
…l_config (vllm-project#7786)

Signed-off-by: zwhzzz0821 <2831474076@qq.com>
…rent issues (vllm-project#7755)

Signed-off-by: wangyu <410167048@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…vLLM normalization (vllm-project#7742)

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
…g sampling check (vllm-project#7803)

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
…m-project#7777)

Signed-off-by: Joshna Medisetty <joshna.medisetty@intel.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…m-project#7134)

Signed-off-by: levius <2114377220@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
…oject#7808)

Signed-off-by: Joshna Medisetty <joshna.medisetty@intel.com>
…lan (vllm-project#7326)

Signed-off-by: specture724 <specture724@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
…ct#6654)

Signed-off-by: ZhengWG <zwg0606@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
…project#7628)

Signed-off-by: psv666 <2693925048@qq.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
…eaccuracytesthunyuanimage3pixelaccuracypytesthunyuanimage3pixelaccuracyoffline-cant
@codeant-ai

codeant-ai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Skipping CodeAnt AI review — this PR changes more than 100 files, which usually means a migration, codemod, or vendored drop. Line-level review on diffs this large produces duplicate findings on the same rewrite pattern and drowns out anything that actually matters.

If you still want a review, comment @codeant-ai : review. For better signal, consider splitting the PR into smaller chunks.

TaffyOfficial and others added 2 commits September 19, 2026 15:37
Signed-off-by: TaffyOfficial <2324465096@qq.com>
Signed-off-by: zuiho <2324465096@qq.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: TaffyOfficial <2324465096@qq.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Only construct AR inputs when the deployment contains an AR stage. Keep
the DiT-only accuracy request on the same string-prompt/bot_task=none path
as online serving, and explicitly enable expert parallelism in the shared
offline CLI so its default cannot override the deploy configuration.

Cover script existence, real CLI parsing and both DiT-only/AR+DiT request
construction. Patch only tokenizer loading in existing tests so model
imports can still access the rest of transformers.

Validation: 80 related unit tests and all applicable pre-commit hooks pass.
The new regressions fail twice against the original PR source as expected.
Both full accuracy cases pass on 4 L20X with a local Triton MoE override:
SSIM 0.987250, PSNR 35.653664 dB, with identical online/offline PNG hashes.
The exact nightly FlashInfer CUTLASS configuration cannot initialize on
L20X; H100/B200/A3 validation remains with CI. No thresholds were changed.

AI assistance: Codex inspected the review, implemented the code and tests,
and ran local validation.

Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
The latest main rejects unknown diffusion configuration fields. The shared
offline example still supplied mode="text-to-image", which has no consumer
and makes the HunyuanImage3 offline accuracy launch fail before loading.
Remove it and exercise the real ingress validator in the request-building
regression test instead of allowing every mocked Omni keyword silently.

Validation on main 698f716: 458 CPU tests passed, 2 skipped, 4 deselected;
all applicable pre-commit hooks passed. Four-card L20X/Triton offline
accuracy passed with SSIM 0.987251 and PSNR 35.653656 dB, matching the online
case and its PNG hash. Original accuracy thresholds remain unchanged.

AI assistance: Codex implemented and validated this compatibility fix.

Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
akshatvishu and others added 4 commits September 19, 2026 16:16
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
…oject#7826)

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
…eaccuracytesthunyuanimage3pixelaccuracypytesthunyuanimage3pixelaccuracyoffline-cant
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files

Projects

None yet