[CI][ROCm] Add and stabilize HunyuanImage3 nightly coverage

[haic0]

Purpose

Add non-blocking ROCm nightly coverage for HunyuanImage3 and make that coverage reliable on AMD hardware.

The nightly job runs tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py::test_hunyuan_image3_pixel_accuracy_offline with tencent/HunyuanImage-3.0-Instruct on four MI300 GPUs. It uses the full_model and rocm and MI325 and cards_4 selector, --run-level full_model, a fail-closed collect-only preflight, TORCH_SDPA, the platform-native automatic ROCm MoE backend, and uploads PNG/JUnit artifacts under artifacts/rocm-hunyuanimage3/.

ROCm stabilization

• Keep HunyuanImage3 GroupNorm on the PyTorch implementation on ROCm. AITER GroupNorm does not safely handle this model's FP16 activations with BF16 parameters under autocast and caused memory-access faults during VAE decode. Other ROCm VAE models continue to use AITER GroupNorm.
• Add focused coverage for both the Hunyuan exclusion and the unchanged non-Hunyuan path.
• Use a HunyuanImage3-only ROCm PSNR floor of 29.0. CUDA keeps its existing default/B200 thresholds and NPU keeps 26.0.

Validation

Qualified PR head: e52c180504c98126784d6b4f092cf17960c4ec20

• AMD/nightly build 13399: all 38 jobs passed with no hard or soft failures.
  • mi300_4: HunyuanImage3 Offline Pixel Accuracy passed without retry.
  • mean absolute difference: 0.005328
  • p99 absolute difference: 0.047059
  • SSIM: 0.992198
  • PSNR: 37.103355 dB (ROCm threshold: 29.0 dB)
  • Uploaded hunyuanimage3-pytest.xml and vllm_omni_offline.png.
• CUDA build 16799: all 20 jobs passed with no hard or soft failures.
  • HunyuanImage3-DIT · Accuracy Test passed 3 tests without retry.
• GitHub Python 3.11/3.12 wheel builds, pre-commit, docs, and DCO passed.
• Changed-file pre-commit passed locally, including Ruff, formatting, mypy, CI marks, SPDX, forbidden imports, and Buildkite pipeline validation.
• Rebase range-diff preserved all three commits exactly.

The PR was squash-merged as a0d915254b4815b085d6d713de73e3ea6c76a0be. Its stable patch ID matches the qualified PR diff exactly.

[vllm-omni-review-bot]

This PR appears to be related to model: HunyuanImage.

Model owners: @Bounty-hunter @yenuo26 @congw729

Routing: @Bounty-hunter via model owner; @yenuo26 via CI owner, CODEOWNERS; @congw729 via CODEOWNERS

@haic0, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[haic0]

Self-review

Updated after the true rebase. I reviewed every final changed line on exact head 9fc31c15fa78361b35eca67d429991de86f5883f against frozen upstream main target aad30908772a8188c17997341d62768dc0bab3b7.

• History is linear: one commit, no merge commits, and the commit parent is the exact target base.
• The old and final net diffs have identical stable patch IDs. The only upstream overlap was current main's removal of the default flashinfer_cutlass MoE backend; that removal is preserved, while the PR's ROCm-only auto override remains.
• The job remains four-GPU MI300, non-blocking, 180 minutes, exact-node/fail-closed collection, TORCH_SDPA, and checkout-relative PNG/JUnit staging.
• Validation: isolated AMD pipeline/suite tests 27 passed; changed-file pre-commit passed, including Buildkite schema; diff check, Unicode format-control scan, and exact DCO/AI trailers passed.
• No new model-specific example, code-quality fragility pattern, or evidence-backed simplification issue was found in the diff.

Residual validation is an exact-head MI300 model run. The local environment lacks vLLM, so exact model collection and GPU execution were not rerun.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Resolved as of 63cf3e2c64bd: the high-priority or low-quality signal noted on an earlier commit no longer applies.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@haic0 this pull request has had no human commit, comment or review since 2026-09-22. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[haic0]

Addressed both yenuo26 review threads in 63cf3e2c:

• removed the HunyuanImage3-specific pipeline UT and all job-specific assertions; tests/buildkite/test_amd_pipeline.py now matches current main;
• removed the redundant source depends_on; native MiniJinja rendering still emits depends_on: amd-build;
• aligned with [CI][ROCm] Add Cosmos3 nightly coverage #7933 command conventions by staging both the PNG and JUnit XML;
• merged current main to clear the branch conflict while preserving the exact node, fail-closed collection, marker/run-level, MI300 four-GPU request, and non-blocking behavior.

Validation: native MiniJinja 2.3.1 render plus YAML semantic checks passed; generic AMD pipeline suite passed (7 tests); changed-file pre-commit and git diff --check upstream/main...HEAD passed. Exact model collection was attempted but this lightweight .venv lacks PyTorch. Both review threads were replied to and resolved.

[yenuo26]

LGTM