Skip to content

[Bugfix][ROCm] Skip Qwen-Image fused QK-norm assertions on HIP - #9229

Merged
andyluo7 merged 2 commits into
vllm-project:mainfrom
haic0:bugfix/qwen-image-fused-qk-rocm-none
Oct 21, 2026
Merged

andyluo7 merged 2 commits into
vllm-project:mainfrom
haic0:bugfix/qwen-image-fused-qk-rocm-none

Conversation

@haic0

@haic0 haic0 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

#7594 gates the fused kernel with current_platform.is_cuda(), which is false on ROCm. The new tests used torch.cuda.is_available(), which is true on HIP. They ran on MI300 and expected a non-None CUDA table. AMD builds 14204, 14330, and 14334 failed those assertions. This is not a change to enable the fused kernel on ROCm.

Original Test Plan

vLLM Version: the version installed in this checkout's .venv

vLLM-Omni Commit: 857b05b52 on 6b35bca6ffff

Original Test Result

On this ROCm host (torch.version.hip set, current_platform.is_rocm() true):

.venv/bin/python -m pytest tests/diffusion/models/qwen_image/test_qwen_image_fused_qk_norm_rope.py -q --override-ini addopts= -k "joint_launch or packed_table or min_tokens"

3 passed, 5 skipped. The five NVIDIA fused-table cases skipped. test_packed_table_skipped_when_fused_path_unavailable and test_packed_table_is_none_on_rocm passed and both assert the ROCm table is None even when the token gate is open.

pre-commit and mypy-3.12 on the test file passed.

Maintainer follow-up and current-head validation

Current head 6be07c8a2c944e2a6b4ac85f3c546493bd0500d4 additionally applies the NVIDIA-only guard to every test whose assertion depends on the fused CUDA kernel and requires an available HIP GPU for the dedicated ROCm fallback test.

  • Full changed-ref pre-commit passed, including Ruff, formatting, mypy, test marks, SPDX, forbidden imports, and the torch.cuda policy.
  • All GitHub checks passed: wheel builds for Python 3.11/3.12, pre-commit, CodeQL, DCO, and Read the Docs.
  • Exact synthetic merge 0e3a3570b35b1a5c7f371ad5a448afb2ba64fe76 has parents 6b35bca6ffff3c6bee4723984704fa39c9ac2af0 and 6be07c8a2c944e2a6b4ac85f3c546493bd0500d4.
  • AMD #14336 passed all 32 jobs with no retries or soft failures. The critical mi300_1: Diffusion · Model Test job reported 105 passed and 102 skipped; the Qwen-Image file contributed 5 passes and 16 intentional NVIDIA-only skips, with none of the six prior ROCm failures.
  • CUDA #17775 passed 18/18 jobs, Intel #10546 passed 2/2, and NPU #9510 passed 21/21, all without retries or soft failures.

torch.cuda.is_available() is true on ROCm, but the fused kernel
requires NVIDIA CUDA and returns None. Assert that fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
@haic0
haic0 requested review from NickCao and yenuo26 as code owners October 21, 2026 20:30
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-21T20:30:28.776575Z
We couldn’t complete this review. Comment @codex review to try again.
857b05b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to be related to model: qwen-image.

Model owners: @NumberWan @NickCao @yenuo26

Routing: @NumberWan via semantic router, model owner; @NickCao via CODEOWNERS; @yenuo26 via CODEOWNERS

@haic0, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

Signed-off-by: andyluo7 <andy.luo@amd.com>
@andyluo7 andyluo7 added ready label to trigger buildkite CI ROCm PR related to AMD hardware bug Something isn't working diffusion codes related to diffusion models CI/CD codes related to changes to CI/CD labels Oct 21, 2026

@andyluo7 andyluo7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at exact head 6be07c8. The ROCm guards now cover every NVIDIA fused-path assertion while preserving CPU and cross-platform fallback coverage. All GitHub checks and exact synthetic-merge Buildkite pipelines passed with no retries or soft failures; AMD #14336 also confirms the prior Qwen-Image ROCm failures are gone.

@andyluo7
andyluo7 merged commit c04bef8 into vllm-project:main Oct 21, 2026
11 checks passed
lishunyang12 pushed a commit that referenced this pull request Oct 23, 2026
 #8798 (#9251)

Signed-off-by: aayc123 <2803023760@qq.com>
Signed-off-by: aayc123 <115020989+aayc123@users.noreply.github.com>
Signed-off-by: 曾妮 <2803023760@qq.com>
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: caoshijie <shijiecao4@gmail.com>
Signed-off-by: wuli666 <421774554@qq.com>
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: aayc123 <115020989+aayc123@users.noreply.github.com>
Co-authored-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Shijie_cao <shijiecao4@gmail.com>
Co-authored-by: dengyunyang <584797741@qq.com>
Co-authored-by: wuli666 <421774554@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: NATURE <wzliu@connect.hku.hk>
Co-authored-by: Iryna Boiko <iryna.boiko@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD codes related to changes to CI/CD diffusion codes related to diffusion models ready label to trigger buildkite CI ROCm PR related to AMD hardware

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants