[Bugfix] Resolve #6931 hub FA3 on torch 2.13 via kernels 0.16.1

[NumberWan]

Body

Why this broke (root cause)

Nightly CUDA images run torch 2.13 + CUDA 13. Hub FA3 is loaded by kernels.get_kernel("kernels-community/flash-attn3"), which does not pick “the best FA3 kernel”. It builds an exact folder name from the local environment, e.g.:

torch213-cxx11-cu130-x86_64-linux

and downloads only that build/ variant.

kernels-community/flash-attn3 v1/build currently has:

• exact-version wheels through torch212-cxx11-cu130-… (no torch213-…)
• a cross-torch wheel torch-stable-abi29-cu130-x86_64-linux (LibTorch stable ABI 2.9+, intended to run on 2.13)

kernels==0.14.1 / 0.15.2 only match exact torch{major}{minor}-cxx11-… tags. They do not treat torch-stable-abi* as a candidate, so get_kernel fails with Cannot find a build variant for this system even though the compatible ABI29 build is already on the Hub.

kernels>=0.16.0 adds that matching: on torch 2.13 / cu130 it selects torch-stable-abi29-cu130-x86_64-linux. This is the same failure @zhtmike reported in #6971 (and the attention path behind #6931).

This is not “torch 2.13 cannot run FA3”. The algorithm is there; the client pin cannot see the published ABI-stable wheel.

Why #6931 looked like an SSIM flake (and why it is still a real bug)

After the hub miss, tests/e2e/accuracy/test_qwen_image.py probed with _flash_attn3_hub_available() and silently matched both Omni and Diffusers onto torch SDPA, then used a second gate SSIM ≥ 0.96.

That is the wrong comparison. The accuracy test is meant to pin FA3 hub vs FA3 hub. With --fa-deterministic on H100 that path historically landed on a stable SSIM (around 0.958, gate ≥ 0.94). A real Omni/Diffusers drift would move that number, which is the signal we want.

SDPA is not that invariant: it has run-to-run scatter, so the same SHAs can sit above or below 0.96. Nightly got 0.958075, expected >= 0.960000 is that noisy SDPA pair against a gate calibrated on one high night (~0.9687, build 2954). The job looks flaky; the actual defect is we are no longer measuring FA3.

Hub miss also does not fail the test or an explicit hub backend, so training / serving that asked for FA3 can degrade to native attention with no hard error (#6971). This PR treats that as a failure, not a fallback.

What this PR changes

1. Pins: kernels==0.16.1; transformers >= 5.13.0, < 5.15.
   • 0.16.1 can select torch-stable-abi29-cu130-*.
   • transformers < 5.13 builds LayerRepository without version=/revision=; kernels 0.16 requires one of those at import time (import vllm dies). Floor 5.13 is the companion from [Bug]: kernels==0.14.1 pin cannot select the stable-ABI FA3 Hub build on torch 2.13 / CUDA 13 — silent fallback to native attention #6971.
2. Loud miss: explicit FLASH_ATTN_HUB / FLASH_ATTN_3_HUB resolve the Hub build at backend selection. Variant miss raises; no silent native/SDPA degrade (per @zhtmike).
3. Qwen-Image accuracy: delete _flash_attn3_hub_available, the SDPA fallback, and the second threshold. Diffusers always set_attention_backend("_flash_3_hub"). One gate: SSIM ≥ 0.94. If hub FA3 cannot load, the test fails instead of comparing two SDPA implementations.

Test plan

• test_explicit_kernels_hub_selection_does_not_fallback
• test_explicit_kernels_hub_variant_miss_does_not_fallback
• Local test_qwen_image_matches_diffusers (L20X, not H100): PASSED, SSIM 0.987344 / PSNR 34.32, gate 0.94, no using matched torch SDPA
• Nightly Diffusion X2I Accuracy on H100 only (@hardware_test(res={"cuda": "H100"})): FA3 hub vs FA3 hub, threshold>=0.940000, no SDPA fallback log.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/index.md.

Module owners: @Isotr0py @princepride @SamitHuang @wtomin @ZJY0516 @RuixiangMa @david6666666 @xuechendi

@NumberWan, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[NumberWan]

Root cause vs pin: torch 2.13/cu130 has no torch213-cxx11-cu130 in hugging face

1. Update kernel to let the test be able to auto-select the stable version FA3 on Hugging Face. (Details: [Bug]: kernels==0.14.1 pin cannot select the stable-ABI FA3 Hub build on torch 2.13 / CUDA 13 — silent fallback to native attention #6971)
2. Since after rebase, there are 2 set of threshold for 2different backend, but it is not the optimized way for the accuracy test, so we will remove the SPDA backend for the qwen-image accuracy test.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 4652c697ca60 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[NumberWan]

@yenuo26 @Champollion9012 @zhtmike
PTAL

[zhtmike]

@yenuo26 @Champollion9012 @zhtmike PTAL

LGTM

[Gaohan123]

LGTM. Thanks