[CI/Build][ROCm] Normalize SenseNova paged-decode hardware markers

[andyluo7]

Purpose

Current main now contains the focused vllm_flash_attn_only guard for the four SenseNova tests that exercise the bundled paged FlashAttention API (ea3cf998). This refresh rebases the PR onto current main, keeps that upstream guard as the single source of truth, and retains the remaining CI/test cleanup:

• replace the six touched direct cuda/L4 marks with hardware_test(res={"cuda": "L4", "rocm": "MI325"}) so CUDA and ROCm resource metadata are explicit;
• preserve the upstream guard split: the four paged-kernel tests use cuda_only plus vllm_flash_attn_only, while the two graph-only tests use only cuda_only and remain runnable on ROCm;
• type the recorder's positional and keyword captures so changed-file mypy stays clean.

There is no production-path change and no new backend, model-name, or device-SKU conditional. The prior duplicate guard and its follow-up review-fix commit were not replayed during the rebase because upstream now provides that behavior.

Test Plan and Results

vLLM-Omni Commit: 896f3b433fe3d66120d78b20e06137329901a5f1

Local validation on this exact head:

• pre-commit run --files tests/diffusion/models/sensenova_u1/test_sensenova_u1_paged_decode.py — passed all applicable hooks, including Ruff, formatting, mypy, CI-mark validation, forbidden imports, and the CUDA API guard;
• python3 -m compileall -q tests/diffusion/models/sensenova_u1/test_sensenova_u1_paged_decode.py — passed;
• python3 tools/pre_commit/check_test_marks.py tests/diffusion/models/sensenova_u1/test_sensenova_u1_paged_decode.py — passed;
• git diff --check origin/main...HEAD — passed;
• AST validation confirmed all six hardware decorators and the intended four paged-kernel versus two graph-only guard split.

Exact-head GitHub checks are green: pre-commit, Python 3.11 and 3.12 wheel builds, DCO, and docs all passed.

The local macOS Python environment does not contain PyTorch. The retained ready label also did not register new CUDA or AMD Buildkite contexts after the force-push, and this account cannot toggle repository labels or start Buildkite directly. Exact-head accelerator runtime validation is therefore still pending a maintainer re-trigger.

Prior hardware evidence established the intended backend behavior before this refresh, but is not being treated as exact-head validation:

• AMD #11318 Diffusion · Model Test: exactly four paged-kernel cases skipped and both graph-only cases passed (12 passed, 5 skipped); the full AMD build passed 19/19 jobs.
• CUDA #14554 Diffusion · Model Test: all six SenseNova accelerator cases passed (16 passed, 1 unrelated skip).

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR was classified as CI work.

CI owner: @yenuo26

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[andyluo7]

Self-review completed against base e51fe6ec.

• Confirmed the repeated AMD failures are the same four tests calling PagedDecodeCache.attend(), whose lazy import of vllm.vllm_flash_attn is unsupported on ROCm.
• Used the existing paged_decode_supported(device, head_dim) probe rather than checking a backend/model name. This also skips unsupported CUDA installations instead of assuming every CUDA wheel has the required paged arguments.
• Kept the two graph-only tests on the broader torch.cuda-compatible accelerator guard, so this change does not remove their ROCm coverage.
• Converted the touched direct SKU marks to hardware_test(...); CUDA L4 selection remains present and ROCm selection is explicit.
• Reviewed the full diff for broad exception handling, new Any, hot-path copies/synchronization, dead branches, and new examples; none were introduced.
• Local checks passed: complete changed-file pre-commit, compileall, and git diff --check.

The remaining proof must be on hardware. @yenuo26, could you please add the labels needed to run both ready and merge-test on this exact head? I will verify that AMD skips exactly the four paged-kernel cases while the two graph-only cases still execute, and that supported CUDA still executes the paged-kernel tests rather than skipping them.

[andyluo7]

@vllm-omni-review-bot please run an automated review of the current head 7557dad. Please focus on whether the capability-based skip is evaluated narrowly enough that supported CUDA still executes all four paged-kernel tests, while unsupported accelerator builds skip only those tests and retain the two graph-only tests. Also flag any backend/model hardcoding or unintended collection changes.

[andyluo7]

@yenuo26 one CI-routing clarification for the requested hardware proof: could you add amd-test, ready, and merge-test? The AMD PR bootstrap defaults to the ready/L2 YAML, so please also trigger an AMD build with DEBUG_TEST_YAML=merge,ready (or a separate DEBUG_TEST_YAML=merge run) on head 7557dad. That is needed to verify both the AMD L2/L3 behavior and the CUDA ready/merge selection on the same commit.

[andyluo7]

Rebased patch-identically onto current main (5d20f6b9); the new exact head is 32bea50a. The commit's stable patch ID is unchanged. Changed-file pre-commit, mypy, test-mark validation, compileall, and diff checks pass.

A parallel workaround on #4876 confirms the capability split matters: its AMD Diffusion Model job skipped all six accelerator tests under one shared marker, while its CUDA Diffusion Model job executed all six successfully. This PR keeps the capability guard on only the four paged-kernel tests and retains both graph-only tests on ROCm.

@yenuo26 @NickCao, could you please review this current head and add amd-test, ready, and merge-test? The AMD proof still needs a DEBUG_TEST_YAML=merge,ready (or separate merge) run so both relevant L2/L3 collections are visible.

[andyluo7]

Exact-head hardware validation is now complete for the capability split on 32bea50a:

• AMD build #11313, Diffusion · Model Test finished with exit 0. All six SenseNova cases were collected: exactly the four paged-flash-attention cases were skipped by the capability probe, while test_the_decode_graph_captures_into_the_shared_platform_pool and test_a_second_request_replays_the_first_capture both executed and passed. The job summary was 12 passed, 5 skipped; the fifth skip was the unrelated multi-GPU GLM test.
• CUDA build #14547, Diffusion · Model Test finished with exit 0. All four paged-kernel cases and both graph-only cases executed and passed. The job summary was 16 passed, 1 skipped; the sole skip was the unrelated multi-GPU GLM test.

This verifies that the guard is capability-based and narrowly scoped: it removes the unsupported ROCm paged-kernel failures without reducing supported-CUDA coverage or suppressing the ROCm graph-only tests. The aggregate CUDA build is red because the separate Tiny Model Diffusion job hit a Sana image-to-video HTTP 503; the changed-file target itself is green.

Pipeline-selection note: AMD test-amd-merge.yml does not run this SenseNova file; its similarly named Diffusion Model Test currently runs only tests/e2e/offline_inference/test_z_image.py. The relevant AMD coverage is therefore the ready-suite Diffusion · Model Test in #11313, which runs tests/diffusion/models/ and is terminal green above. A merge-suite override would not add coverage for this patch.

Aggregate completion: AMD #11313 subsequently finished all 19 jobs successfully in 58m29s. CUDA #14547 remained red only in the unrelated Tiny Model Tests · Diffusion (base) job, where SanaImageToVideoPipeline returned HTTP 503; the changed Diffusion · Model Test remained green.

[andyluo7]

@hsliuustc0106, could you please retry only Tiny Model Tests · Diffusion (base) in CUDA build #14547? The changed Diffusion · Model Test is green on the same head, and the failed Sana image-to-video case returned an unrelated HTTP 503. AMD #11313 has now completed all 19 jobs successfully.

[andyluo7]

Current-head target validation is complete on e0b212e1 after the upstream-main merge:

• AMD #11318, Diffusion · Model Test: 12 passed, 5 skipped; exactly the four unsupported paged-kernel cases skipped and both graph-only cases passed.
• CUDA #14554, Diffusion · Model Test: 16 passed, 1 skipped; all six SenseNova accelerator cases passed.

AMD #11318 is also terminal green across all 19 jobs (19/19 exit 0). The CUDA aggregate is red only in the unrelated Mooncake RDMA stress test (test_concurrent_put_get_integrity, MD5 mismatch); the changed-file target is green. The feature diff is unchanged, and current-head changed-file pre-commit also passes.

@hsliuustc0106, could you retry only the failing Distributed Test job in CUDA #14554? @yenuo26 @NickCao, the capability behavior is ready for human review.

[yenuo26]

@yenuo26 one CI-routing clarification for the requested hardware proof: could you add amd-test, ready, and merge-test? The AMD PR bootstrap defaults to the ready/L2 YAML, so please also trigger an AMD build with DEBUG_TEST_YAML=merge,ready (or a separate DEBUG_TEST_YAML=merge run) on head 7557dad. That is needed to verify both the AMD L2/L3 behavior and the CUDA ready/merge selection on the same commit.

Actually, both CUDA and NPU CI uniformly use labels such as ready, merge-test, and nightly-test to trigger different levels of CI in PRs. However, AMD CI currently does not support this. Could it be modified to adopt the same logic?

[yenuo26]

Current-head target validation is complete on e0b212e1 after the upstream-main merge:

• AMD #11318, Diffusion · Model Test: 12 passed, 5 skipped; exactly the four unsupported paged-kernel cases skipped and both graph-only cases passed.
• CUDA #14554, Diffusion · Model Test: 16 passed, 1 skipped; all six SenseNova accelerator cases passed.

AMD #11318 is also terminal green across all 19 jobs (19/19 exit 0). The CUDA aggregate is red only in the unrelated Mooncake RDMA stress test (test_concurrent_put_get_integrity, MD5 mismatch); the changed-file target is green. The feature diff is unchanged, and current-head changed-file pre-commit also passes.

@hsliuustc0106, could you retry only the failing Distributed Test job in CUDA #14554? @yenuo26 @NickCao, the capability behavior is ready for human review.

Just retried it.

[yenuo26]

Actually, I think this kind of skipping could cause CI to miss a lot of test cases that are directly skipped due to configuration errors. If CI doesn't turn red, we wouldn't actually go through each skipped case one by one to check whether the skip is justified. Perhaps failing the build directly would help us catch issues more effectively.

[andyluo7]

Good point. Addressed in 47f20cc2 by removing the runtime capability probe from the skip predicate. The four kernel-dependent tests now skip only when the active Omni platform is not CUDA. On CUDA they always execute, so a missing extension or incompatible paged API fails the test/build instead of becoming a silent skip. ROCm still skips only those four tests, while the two graph-only tests retain ROCm coverage. Current-head pre-commit, compileall, and diff checks pass; exact-head AMD and CUDA target reruns are still required.

[andyluo7]

@yenuo26 @hsliuustc0106 the review concern is addressed on exact head 47f20cc2: CUDA no longer skips when its paged-flash extension is missing or incompatible; only a non-CUDA Omni platform skips the four kernel-dependent cases. The two graph-only cases remain backend-generic.

The current ready label predates this head, and this account cannot toggle repository labels. Please remove and reapply ready once. The #6938 label event showed that this launches both CUDA ready and the current AMD L2 fallback; use amd-test only if the AMD context does not register.

The relevant target on both backends is Diffusion · Model Test. No AMD merge override is needed for this PR because the AMD merge suite does not collect the SenseNova file.

Once both exact-head targets are terminal, I will verify collection and per-test results. @yenuo26, please then confirm whether the review concern is satisfied; I am leaving the reviewer thread open for that confirmation.

[andyluo7]

@yenuo26 I implemented the AMD L2/L3 label-routing follow-up separately in #6966. It maps ready to L2, merge-test to L3, combines both labels behind one image build, preserves the existing amd-test-only L2 fallback, and leaves nightly-test explicitly pending until an AMD L4 suite exists. This keeps the routing change out of the SenseNova correctness PR.

[yenuo26]

@andyluo7 please resolve conflicts

[andyluo7]

Rebased onto current main (b1ec136b); the new exact head is 896f3b43.

The conflict came from main gaining the same functional guard in ea3cf998. I kept upstream's vllm_flash_attn_only as the single guard, did not replay the now-duplicate review-fix commit, and retained only the remaining test changes: six hardware_test(...) conversions plus the recorder typing cleanup. The final diff is one file and one signed commit.

Exact-head validation completed:

• changed-file pre-commit: passed all applicable hooks, including Ruff, formatting, mypy, test-mark validation, forbidden imports, and the CUDA API guard;
• compileall, the standalone test-mark checker, diff checks, and a structural 4-kernel/2-graph decorator check: passed;
• GitHub pre-commit, Python 3.11/3.12 wheel builds, DCO, and docs: all green.

The retained ready label did not register new CUDA or AMD Buildkite contexts after the force-push. This account has no label-triage permission or Buildkite API token, so I cannot safely retrigger them. @yenuo26 @hsliuustc0106, please remove and reapply ready once for exact-head accelerator validation. The relevant target remains Diffusion · Model Test: AMD should skip exactly the four paged-kernel cases and execute the two graph-only cases; CUDA should execute all six.

[yenuo26]

LGTM