[VincyZhang]

Overview:

Avoid running XPU cases currently blocked by the vLLM XPU v0.29.0 paged-decode kernel build, while leaving GPU coverage unchanged.

Details:

• Skip the affected multimodal serve profiles with the missing kernel tuple 8,128,16,false,false,false recorded in the reason. Paged decode kernel tuple not compiled for this configuration.
• Remove xpu_1 selection from the MM router overlap and Omni audio tests. The Omni audio case is a separate XPU startup issue due to lack of KV cache memory, should be enabled as two-cards test for XPU suite.

Where should the reviewer start?

Start with tests/serve/multimodal_profiles/vllm_xpu.py for the affected profile markers, then review tests/mm_router/test_vllm_mm_router_e2e.py and tests/serve/test_vllm_omni.py for the XPU lane selection changes.

Related Issues

Part of #15417

Summary by CodeRabbit

• Tests
  • Multimodal routing end-to-end coverage now runs on GPU rather than XPU.
  • Selected XPU multimodal profile checks are skipped where the required decode capability is unavailable.
  • Audio test coverage remains enabled for GPU and pre-merge runs, but is no longer selected for XPU.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dynamo-ops]

/ok to test f932e78

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4146a68a-ee49-445e-9f2c-01aac4e43630

📥 Commits

Reviewing files that changed from the base of the PR and between 8c226c9 and f932e78.

📒 Files selected for processing (3)

• tests/mm_router/test_vllm_mm_router_e2e.py
• tests/serve/multimodal_profiles/vllm_xpu.py
• tests/serve/test_vllm_omni.py

💤 Files with no reviewable changes (2)

• tests/mm_router/test_vllm_mm_router_e2e.py
• tests/serve/test_vllm_omni.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The changes add kernel-related skips to five multimodal profiles and remove XPU markers from two test configurations.

Changes

vLLM XPU test configuration

Layer / File(s)	Summary
Skip profiles missing the paged decode kernel tests/serve/multimodal_profiles/vllm_xpu.py	Adds a skip for the missing vLLM XPU v0.29.0 paged decode kernel to five multimodal profiles. Existing post-merge and XPU markers remain where present.
Adjust XPU test markers tests/mm_router/test_vllm_mm_router_e2e.py, tests/serve/test_vllm_omni.py	Removes the XPU marker from the multimodal router end-to-end test and the omni_audio test configuration.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: ⚪ Minimal · up to f932e

The change avoids running XPU cases blocked by the current kernel and startup limitations while preserving the intended GPU selections. No material merge-blocking risk remains.

Pre-merge checks · 5

Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly identifies the main change: skipping XPU test cases blocked by a missing paged decode kernel.
Description check		The description includes all required sections, explains the affected profiles and test-selection changes, identifies review starting points, and references issue #15417.

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-ops]

/ok to test c5c8fba

[dmitry-tokarev-nv]

Approving c5c8fbaa26. The skip reason is correct, the PR-XPU lanes lose no case, and no CUDA lane changes. One P2 and one P3 stay open, inline:

• [P2] 13 XPU cases leave CI with no way back (tests/serve/multimodal_profiles/vllm_xpu.py:66).
• [P3] The XPU-only branch at tests/serve/test_vllm_omni.py:317-321 no longer runs in CI.

What I measured.

• I collected all 66 lane expressions from the workflows at the merge base 3f46e3d3c5 and at this head. Exactly 13 cases change: 9 get the skip, and 4 lose xpu_1. The local XPU selection matched the CI shard lists in 8 of 8 comparisons.
• PR-XPU keeps all 475 + 3 cases, because its not multimodal already excludes all 13. Post-merge and nightly stop running 8 cases on 1 card and 5 cases on 2 cards.
• In both post-merge XPU jobs at the merge base (run 36670787900), these 13 cases were the only failures. The runs at 777977d5ee show the same. The nightly 1-card job also has 11 failures in etcd_ha and prompt_embeds that this PR does not touch.
• On CUDA, 0 of 60 lane expressions change their selection or their skips. At this head, vllm-runtime / Test cuda13.0, amd64 ran and passed the three test_vllm_mm_overlap_all cases and omni_audio.
• Not measured: I have no XPU hardware. The CI log does not show the root cause of the test_vllm_mm_overlap_all failure on XPU.

[dynamo-ops]

/ok to test a5e694e

Withdrawing my approval because new commits landed after it. I will review the new commits before I approve again.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: The XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch at lines 318-321 remains unreachable: omni_audio no longer has an XPU marker, so no CI lane can execute it. The new issue reference tracks re-enabling coverage but does not remove the dead branch or add the proposed two-card XPU case.
• Original discussion: The XPU-only omni_audio environment override at lines 318-321 remains reachable only when detect_target_device() == "xpu", but this PR removes the configuration's sole XPU marker, so no XPU CI lane exercises it.
• Original discussion: The omni_audio config still has no xpu_1 marker, while test_omni_serve_deployment still contains the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch for omni_audio. With no XPU CI selection for that parameter, the branch remains unreachable in the intended lanes.
• Original discussion: The unconditional XPU multimodal skips discussed there are still present on the profile marker blocks in tests/serve/multimodal_profiles/vllm_xpu.py, including the Qwen3-VL-2B agg and agg_router cases. They still collect skipped parameters rather than a conditional case that fails or re-enables when the missing paged-decode kernel is available; the added #15417 reference documents the blocker but does not make the skips self-expiring.
• Original discussion: Verified still present: removing omni_audio's xpu_1 marker leaves no selected XPU path for the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch at lines 318-321, while the branch remains in the test.
• Original discussion: Verified: omni_audio no longer has an XPU marker, so its XPU-only KV-cache override at lines 318-321 has no selected CI caller. The dead branch remains present.

[dynamo-ops]

/ok to test ede3f6f

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Verified still present: omni_audio no longer has an XPU marker, while test_omni_serve_deployment still keeps the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch for omni_audio. With no selected XPU CI parameter for that config, the branch remains unreachable in the intended lanes.
• Original discussion: Verified still present: removing omni_audio's xpu_1 marker leaves its XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch unreachable by CI, while the branch remains in test_omni_serve_deployment.
• Original discussion: Verified still present: removing omni_audio's only xpu_1 marker leaves the XPU-only KV-cache override in test_omni_serve_deployment without a selected XPU CI caller. The configuration is now GPU-only, while the branch at lines 318-321 remains gated on detect_target_device() == "xpu".
• Original discussion: Verified still present: removing omni_audio's sole xpu_1 marker leaves no selected XPU path for its XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch at lines 318-321. The added comment tracks a future two-card case, but the branch remains unexercised by CI.
• Original discussion: Verified still present: the XPU multimodal profile blocks in tests/serve/multimodal_profiles/vllm_xpu.py still use unconditional skip markers for the paged-decode-kernel blocker, including the Qwen3-VL-2B agg and router cases and the Qwen2/Qwen2.5 router cases. The #15417 reference documents the blocker but the collected parameters remain permanently skipped, so they still cannot fail or re-enable automatically when the kernel becomes available.

[VincyZhang]

Previously reported defects still present:

• Original discussion: The XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch at lines 318-321 remains unreachable: omni_audio no longer has an XPU marker, so no CI lane can execute it. The new issue reference tracks re-enabling coverage but does not remove the dead branch or add the proposed two-card XPU case.
• Original discussion: The XPU-only omni_audio environment override at lines 318-321 remains reachable only when detect_target_device() == "xpu", but this PR removes the configuration's sole XPU marker, so no XPU CI lane exercises it.
• Original discussion: The omni_audio config still has no xpu_1 marker, while test_omni_serve_deployment still contains the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch for omni_audio. With no XPU CI selection for that parameter, the branch remains unreachable in the intended lanes.
• Original discussion: The unconditional XPU multimodal skips discussed there are still present on the profile marker blocks in tests/serve/multimodal_profiles/vllm_xpu.py, including the Qwen3-VL-2B agg and agg_router cases. They still collect skipped parameters rather than a conditional case that fails or re-enables when the missing paged-decode kernel is available; the added Restore 13 XPU multimodal CI cases disabled by vLLM paged-decode and Omni startup issues #15417 reference documents the blocker but does not make the skips self-expiring.
• Original discussion: Verified still present: removing omni_audio's xpu_1 marker leaves no selected XPU path for the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch at lines 318-321, while the branch remains in the test.
• Original discussion: Verified: omni_audio no longer has an XPU marker, so its XPU-only KV-cache override at lines 318-321 has no selected CI caller. The dead branch remains present.

The six skip reasons and both XPU marker removal sites now reference #15417, which tracks restoring all 13 CI cases. These skips are not self-expiring; a vLLM version check would not establish that this particular kernel tuple is compiled in the XPU image. We will validate the suggested block-size alternative on XPU before changing the test configuration.

For omni_audio, removing xpu_1 removes it from XPU CI selection, but does not make the override unreachable for an explicitly run XPU test. We kept the existing override for that use. This does leave a CI coverage gap; #15417 tracks adding and validating a two-card XPU case.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: The six unconditional skip markers still remove the affected XPU multimodal and routing cases from execution until someone manually revisits #15417; the previously reported loss of XPU coverage remains present.
• Original discussion: Verified still present: omni_audio no longer has an XPU marker, while test_omni_serve_deployment still keeps the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch for omni_audio. With no selected XPU CI parameter for that config, the branch remains unreachable in the intended lanes.
• Original discussion: Verified still present: removing omni_audio's sole xpu_1 marker leaves the XPU-only _PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES branch in test_omni_serve_deployment without a selected XPU CI caller. The #15417 reference tracks future restoration but does not exercise or remove this branch.
• Original discussion: Verified still present: the XPU multimodal profile blocks still use unconditional skip markers for the paged-decode-kernel blocker, including the Qwen3-VL-2B agg and router cases and the Qwen2/Qwen2.5 router cases. The #15417 reference documents the blocker but the collected parameters remain permanently skipped, so they still cannot fail or re-enable automatically when the kernel becomes available.
• Original discussion: omni_audio no longer carries an XPU marker, while test_omni_serve_deployment retains its XPU-only KV-cache override. No selected XPU CI parameter exercises that branch, so the previously reported coverage gap remains present.

[dmitry-tokarev-nv]

Approving ede3f6fa4b. The new commit adds #15417 to the six skip reasons of this PR and to both xpu_1 removal sites. The lanes are the same as at c5c8fbaa26. No finding stays open.

What I checked.

• The P2 is fixed. The six new skip reasons and both removal sites name #15417, and #15417 lists all 13 cases. I resolved that thread.
• The P3 is closed. After your comment of 11:07Z, I withdrew my ask to delete the Omni branch. An explicit XPU run still reaches it, and #15417 tracks the CI case.
• I collected all 66 lane expressions at the merge base 3f46e3d3c5 and at this head. The same 13 cases leave the post-merge and nightly XPU lanes: 8 on one card and 5 on two cards. PR-XPU keeps its 475 + 3 cases, and 0 of 60 CUDA lanes change.
• This head merges into the current main without conflicts. Since the merge base, main changed no file that selects these lanes.
• CI at this head: pre-commit, Pre Merge, and PR-XPU passed. PR-XPU ran the same 475 + 3 cases that I collected. When I posted, PR was still running.
• Not measured: I have no XPU hardware.

[dynamo-ops]

/ok to test c66a26d

[devin-ai-integration]

✅ Dynamo PR CI passed — run 38354988312 (attempt 1) on c66a26d0ce

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 38354988312. Updated on every full-CI run of this PR.}

[devin-ai-integration]

External PR triage (2026-10-14)

Blocker: CI is green but the PR is still BLOCKED (REVIEW_REQUIRED). Full CI passed on c66a26d0c (run 38354988312). @dmitry-tokarev-nv approved ede3f6fa4, but that approval came before the 2026-10-13 push (a merge of main plus test(xpu): restore vLLM Omni audio coverage). Also, tests/mm_router/test_vllm_mm_router_e2e.py is owned only by @ai-dynamo/dynamo-multimodal-codeowners, and @dmitry-tokarev-nv is not in that team. So this PR needs an approval on the current head from a multimodal codeowner.

Next action (maintainer): a member of @ai-dynamo/dynamo-multimodal-codeowners, please review and approve c66a26d0c (test-only XPU skip changes). @dmitry-tokarev-nv, please re-approve the current head if your approval was dismissed as stale.

Note for @VincyZhang: Require an issue link is failing on this head, and the description only says "Part of #15417". Add a closing reference that the check accepts (advisory until 2026-10-21).

_{Automated triage of external contributions; this comment is edited in place on each run. Written by Devin}