[CI][ROCm] Fix deterministic AMD L3 blockers

[andyluo7]

Summary

• preserve the Qwen3-TTS Base advanced-model marker and run level by removing the nested shell quoting that truncated the pytest command
• give the Qwen3-Omni colocate sleep/abort regression a ROCm-only 2 GiB KV-cache budget instead of percentage-based over-allocation
• keep the CUDA test configuration unchanged and add focused CI/config regressions

Why

Recent AMD L3 main runs repeatedly fail these two blocking jobs. The TTS job falls back to core-model dummy weights because its nested single quotes terminate the outer bash command before --run-level reaches pytest. The colocate test loads about 64.5 GiB of weights, then percentage sizing attempts about 125 GiB of KV cache for an 8K-token control-plane test and fails during allocation.

Validation

• Buildkite YAML regression: 1 passed locally
• rendered Bash argument-flow check confirms marker advanced_model and cuda and run level advanced_model
• ROCm overlay materializes kv_cache_memory_bytes=2147483648 and removes percentage sizing only on ROCm
• Python compileall and Ruff 0.14.10 pass

Full project pytest is unavailable on the local macOS checkout because its Torch/vLLM environment is not installed. This draft therefore requires a real AMD L3 merge-test run before either blocker is considered resolved.

CI request

Please trigger the AMD merge-test/L3 suite on this exact head. The acceptance gate is that both Qwen3-TTS Base E2E Test and Qwen3-Omni colocate async complete successfully with real weights on MI300.

[andyluo7]

Self-review completed:

• Traced the AMD Jinja template: the YAML command list is joined into one TEST_COMMANDS shell, so separate export lines persist while avoiding the nested single-quote truncation. A Bash argument-flow check confirms pytest receives both the full advanced_model and cuda marker and --run-level advanced_model.
• Traced kv_cache_memory_bytes from the deploy overlay through stage engine_extras into the vLLM cache config. The 2 GiB cap is under platforms.rocm only; the focused config regression asserts CUDA retains gpu_memory_utilization=0.9 and receives no explicit KV cap.
• Ran the focused Buildkite YAML test, Python compileall, Ruff 0.14.10, YAML materialization, and shell argument-flow validation locally.

The remaining acceptance test must be the real AMD L3 suite on this exact head. @yenuo26 @hsliuustc0106, could one of you please add the merge-test and amd-test labels? I attempted it, but the fork author account does not have label permission.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR was classified as CI work.

CI owner: @yenuo26

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[andyluo7]

Author self-review (after CI-owner assignment): I verified the full diff against current main, confirmed the AMD Jinja renderer joins the separate export and pytest lines into one shell, reproduced the final pytest argv, and checked the ROCm-only cache override through the deploy projection with an explicit CUDA non-regression assertion. Static checks are green. The change still requires the requested real AMD L3 run before merge.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 0cbd91734887 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[andyluo7]

@tjtanaa, this follows the current AMD L3 layout from #6830 and targets the two deterministic failures seen after that merge. I cannot request you as a reviewer or apply labels from the fork account. Could you help add merge-test and amd-test so the exact head receives the full MI300 L3 run?

[hsliuustc0106]

L2 blockers: https://buildkite.com/vllm/vllm-omni-amd-ci/builds/11263/canvas?sid=01a05a6c-f3f7-4aeb-b936-9dc7299c4c3d&tab=output

[andyluo7]

Thanks. I checked build #11263: it ran on PR #6829 at d122a9b, not this PR head. Its only blocking job was Simple · Diffusion Test, where test_cross_attn_key_padding_vs_sdpa[40] and [256] hit BF16 max diff 0.015625 against the old strict <0.01 limit.

This PR head already contains #6830’s ROCm-specific comparison, and all four Simple Diffusion shards passed on MI300 in AMD build #11230. That build then failed on the Qwen3-Omni and Qwen3-TTS Base L3 jobs addressed by this PR.

Could you trigger a fresh L2 run and the AMD L3 merge-test on head 8ad4cd7a? The branch is currently three commits behind main; I can rebase first if you prefer.

[congw729]

The code changes look reasonable to me, and I did not find a blocking issue in the diff. The focused regressions cover both the Qwen3-TTS argument preservation and the ROCm-only KV-cache override while keeping CUDA unchanged.

Before approval, I would still like to see the AMD L3 merge test run on the current head 8ad4cd7, especially the Qwen3-TTS Base and Qwen3-Omni colocate-async jobs, since those are the behaviors this PR is intended to fix.

[andyluo7]

Thanks @congw729. The branch is now at 0cbd9173 after merging the latest main. AMD L2 on this exact head passed in build #11281.

I checked the bootstrap log: that run rendered test-amd-ready.yml, so it did not run Qwen3-TTS Base E2E Test or Omni Model Test Qwen3-Omni.

@hsliuustc0106 @yenuo26, could you share the Buildkite build number if the AMD L3 run is already registered, or trigger vllm-omni-amd-ci on 0cbd9173 with DEBUG_TEST_YAML=merge? I will verify those two target jobs and report the results here.