[ROCm][CI] Revamping AMD mirrors

[AndreasKaratzas]

In this PR, we are refining gating tests to utilize MI250 agents as well. I've put this PR up to start the evaluation of these tests and get tests ready for merging as soon as issues are confirmed to be resolved.

Mirrored AMD Tests

File	Test Label	AMD Device
basic_correctness.yaml	Basic Correctness	mi250_1
distributed.yaml	Distributed (2 GPUs)	mi250_2
distributed.yaml	Distributed Torchrun + Examples (4 GPUs)	mi250_4
distributed.yaml	Distributed Tests (2 GPUs)(H100-MI325)	mi250_2
engine.yaml	Engine (1 GPU)	mi325_1
engine.yaml	e2e Scheduling (1 GPU)	mi325_1
engine.yaml	e2e Core (1 GPU)	mi325_1
engine.yaml	V1 e2e (2 GPUs)	mi250_2
engine.yaml	V1 e2e (4 GPUs)	mi325_4
entrypoints.yaml	Entrypoints Integration (API Server 1)	mi325_1
entrypoints.yaml	Entrypoints Integration (API Server 2)	mi250_1
entrypoints.yaml	Entrypoints V1	mi250_1
lora.yaml	LoRA %N	mi250_1
misc.yaml	V1 Others	mi325_1
misc.yaml	Examples	mi250_1
models_basic.yaml	Basic Models Tests (Other)	mi250_1
models_language.yaml	Language Models Tests (Standard)	mi325_1
models_language.yaml	Language Models Tests (Extra Standard) %N	mi250_1
models_language.yaml	Language Models Test (Extended Generation)	mi325_1
models_language.yaml	Language Models Test (Extended Pooling)	mi250_1
models_multimodal.yaml	Multi-Modal Models (Standard) 1: qwen2	mi325_1
models_multimodal.yaml	Multi-Modal Models (Standard) 2: qwen3 + gemma	mi325_1
models_multimodal.yaml	Multi-Modal Models (Standard) 3: llava + qwen2_vl	mi325_1
models_multimodal.yaml	Multi-Modal Models (Standard) 4: other + whisper	mi325_1
models_multimodal.yaml	Multi-Modal Processor	mi325_1
models_multimodal.yaml	Multi-Modal Models (Extended) 1	mi325_1
models_multimodal.yaml	Multi-Modal Models (Extended) 2	mi250_1
plugins.yaml	Plugin Tests (2 GPUs)	mi250_2
quantization.yaml	Quantized Models Test	mi355_1
samplers.yaml	Samplers Test	mi250_1

Total mirrored AMD tests: 30

Device Breakdown

Device	Count
mi325_1	13
mi250_1	8
mi250_2	4
mi325_4	1
mi250_4	1
mi355_1	1

Dependencies:

• [ROCm][CI] Optimize ROCm Docker build: registry cache, DeepEP, and ci-bake script #36949
• [ROCm][CI] Add ROCm Docker Hub registry cache and weekly cleanup pipeline ci-infra#307
• [ROCm][CI] Cleaning and restructuring amd-ci legacy pipeline #34839
• [ROCm][CI] Retrying in case of batch variance effects and reducing flakiness #36442
• [CI] Stabilize multinode DP internal LB completion tests #36356
• [ROCm] Validate block_size for explicitly selected attention backends #36846
• [ROCm][CI] Retrying in case of batch variance effects and reducing flakiness #36442
• [Core][CI] Add opt-in media URL caching via VLLM_MEDIA_CACHE #37123
• [CI] Add persistent cache mounts and fix test download paths #36951

cc @kenroche

[gemini-code-assist]

Code Review

This pull request attempts to fix a quotation issue by stripping surrounding double quotes from a variable. However, the implementation introduces code duplication in two places within the re_quote_pytest_markers function. My review includes a suggestion to refactor this duplicated logic into a reusable shell function to improve code maintainability.

[gemini-code-assist]

This logic for stripping surrounding quotes is duplicated later in the script (lines 253-256). To improve maintainability and avoid code duplication, consider extracting this logic into a reusable shell function.

For example, you could define a function at the start of re_quote_pytest_markers:

_strip_surrounding_quotes() {
    local s=$1
    if [[ "$s" == '"'*'"' ]]; then
        s="${s#\"}"
        s="${s%\"}"
    fi
    echo "$s"
}

Then you can replace this block and the duplicated one with a call to this function:

marker_buf=$(_strip_surrounding_quotes "$marker_buf")

[AndreasKaratzas]

DRYing the code in this point is pointless. This script will and has to heavily change. We are simply putting some patches in place to accelerate the gating test process.

[AndreasKaratzas]

Manually cancelled the runs to save resources. Will re-enable ready label and CI runs when the aforementioned dependencies are resolved.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @AndreasKaratzas.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[AndreasKaratzas]

Observed 2 failures:

• AMD: Entrypoints V1 (mi250_1): Network issue on MI250. This is a problem that will be addressed in 3 PRs:
  • [ROCm][CI] Add ROCm Docker Hub registry cache and weekly cleanup pipeline ci-infra#307
  • [CI] Add persistent cache mounts and fix test download paths #36951
  • [Core][CI] Add opt-in media URL caching via VLLM_MEDIA_CACHE #37123
• AMD: Language Models Test (Extended Pooling) (mi250_1): Passes on MI325 so probably will revert it to MI325. Nonetheless, we are also working on amending the cascade effect where one failure results to all of the rest tests failing because the model that failed did not release the memory. Will be posting that PR here too.

The above comments were made based on: https://buildkite.com/vllm/ci/builds/56267/steps/canvas

[AndreasKaratzas]

Update on AMD: Language Models Test (Extended Pooling) (mi250_1)

Looks like it was an intermittent failure: https://buildkite.com/vllm/amd-ci/builds/6559/steps/canvas?sid=019cf53b-06a0-4731-aa2a-420b95f27459&tab=output

Will be monitoring in the upcoming nightlies as well as the upcoming re-evals of this PR.

[AndreasKaratzas]

I also put a PR up to stabilize the RemoteOpenAIServer class:

• [CI] Fix GPU memory leak when RemoteOpenAIServer fails to start in __init__ #37230

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @AndreasKaratzas.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork