[CI] [ROCm] Bugfix device environment issue

[tjtanaa]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Bugfix the Inconsistent GPU visibility env vars: HIP_VISIBLE_DEVICES='0' vs CUDA_VISIBLE_DEVICES='1'. Please set only one, or ensure they match. environ issue that surfaced by CI https://buildkite.com/vllm/vllm-omni-amd-ci/builds/3312/steps/canvas?sid=019d004c-038a-43b4-88c7-72079cb9d004&tab=output .

In latest vLLM version, on ROCm we have to manage two environment variables (HIP_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES).

Introducing platform helper functions

class OmniPlatform(Platform):
    ...
    
    @classmethod
    def set_device_control_env_var(cls, devices: str | int | None) -> None:
        import os

        os.environ[cls.device_control_env_var] = devices

    @classmethod
    def unset_device_control_env_var(cls) -> None:
        import os

        os.environ.pop(cls.device_control_env_var, None)

Test Plan

Local test

export CUDA_VISIBLE_DEVICES=6,7
pytest -s -v tests/e2e/offline_inference/test_qwen2_5_omni.py
pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py
pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py

Manual Trigger CI Run

https://buildkite.com/vllm/vllm-omni-amd-ci/builds/3331/steps/canvas

Test Result

Local test results

===================== 2 passed, 4 warnings in 99.59s (0:01:39) =================

================== 1 passed, 3 warnings in 173.22s (0:02:53) =================================

================== 2 passed, 4 warnings in 221.36s (0:03:41) ===================

CI Run result

https://buildkite.com/vllm/vllm-omni-amd-ci/builds/3339/steps/canvas (Running)

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a45cc3dc1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Restore ROCm visibility from both env vars after stage launch

On ROCm this block only snapshots os.environ[device_control_env_var] before launch, but unset_device_control_env_var() now clears both HIP and CUDA. In environments that enter with only HIP_VISIBLE_DEVICES set (our AMD wrapper does this), previous_visible_devices is None, so the first stage launch drops the inherited HIP mask entirely. Any later stage/worker then inherits full-node visibility, and the multi-stage ROCm configs under tests/e2e/stage_configs/rocm/ get remapped against physical GPUs instead of the shard-local subset.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Map ROCm logical ids from HIP visibility before mirroring both vars

This assignment happens only after selected_physical has been derived from os.environ.get(env_var) above, and on ROCm device_control_env_var is still CUDA_VISIBLE_DEVICES. If the parent process only exports HIP_VISIBLE_DEVICES (for example .buildkite/scripts/hardware_ci/run-amd-test.sh), a stage configured as devices: "0" or "1" never sees the shard-local mapping and this line rewrites both env vars to the raw logical id. A container pinned to GPU 1 therefore re-exports HIP_VISIBLE_DEVICES=0, so its workers launch on the wrong physical GPU instead of the assigned shard.

Useful? React with 👍 / 👎.

[gcanlin]

Does ROCm also need CUDA_VISIBLE_DEVICES?

[tjtanaa]

@gcanlin yes. Some of the libraries like ray backend on vLLM under certain conditions it will find CUDA_VISIBLE_DEVICES instead of HIP_VISIBLE_DEVICES. We will try to find a better way to fix this on vLLM core.

In the latest vLLM platform code, they synced/set both CUDA_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES during platform module import.

[gcanlin]

LGTM