Add test_rocm_wheels.yml workflow for testing ROCm Python packages

[ScottTodd]

Motivation

Progress on #1559 which will build and test ROCm Python packages (and later PyTorch Python packages) as part of our CI workflows so we can catch ROCm packaging and framework build/test issues earlier in development.

Technical Details

This workflow is forked from .github/workflows/test_pytorch_wheels.yml. We might also want to run some of the system health steps from other workflows like .github/workflows/test_sanity_check.yml (maybe time to move those into reusable workflows or onto the runners as pre- and post- job scripts).

To start with, this is not integrated into any existing workflows. My plan is to use this from ci_linux.yml / ci_windows.yml once the build_portable_linux_python_packages.yml and build_windows_python_packages.yml workflows upload the packages they build to S3 / some index.

I also considered keeping the test code in the "build" workflow given how simple it is, but a separate workflow file like this is easier to share across Windows and Linux and makes it easier to test packages from arbitrary sources without first needing to build.

Alternately we could adopt the same "build and upload to staging -> test -> upload" setup as build_portable_linux_pytorch_wheels.yml. That would still be compatible with this being its own workflow file.

Test Plan

• Triggered test workflows in my fork using standard GitHub-hosted runners (no GPU)
• Will trigger test workflows here using our self-hosted runners after merge (or I can send a PR from a branch in this common repository once the changes are reviewed)

Test Result

• windows-2022, gfx110X-all, rocm 7.11.0a20260121 from https://rocm.nightlies.amd.com/v2: https://github.com/ScottTodd/TheRock/actions/runs/21374228045/job/61526284273
  • 25 tests with 1 skipped, 0 failures, 0 errors
  • 4 minutes total: 3 minutes downloading packages and 40 seconds running tests (includes devel package init)
• ubuntu 24.04, gfx1151, rocm 7.12.0a20260126 from https://rocm.nightlies.amd.com/v2: https://github.com/ScottTodd/TheRock/actions/runs/21374270003/job/61526430258
  • 25 tests with 1 failure, 2 errors (amd-smi, rocminfo, and rocm-smi tests all fail on systems with no AMD GPU)
  • 1m40s total: 1 minute downloading packages and 36 seconds running tests (includes devel package init)

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[marbre]

Suggested change

	run-name: Test ROCm Wheels (${{ inputs.amdgpu_family }}, ${{ inputs.rocm_version }}, ${{ inputs.test_runs_on }})
	run-name: Test ROCm Wheels (${{ inputs.amdgpu_family }}, ${{ inputs.rocm_version }}

Not sure we really need the runner name here or if we want it, we could drop the amdgpu_family.

[ScottTodd]

Copied from the style used for pytorch wheel tests:

TheRock/.github/workflows/test_pytorch_wheels.yml

Lines 64 to 68 in cf13cfd

	run-name: Test PyTorch (${{ inputs.amdgpu_family }}, ${{ inputs.torch_version}}, ${{ inputs.test_runs_on }})
	jobs:
	test_wheels:
	name: Test PyTorch | ${{ inputs.amdgpu_family }}

I think it's fine to include for workflow_dispatch:

Most use of this workflow once integrated will be using workflow_call

[marbre]

Well with the default for test_runs_on, we would get linux-mi325-1gpu-ossci-rocm-frac which already includes the GPU kind of (even though not the gfx architecture). Fine as is for me though.

[marbre]

I think this needs more work as we won't have a full index for artifacts. Fine to address in a follow up though but might want to add a todo here (if it isn't tracked in the issue).

[ScottTodd]

Yep, using this for CI / dev artifacts will need some extra work (changes to setup_venv.py and/or this workflow). I wanted to get the base workflow in place and tested using our existing dev/nightly packages.

[ScottTodd]

Testing on our self-hosted runners with GPUs now:

• Windows gfx1151: https://github.com/ROCm/TheRock/actions/runs/21376425109 (queued)
• Linux gfx942, version 7.11.0a20260121: https://github.com/ROCm/TheRock/actions/runs/21376381392/job/61533361931
• Linux gfx942, version 7.12.0a20260125: https://github.com/ROCm/TheRock/actions/runs/21376465608/job/61533625600

Both Linux runs failed with

testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) ... ++ Exec [/__w/TheRock/TheRock]$ /__w/TheRock/TheRock/.venv/bin/python -P -m rocm_sdk path --root
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libgfortran.so.5: cannot open shared object file: No such file or directory
  testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1.0')) ... ERROR
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libgfortran.so.5: cannot open shared object file: No such file or directory
  testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1')) ... ERROR
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libgfortran.so.5: cannot open shared object file: No such file or directory
  testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so')) ... ERROR
======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1.0'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/rocm_sdk/tests/devel_test.py", line 153, in testSharedLibrariesLoad
    subprocess.check_call(
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/__w/TheRock/TheRock/.venv/bin/python', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', '/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1.0']' returned non-zero exit status 1.
======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/rocm_sdk/tests/devel_test.py", line 153, in testSharedLibrariesLoad
    subprocess.check_call(
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/__w/TheRock/TheRock/.venv/bin/python', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', '/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so.1']' returned non-zero exit status 1.
======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=PosixPath('/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/rocm_sdk/tests/devel_test.py", line 153, in testSharedLibrariesLoad
    subprocess.check_call(
  File "/__w/_tool/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/__w/TheRock/TheRock/.venv/bin/python', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', '/__w/TheRock/TheRock/.venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/libhipsolver_fortran.so']' returned non-zero exit status 1.
----------------------------------------------------------------------
Ran 25 tests in 12.576s
FAILED (errors=3)

Looks similar to #1877 - I thought we fixed that? Is this a real issue or is the test workflow not configured correctly?

[ScottTodd]

I can reproduce this locally with our test dockerfile. Will follow-up with a new issue.