Skip to content

fix: migrate GPU topology from rocm_smi to amd_smi (ROCm 10.1+) - #663

Merged
QizhouZhang97 merged 4 commits into
ROCm:mainfrom
srinivamd:fix/migrate-rocm-smi-to-amd-smi-v2
Sep 30, 2026
Merged

QizhouZhang97 merged 4 commits into
ROCm:mainfrom
srinivamd:fix/migrate-rocm-smi-to-amd-smi-v2

Conversation

@srinivamd

Copy link
Copy Markdown
Contributor

TheRock#6852 (merged 2026-08-31) removed rocm_smi_lib from ROCm 10.1+. This breaks MORI's GPU topology build (rocm_smi/rocm_smi.h not found).

Keep both code paths, version-guarded via ROCM_VERSION from rocm_version.h:

  • MORI_USE_AMDSMI=1 (ROCm >= 10.1): amd_smi API, socket/processor handles
  • MORI_USE_AMDSMI=0 (ROCm <= 10.0): rocm_smi API, flat device indices

Key API migration:

  • rsmi_init(0) → amdsmi_init(AMDSMI_INIT_AMD_GPUS)
  • rsmi_num_monitor_devices → amdsmi_get_socket/processor_handles
  • rsmi_dev_pci_id_get → amdsmi_get_gpu_device_bdf (BDF struct)
  • rsmi_is_P2P_accessible → amdsmi_topo_get_p2p_status (4 args)
  • RSMI_IO_LINK_TYPE → amdsmi_link_type_t
  • rsmi_status_string → amdsmi_status_code_to_string

Uses __has_include for rocm_version.h path portability.

Fixes #638

Motivation

MORI build fails on ROCm 10.1+ with fatal error: 'rocm_smi/rocm_smi.h' file not found.

ROCm/TheRock#6852 (merged 2026-08-31) disabled
rocm_smi_lib from ROCm 10.1+ builds. MORI's GPU topology code (gpu.cpp, gpu.hpp, check.hpp)
includes rocm_smi/rocm_smi.h and uses the rsmi_* API, which no longer ships with ROCm 10.1+.

Fixes #638

Technical Details

Version-guarded dual code using ROCM_VERSION_MAJOR/ROCM_VERSION_MINOR from rocm_version.h. Both old
and new code paths live side-by-side:

  • MORI_USE_AMDSMI=1 (ROCm >= 10.1): amdsmi_* API with socket/processor handle model
  • MORI_USE_AMDSMI=0 (ROCm <= 10.0): rsmi_* API with flat device indices (unchanged)

Key API migration:

rocm_smi amd_smi Notes
rsmi_init(0) amdsmi_init(AMDSMI_INIT_AMD_GPUS)
rsmi_num_monitor_devices(&n) amdsmi_get_socket_handles + amdsmi_get_processor_handles
Socket/processor handle model
rsmi_dev_pci_id_get(i, &id) amdsmi_get_gpu_device_bdf(handle, &bdf) BDF struct instead of
packed uint64
rsmi_is_P2P_accessible(i, j, &ok) amdsmi_topo_get_p2p_status(h_i, h_j, &type, &cap) 4 args;
returns link type + capability struct
RSMI_IO_LINK_TYPE amdsmi_link_type_t Enum type change
rsmi_status_string amdsmi_status_code_to_string

Uses __has_include for rocm_version.h path portability (<rocm-core/rocm_version.h> on ROCm 7.2+,
<rocm_version.h> fallback for older).

Files changed:

  • include/mori/application/topology/gpu.hpp — version guard, include switch, amdsmi_link_type_t type
    field
  • include/mori/application/utils/check.hpp — version guard, dual ROCM_SMI_CHECK macro
  • src/application/topology/gpu.cpp — dual Load() implementations with OpenAmdSmi/OpenRocmSmi

Test Plan

  • Build MORI from source on ROCm 10.1.0 nightly (TheRock build) — verifies MORI_USE_AMDSMI=1 path
  • Build MORI from source on ROCm 10.0.0 GA — verifies MORI_USE_AMDSMI=0 path (no regression)
  • Both tested as part of vllmsrc docker build (MORI is a build dependency of vLLM)

Test Result

  • ROCm 10.1.0 nightly (MORI_USE_AMDSMI=1): MORI build + import mori smoke test pass — CI run
    #34608518940
  • ROCm 10.0.0 GA (MORI_USE_AMDSMI=0): MORI build passes (downstream torchaudio ABI issue unrelated
    to this change)

Submission Checklist

@QizhouZhang97 QizhouZhang97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for resolving this issue. Please address the following issue before merging it

Comment thread src/application/topology/gpu.cpp Resolved

// Bootstrap-only helper: this macro terminates the process and should not be
// used in per-connection or per-transfer runtime paths.
#define SYSCALL_RETURN_ZERO(stmt) \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retain this comment

Comment thread src/application/topology/gpu.cpp Resolved
@srinivamd

Copy link
Copy Markdown
Contributor Author

@QizhouZhang97 — All three review comments from your Sep 16 review have been addressed in the latest commits:

  1. Loop all sockets (not just sockets[0]) — Fixed in 1a55e2d. amd_smi models one socket per GPU, so the code now iterates all sockets to collect processor handles from each.
  2. Guard socketCount == 0 — Fixed in 1a55e2d. Added early-exit with error message before accessing the sockets array.
  3. Restore "Bootstrap-only helper" comment on SYSCALL_RETURN_ZERO — Fixed in 1a55e2d.

CI status on latest commit (3017c3c):

Check Result
pre-commit (clang-format) passed
mori build (MI355X_AINIC) passed
mori build (MI300X_MLX) passed
mori intranode test (MI355X_AINIC) passed
mori intranode test (MI300X_MLX) passed
mori intranode test (ROCm 10, MI355X_AINIC) passed
mori JAX intranode test (MI355X_AINIC) passed
mori umbp unit test (MI355X_AINIC) passed
mori internode test (MI300X_MLX) passed
mori internode test (MI355X_AINIC) failed — HIP error no ROCm-capable device (runner infra, not code)
CCO unit test (MI355X_AINIC) failed — same MI355X_AINIC runner infra issue

The two failures are both on the MI355X_AINIC internode runner reporting "no ROCm-capable device is detected" — a GPU device-passthrough issue on that CI node, not related to this PR's code changes. All builds and all other test configurations passed.

Ready for re-review and approval when you get a chance. Thanks!

@QizhouZhang97

Copy link
Copy Markdown
Contributor

Hi, @srinivamd, could you rebase the latest main to retrigger CI? because I have made some changes to MI355x runner, I will merge it once CI pass

@srinivamd

Copy link
Copy Markdown
Contributor Author

@QizhouZhang97 — Friendly follow-up (10 days since fixes pushed). All three review items from your Sep 16
review are addressed:

  1. Loop all sockets (not just sockets[0]) — done in 1a55e2d
  2. Guard socketCount == 0 — done in 1a55e2d
  3. Restore "Bootstrap-only helper" comment — done in 1a55e2d

Pre-commit passes. The two CI failures (MI355X_AINIC internode + CCO) are "HIP error no ROCm-capable device" —
runner infra, not code. All other 9 MoRI CI jobs pass including ROCm 10 build.

This unblocks MORI on ROCm 10.1+ (TheRock#6852 removed rocm_smi_lib). Could you re-review when you get a
chance? Thanks!

TheRock#6852 (merged 2026-08-31) removed rocm_smi_lib from ROCm 10.1+.
This breaks MORI's GPU topology build (rocm_smi/rocm_smi.h not found).

Keep both code paths, version-guarded via ROCM_VERSION from rocm_version.h:
- MORI_USE_AMDSMI=1 (ROCm >= 10.1): amd_smi API, socket/processor handles
- MORI_USE_AMDSMI=0 (ROCm <= 10.0): rocm_smi API, flat device indices

Key API migration:
- rsmi_init(0) → amdsmi_init(AMDSMI_INIT_AMD_GPUS)
- rsmi_num_monitor_devices → amdsmi_get_socket/processor_handles
- rsmi_dev_pci_id_get → amdsmi_get_gpu_device_bdf (BDF struct)
- rsmi_is_P2P_accessible → amdsmi_topo_get_p2p_status (4 args)
- RSMI_IO_LINK_TYPE → amdsmi_link_type_t
- rsmi_status_string → amdsmi_status_code_to_string

Uses __has_include for rocm_version.h path portability.

Fixes ROCm#638
Formatting only — no logic changes:
- check.hpp: align ROCM_SMI_CHECK macro trailing backslashes
- gpu.cpp: compress candidates[] array, break long if-statement
… guard

1. gpu.cpp: Loop over ALL sockets to collect processor handles.
   amd_smi models one socket per GPU (not per host socket), so
   sockets[0] only sees 1 GPU on an 8-GPU box. Collect from all.
2. gpu.cpp: Guard socketCount == 0 before accessing sockets array.
3. check.hpp: Restore "Bootstrap-only helper" comment on SYSCALL_RETURN_ZERO.
@srinivamd
srinivamd force-pushed the fix/migrate-rocm-smi-to-amd-smi-v2 branch from 3017c3c to 9c6566b Compare September 29, 2026 21:10
@QizhouZhang97
QizhouZhang97 merged commit 2958fce into ROCm:main Sep 30, 2026
45 of 55 checks passed
junyyang-amd pushed a commit to ROCm/ATOM that referenced this pull request Nov 3, 2026
The amd-mori-nightly wheel is built against ROCm 7 and picks rocm_smi at
compile time (ROCm/mori#663), so it dlopens librocm_smi64, which the 10.1
SDK no longer ships: every MORI-backed model (DeepSeek-V4, dp-attention)
exits right after loading weights. Add a parallel build_mori stage
(BUILD_MORI=1 on rocm10.1) that compiles MORI against the SDK, where it
uses amd_smi, and link the unversioned libamd_smi.so it dlopens.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: Build fails on ROCm 10.x — rocm_smi/rocm_smi.h not found (replaced by amd_smi)

2 participants