fix(ck-tile): Prefer using amd-smi over rocm-smi by geyyer · Pull Request #9731 · ROCm/rocm-libraries

geyyer · 2026-07-22T21:37:36Z

Motivation

CK tooling invokes rocm-smi directly in many places for GPU discovery and info. amd-smi is the modern replacement and is preferred going forward, but call sites should not need to know which tool is present. This PR centralizes GPU SMI access behind a single set of wrappers that prefer amd-smi and transparently fall back to rocm-smi, so every call site gets consistent behavior with no duplicated detection logic.

Technical Details

Added tile_engine/ops/common/smi_utils.py as the single source of truth for SMI access: parsers for both tools plus wrappers detect_gpu_ids(), count_gpus(), show_gpu_info(), check_gpu_available(), and show_version().
Tool order is chosen by _smi_order(): amd-smi first, rocm-smi fallback if amd-smi is missing or fails. CK_SMI_TOOL=rocm-smi forces rocm-smi first (mainly for testing).
Added tile_engine/ops/common/smi_cli.py, an argparse CLI (list-ids, count, show-info, check, show-version) so shell scripts can reuse the same Python logic.
Added thin ck_smi_* delegates in script/tools/common.sh that call smi_cli.py (no parsing logic in Bash).
Migrated call sites off direct rocm-smi: gemm_full_benchmark.py now uses detect_gpu_ids(); generate_test_dataset.sh, ck-status, ck-start, ck-docker, ck-exec, ck-shell, ck-rocprof.md, and the GEMM README.md now use the wrappers.

Test Plan

python3 -m unittest tile_engine.ops.common.test_smi_utils -v live tests comparing rocm-smi vs amd-smi normalized fields (GPU IDs, product, gfx, driver) and verifying each wrapper against live output.
CK_SMI_TOOL=rocm-smi python3 -m unittest tile_engine.ops.common.test_smi_utils -v verifies the rocm-smi override path.
bash script/tools/test_ck_smi_helpers.sh pure-bash live comparison of the two tools' fields (no Python dependency).
Ran on a GPU host.

Test Result

All 11 unittest cases pass; the bash comparison reports all fields matching. Both the default (amd-smi first) and CK_SMI_TOOL=rocm-smi paths return identical GPU IDs.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

JIRA ID : AICK-1649

therock-pr-bot · 2026-07-22T21:39:38Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

therock-pr-bot · 2026-07-22T21:39:39Z

🎉 All checks passed! This PR is ready for review.

Copilot

Pull request overview

This PR centralizes Composable Kernel tile-engine GPU SMI interactions behind Python wrappers that prefer amd-smi and fall back to rocm-smi, and updates Python/Bash call sites to use the unified interface.

Changes:

Introduces smi_utils.py wrappers + smi_cli.py CLI to standardize GPU discovery/info across amd-smi/rocm-smi.
Updates existing Python and shell tooling to call the wrappers (via ck_smi_* Bash delegates) instead of invoking rocm-smi directly.
Adds live (GPU-host) Python unittest coverage and a pure-bash comparison script to validate normalized fields between tools.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
projects/composablekernel/tile_engine/ops/gemm/gemm_full_benchmark.py	Switches device detection to shared `detect_gpu_ids()` wrapper.
projects/composablekernel/tile_engine/ops/common/smi_utils.py	Adds centralized SMI parsing + wrapper functions with amd-smi-first fallback behavior.
projects/composablekernel/tile_engine/ops/common/smi_cli.py	Adds argparse-based CLI for reuse from shell scripts.
projects/composablekernel/tile_engine/ops/common/test_smi_utils.py	Adds live GPU-host unit tests comparing normalized fields and wrapper behavior.
projects/composablekernel/test_data/generate_test_dataset.sh	Replaces direct `rocm-smi` usage with `ck_smi_*` helpers and improves GPU checks/messages.
projects/composablekernel/script/tools/test_ck_smi_helpers.sh	Adds a pure-bash live equivalence test for rocm-smi vs amd-smi fields.
projects/composablekernel/script/tools/common.sh	Adds `ck_smi_*` helper functions and container GPU-info helper.
projects/composablekernel/script/tools/ck-status	Uses container GPU-info helper instead of hardcoded `rocm-smi` call.
projects/composablekernel/script/tools/ck-start	Uses container GPU-info helper instead of hardcoded `rocm-smi` call.
projects/composablekernel/script/tools/ck-shell	Updates help text example to use SMI wrapper helper.
projects/composablekernel/script/tools/ck-rocprof.md	Updates documentation to reference new SMI wrapper tooling.
projects/composablekernel/script/tools/ck-exec	Updates help text to reference SMI wrapper helper.
projects/composablekernel/script/tools/ck-docker	Uses container GPU-info helper instead of hardcoded `rocm-smi` call.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

andriy-ca

LGTM!

fix(ck-tile): Prefer using amd-smi over rocm-smi ## Motivation CK tooling invokes `rocm-smi` directly in many places for GPU discovery and info. `amd-smi` is the modern replacement and is preferred going forward, but call sites should not need to know which tool is present. This PR centralizes GPU SMI access behind a single set of wrappers that prefer `amd-smi` and transparently fall back to `rocm-smi`, so every call site gets consistent behavior with no duplicated detection logic. ## Technical Details - Added `tile_engine/ops/common/smi_utils.py` as the single source of truth for SMI access: parsers for both tools plus wrappers `detect_gpu_ids()`, `count_gpus()`, `show_gpu_info()`, `check_gpu_available()`, and `show_version()`. - Tool order is chosen by `_smi_order()`: `amd-smi` first, `rocm-smi` fallback if `amd-smi` is missing or fails. `CK_SMI_TOOL=rocm-smi` forces rocm-smi first (mainly for testing). - Added `tile_engine/ops/common/smi_cli.py`, an argparse CLI (`list-ids`, `count`, `show-info`, `check`, `show-version`) so shell scripts can reuse the same Python logic. - Added thin `ck_smi_*` delegates in `script/tools/common.sh` that call `smi_cli.py` (no parsing logic in Bash). - Migrated call sites off direct `rocm-smi`: `gemm_full_benchmark.py` now uses `detect_gpu_ids()`; `generate_test_dataset.sh`, `ck-status`, `ck-start`, `ck-docker`, `ck-exec`, `ck-shell`, `ck-rocprof.md`, and the GEMM `README.md` now use the wrappers. ## Test Plan - `python3 -m unittest tile_engine.ops.common.test_smi_utils -v` live tests comparing `rocm-smi` vs `amd-smi` normalized fields (GPU IDs, product, gfx, driver) and verifying each wrapper against live output. - `CK_SMI_TOOL=rocm-smi python3 -m unittest tile_engine.ops.common.test_smi_utils -v` verifies the rocm-smi override path. - `bash script/tools/test_ck_smi_helpers.sh` pure-bash live comparison of the two tools' fields (no Python dependency). - Ran on a GPU host. ## Test Result All 11 unittest cases pass; the bash comparison reports all fields matching. Both the default (amd-smi first) and `CK_SMI_TOOL=rocm-smi` paths return identical GPU IDs. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID : AICK-1649

detect_devices() in batched_gemm_full_benchmark.py called `rocm-smi --showid` directly for GPU id detection. Delegate to smi_utils.detect_gpu_ids() (added in #9731), which prefers amd-smi and falls back to rocm-smi, matching gemm_full_benchmark.py. Added common/ to sys.path, dropped unused `import re`. JIRA ID : ROCM-28734

…marks (#10204) ## Motivation PR #9731 centralized GPU SMI access behind `tile_engine/ops/common/smi_utils.py`, which prefers `amd-smi` and transparently falls back to `rocm-smi`, and migrated `gemm_full_benchmark.py` off direct `rocm-smi` calls. Three Tile Engine benchmark scripts were left calling `rocm-smi --showid` directly (with an `amd-smi list` fallback), so the migration is incomplete. Although wrapped in try/except today, these direct calls will break once `rocm-smi` is removed from future ROCm releases. This completes the migration for the remaining scripts, as requested by the tech lead and tracked in JIRA **ROCM-28734**. ## Technical Details Migrated the identical hand-rolled `detect_devices()` (env → `rocm-smi --showid` → `amd-smi list` → hardcoded `["0"]`) to a single `smi_utils.detect_gpu_ids()` call, mirroring the already-landed `gemm_full_benchmark.py`: - `tile_engine/ops/gemm/block_scale_gemm/gemm_bquant/gemm_bquant_full_benchmark.py` - `tile_engine/ops/gemm/gemm_multi_d_full_benchmark.py` - `tile_engine/ops/gemm/streamk_gemm_full_benchmark.py` Each script adds `common/` to `sys.path`, imports `detect_gpu_ids`, and drops the now-unused `import re`. Net -67 lines. ## Test Plan / Result Validated on an MI350 (gfx950) host, exercising the exact `sys.path` arithmetic + `smi_utils` import + `detect_gpu_ids()` for both the shallow (`gemm/`) and deep (`block_scale_gemm/gemm_bquant/`) script locations: - Default (amd-smi first): `detect_devices()` returns the live GPU ids; `count_gpus()`/`check_gpu_available()` consistent. - `CK_SMI_TOOL=rocm-smi` (forced fallback): identical result. Both `python3 -m py_compile` cleanly. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. https://amd-hub.atlassian.net/jira/software/c/projects/AICK/boards/4399?selectedIssue=ROCM-28734 JIRA ID : ROCM-28734 Co-authored-by: Muhammed Emin Ozturk <3836908+ozturkosu@users.noreply.github.com>

## Summary Extends the amd-smi-first migration (#9731, #10204) from the Tile Engine to the dispatcher's GPU-architecture detection. - Adds `detect_gpu_arch()` to the shared `tile_engine/ops/common/smi_utils.py` wrapper: amd-smi `static` (`TARGET_GRAPHICS_VERSION`) first, rocm-smi `--showproductname` (`GFX Version`) fallback — same amd-smi-first policy as the rest of the wrapper. - `dispatcher_common.detect_gpu_arch()`: prefers amd-smi via the wrapper, then rocminfo, then the caller-supplied default. Also removes a **duplicate** definition of this function. - `gemm_utils._get_arch()`: tries amd-smi via the wrapper before rocminfo, preserving the raise-on-unresolved (no silent default) behavior. ## Motivation Per the directive behind #9731, device-property access should prefer `amd-smi` (rocm-smi is being phased out). The Tile Engine was already migrated (#9731, #10204); the dispatcher still detected the GPU arch by shelling out to `rocminfo` directly. Tracked in JIRA **ROCM-28734**. ## Design note The dispatcher reaches the sibling `tile_engine/ops/common/smi_utils.py` through a guarded import and falls back to `rocminfo` if the wrapper is unavailable, so behavior is unchanged on hosts without amd-smi. (Open to moving the shared wrapper to a neutral location if reviewers prefer avoiding the dispatcher → tile_engine reach.) ## Test plan (MI350X / gfx950) - [x] New mock-based unit tests for `detect_gpu_arch()` (amd-smi preference, rocm-smi fallback, default) + a live check — 4/4 pass. - [x] `dispatcher_common.detect_gpu_arch()` and `gemm_utils._get_arch()` both return `gfx950` via amd-smi on the live host. - [x] `test_gemm_utils` 21/21 still pass (no arch-detection regression). --------- Co-authored-by: Muhammed Emin Ozturk <3836908+ozturkosu@users.noreply.github.com>

fix(ck-tile): Prefer using amd-smi over rocm-smi ## Motivation CK tooling invokes `rocm-smi` directly in many places for GPU discovery and info. `amd-smi` is the modern replacement and is preferred going forward, but call sites should not need to know which tool is present. This PR centralizes GPU SMI access behind a single set of wrappers that prefer `amd-smi` and transparently fall back to `rocm-smi`, so every call site gets consistent behavior with no duplicated detection logic. ## Technical Details - Added `tile_engine/ops/common/smi_utils.py` as the single source of truth for SMI access: parsers for both tools plus wrappers `detect_gpu_ids()`, `count_gpus()`, `show_gpu_info()`, `check_gpu_available()`, and `show_version()`. - Tool order is chosen by `_smi_order()`: `amd-smi` first, `rocm-smi` fallback if `amd-smi` is missing or fails. `CK_SMI_TOOL=rocm-smi` forces rocm-smi first (mainly for testing). - Added `tile_engine/ops/common/smi_cli.py`, an argparse CLI (`list-ids`, `count`, `show-info`, `check`, `show-version`) so shell scripts can reuse the same Python logic. - Added thin `ck_smi_*` delegates in `script/tools/common.sh` that call `smi_cli.py` (no parsing logic in Bash). - Migrated call sites off direct `rocm-smi`: `gemm_full_benchmark.py` now uses `detect_gpu_ids()`; `generate_test_dataset.sh`, `ck-status`, `ck-start`, `ck-docker`, `ck-exec`, `ck-shell`, `ck-rocprof.md`, and the GEMM `README.md` now use the wrappers. ## Test Plan - `python3 -m unittest tile_engine.ops.common.test_smi_utils -v` live tests comparing `rocm-smi` vs `amd-smi` normalized fields (GPU IDs, product, gfx, driver) and verifying each wrapper against live output. - `CK_SMI_TOOL=rocm-smi python3 -m unittest tile_engine.ops.common.test_smi_utils -v` verifies the rocm-smi override path. - `bash script/tools/test_ck_smi_helpers.sh` pure-bash live comparison of the two tools' fields (no Python dependency). - Ran on a GPU host. ## Test Result All 11 unittest cases pass; the bash comparison reports all fields matching. Both the default (amd-smi first) and `CK_SMI_TOOL=rocm-smi` paths return identical GPU IDs. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID : AICK-1649 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

geyyer and others added 8 commits July 17, 2026 14:59

Add a bash test

c6442c8

Add a python test

3ca4f6c

Add wrappers

fa5c47a

Update usage

48304e6

Minor fix

3779aed

Clean up

318ab48

Minor fix

78722a1

Merge branch 'develop' into users/rogeyyer/ck/aick-1649

09e8fd8

geyyer changed the title ~~Users/rogeyyer/ck/aick 1649~~ fix(ck-tile): Prefer using amd-smi over rocm-smi Jul 22, 2026

github-actions Bot added documentation project: composablekernel labels Jul 22, 2026

therock-pr-bot Bot added the Not ready to Review label Jul 22, 2026

Minor fix

60fd346

therock-pr-bot Bot removed the Not ready to Review label Jul 22, 2026

assistant-librarian Bot added the organization: ROCm label Jul 22, 2026

geyyer marked this pull request as ready for review July 23, 2026 14:36

geyyer requested review from a team as code owners July 23, 2026 14:36

Copilot AI review requested due to automatic review settings July 23, 2026 14:36

Copilot started reviewing on behalf of geyyer July 23, 2026 14:37

View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

geyyer and others added 3 commits July 23, 2026 15:49

Potential fix for pull request finding

ef21df1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

b8df3ae

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Fix the fallback scenario

de1b355

andriy-ca approved these changes Jul 27, 2026

View reviewed changes

geyyer and others added 2 commits July 27, 2026 15:32

Fix scripts

999dd38

Merge branch 'develop' into users/rogeyyer/ck/aick-1649

00917eb

andriy-ca approved these changes Jul 27, 2026

View reviewed changes

geyyer merged commit 4747ebf into develop Jul 27, 2026
26 checks passed

geyyer deleted the users/rogeyyer/ck/aick-1649 branch July 27, 2026 22:32

ozturkosu mentioned this pull request Jul 30, 2026

fix(ck-tile): prefer amd-smi over rocm-smi in remaining TE GEMM benchmarks #10204

Merged

1 task done

ozturkosu mentioned this pull request Aug 5, 2026

fix(ck-tile): prefer amd-smi for dispatcher GPU-arch detection #10445

Merged

3 tasks done

Navigation Menu

Repository navigation

fix(ck-tile): Prefer using amd-smi over rocm-smi - #9731

fix(ck-tile): Prefer using amd-smi over rocm-smi#9731
geyyer merged 14 commits into
developfrom
users/rogeyyer/ck/aick-1649

geyyer commented Jul 22, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andriy-ca left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

geyyer commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Checks Passed — Ready for Review

Uh oh!

therock-pr-bot Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andriy-ca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

geyyer commented Jul 22, 2026 •

edited

Loading

therock-pr-bot Bot commented Jul 22, 2026 •

edited

Loading

therock-pr-bot Bot commented Jul 22, 2026 •

edited

Loading