CI: Migrate MoRI to ROCm 10.1 by QizhouZhang97 · Pull Request #871 · ROCm/mori

QizhouZhang97 · 2026-10-29T08:07:04Z

Motivation

Move MORI's CI, images and wheels from ROCm 7.14 to ROCm 10.1
(rocm/pytorch:rocm10.1.0_ubuntu24.04_py3.12_pytorch_release_2.12.0), keeping a
ROCm 7.14 intranode gate on MI355X, and fix what the move broke.

Technical Details

Bugs fixed:

CCO bench OOM on MI355X-AINIC: ccoMemAlloc: hipMemCreate failed: 2 (out of memory)
on an empty GPU. The KFD actually returns EADDRINUSE: the runners' amdgpu-dkms
(6.14.14) lacks 93aa919, so a VRAM allocation without a VA is checked against SVM
ranges over [0, size). ROCm 10.1's HIP pins pageable copies above 64 KiB (7.14:
above 1 MiB, Optimize h2d and d2h batch copies rocm-systems#6697), so the bench's 256 KiB idx copies get
registered as SVM ranges on the low brk heap. Fix: HSA_USE_SVM=0 in
docker/Dockerfile.dev, replacing MALLOC_MMAP_THRESHOLD_, which did not prevent it.
Drop it once the runners' driver carries 93aa919.
Ranks exit 255 in every CCO Python example from 04_flydsl_lsa_put on: torch/FlyDSL
had already loaded rocm_sdk's libamd_smi.so.27, and mori loaded a second copy from
/opt/rocm, which reports 0 sockets. mori now reuses the copy already in the process.
JAX EP test hangs (exit 124): with one GPU per worker via HIP_VISIBLE_DEVICES,
10.1's RCCL fails topology init. Workers now see all GPUs and get theirs through
local_device_ids.
Every import mori warned about a ROCm mismatch: setup.py baked HIP's version
(7.15) instead of the ROCm release; it now reads rocm-core's rocm_version.h.
torch 2.14 breaks ccoMemImport (symm_mem.empty() offsets data by 0x2400):
the 10.1 images use torch 2.12, as the 7.14 CI did.

Other: 10.1 has no rocm_smi, so CI now runs mori's amd_smi topology path; the JAX job
builds Dockerfile.dev on rocm/jax 10.1 (Dockerfile.jax removed); nightly wheels are
py3.10 / py3.12.

Test Plan

CI on this PR. The OOM was reproduced on the runner host with the CI image: the bench
fails on all 8 ranks without HSA_USE_SVM=0 and passes with it.

Test Result

On e4a4ccf: CCO unit test, CCO internode test and MoRI internode test (MI355X_AINIC)
pass. Before it, CCO unit test failed at the ops-v2 bench.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

Move every CI lane from ROCm 7.14 (and 7.2.4) to ROCm 10.0. The base is rocm/pytorch:rocm10.0_ubuntu24.04_py3.11_pytorch_release_2.13.0, which docker/Dockerfile.dev now defaults to together with the 10.0 apt repo, so the separate intranode-test-rocm10 gate is dropped: intranode-test itself runs on ROCm 10 now, on both MI355X and MI300X. The JAX job moves off the from-source jaxlib 0.9.1 build on a ROCm 7.11 base to AMD's rocm/jax 10.0 image (docker/Dockerfile.jax), and the nightly wheels build against ROCm 10.0's RHEL 8 packages for py3.11 and py3.12 -- ROCm 10.0 ships no py3.10 PyTorch image, and 3.10 is end-of-life. setup.py baked hipconfig's version into the package, which on ROCm 10.0 is HIP's 7.15 rather than the release, so every `import mori` warned that it was built against another ROCm than the rocm_sdk it runs on. It now reads rocm-core's rocm_version.h first, and a test checks the baked pattern against the installed rocm_sdk. Co-authored-by: Cursor <cursoragent@cursor.com>

Everything else now runs on ROCm 10.0, so nothing would notice a change that only builds or runs on it. The new intranode-test-rocm714 job builds what intranode-test builds on the 7.14 base and apt ROCm, checks the install, and runs the EP intranode test, on the MI355X_AINIC runner only. Co-authored-by: Cursor <cursoragent@cursor.com>

The default base is now rocm/pytorch:rocm10.1.0_ubuntu24.04_py3.12_pytorch_release_2.14.0 with the ROCm 10.1 apt toolchain. 10.1 drops rocm_smi, so this is also the first CI to run mori's amd_smi topology path. The 10.1 images run the system Python, which has no `python`, so docker/Dockerfile.dev adds one. The apt repo URL now takes the base's Ubuntu release ({ubuntu}), which lets the same Dockerfile build the ubuntu22.04 py3.10 image: with 10.1 the nightly wheel lanes are py3.10 and py3.12 again, as they were on 7.2.4/7.14. The rocm/jax 10.1 image ships no rocm-sdk-devel, so the JAX job builds docker/Dockerfile.dev on top of it like the torch images do, and docker/Dockerfile.jax goes away. Co-authored-by: Cursor <cursoragent@cursor.com>

On ROCm 10.1, where mori reads GPU topology through amd_smi, importing torch first (directly or through FlyDSL) has already loaded the rocm_sdk wheels' libamd_smi.so.27. OpenAmdSmi() then loaded a second copy from /opt/rocm, which binds part of its state to the first one's symbols and reports 0 sockets, so the rank exits with 255. On PR #871 every CCO Python example from 04_flydsl_lsa_put on died this way. The rocm_smi path never hit it: it opens librocm_smi64.so.1 by soname, which reuses a loaded copy. Reuse a libamd_smi already mapped into the process before opening one by path (MORI_AMD_SMI_PATH still wins). amd_smi reference-counts init and shut_down, so mori's pair leaves torch's session alone. Only the 10.1+ branch changes; 7.14 still builds the rocm_smi one. Co-authored-by: Cursor <cursoragent@cursor.com>

The 10.1 lanes ran the pytorch_release_2.14.0 tags. Nothing in mori needs a torch newer than 2.12, the one the 7.14 CI already ran, and torch 2.14's symm_mem.empty() puts its data 0x2400 bytes into the allocation, which ccoMemImport does not accept (examples/cco/python/08_torch_symm_import). Move every 10.1 image to the pytorch_release_2.12.0 tag of the same ROCm and Python. Co-authored-by: Cursor <cursoragent@cursor.com>

The JAX EP test pinned each of its 8 workers to one GPU through HIP_VISIBLE_DEVICES. On ROCm 10.1 the RCCL that JAX uses then fails to initialize the communicator for the test's own jax.lax.all_gather on every rank but one ("TopoPreset: Local rank N not found in intra-node graph. Topology misconfiguration."). That rank waits in RCCL init for the others, they wait for it at the next jit, and the job times out (exit 124 on PR #871). A pure-JAX 8-process all_gather hangs the same way; 2 or 4 processes do not. Let every worker see every GPU and hand JAX its own through local_device_ids, and set mori's HIP device explicitly, since mori runs on the calling thread's. Co-authored-by: Cursor <cursoragent@cursor.com>

On ROCm 10.1 the CCO ops-v2 bench fails on the MI355X-AINIC runner with "ccoMemAlloc: hipMemCreate failed: 2 (out of memory)" on an empty GPU. The KFD returns EADDRINUSE, not ENOMEM: the runners' amdgpu-dkms (6.14.14) lacks 93aa919, so a VRAM allocation without a VA is checked against the process's SVM ranges over [0, size). The 10.1 HIP pins the bench's 256 KiB idx copies (idx.to(dev), idx.cpu()); 7.14 does not. On a KFD that offers the SVM API, the thunk registers each pinned buffer as an SVM range that outlives the pin, and these come from the brk heap of the non-PIE python, 0.2-1.1 GB up. The first 1.76 GB arena then fails on every rank. MALLOC_MMAP_THRESHOLD_ cannot prevent it: glibc still serves a 256 KiB request from a free heap chunk. HSA_USE_SVM=0 makes the thunk register host memory as userptr, so no SVM range exists. On the runner, with the CI image, the bench fails on all 8 ranks without it and passes with it (both backends, M=8192). Co-authored-by: Cursor <cursoragent@cursor.com>

jhchouuu · 2026-11-04T07:45:51Z

 # the runners' amdgpu-dkms carries it — note the DKMS module overrides the
 # distro kernel's amdgpu, so bumping the Ubuntu kernel alone does nothing.
-ENV MALLOC_MMAP_THRESHOLD_=131072
+ENV HSA_USE_SVM=0


why need this env var? is this a must?

是需要的, bug根因, 实际上 7.14也是存在这个问题

Agreed on the root cause. I checked #6697: the default pinned-copy threshold went from 1 MiB to 64 KiB, and an explicit GPU_PINNED_MIN_XFER_SIZE is still in MiB. So 7.14 has the same latent problem for a pageable copy over 1 MiB from the low heap.

What I would not do is put it in the image:

It is only needed where amdgpu lacks 93aa919. The amdgpu-dkms in ROCm 7.2.1 (6.16.6) already has it: kfd_chardev.c skips the SVM overlap check for a VRAM allocation with va_addr == 0. On an updated host neither ROCm hits this.

It is not free. With SVM off, every pageable copy is pinned again on each call. In the 10.1 image on MI355X:

a 256 KiB pageable H2D copy goes from about 21 us to about 1.2 ms;

a reused 16 MiB buffer goes from about 0.3 ms to 2-6 ms.

Everyone who builds Dockerfile.dev inherits that.

Could the runners get a newer amdgpu-dkms? If not, could HSA_USE_SVM=0 (with this comment) go only on the jobs that run on them, instead of in the image?

jhchouuu · 2026-11-04T07:46:57Z

+# CCO ops-v2 bench 在 mi355-gpu-51 上报 OOM（ROCm 10.1）：交接说明
+
+给在 mi355-gpu-51 上接手的 agent。目标：查清 PR #871 的 CCO CI 为什么在这台机器上
+报 `hipMemCreate failed: 2 (out of memory)`，并给出修复方向。下面的结论和数据都来自
+pit2-p03-g17（另一台 MI355X），在那台机器上复现不了。
+
+## 1. 背景
+
+- PR [ROCm/mori#871](https://github.com/ROCm/mori/pull/871)，分支 `dev/migrate_rocm10`，
+  head `4a4df301`。它把 CI 从 ROCm 7.14 迁到 ROCm 10.1，镜像是
+  `rocm/pytorch:rocm10.1.0_ubuntu24.04_py3.12_pytorch_release_2.12.0`。
+- 这是 pull_request 事件，CI checkout 的是 merge commit `cc973cdd`
+  （`4a4df301` 合进 main 的 `f3cebc3a`），不是 PR head。
+- 失败的 job 是 `ci_cco.yml` 的 "CCO unit test (MI355X_AINIC)"，runner 是
+  `mi355-gpu-51`。这台 runner 用 rootless podman（workflow 里 `CT: podman`），
+  容器用 `docker/ci_run.sh` 起，默认带 `--privileged`。
+- CI 镜像 tag `rocm/mori:ci-cco` 是共用的：main 的 7.14 构建和 PR 的 10.1 构建都写它。
+
+## 2. 现象
+


delete this handoff

jhchouuu · 2026-11-04T07:48:16Z

+void* LoadedAmdSmi() {
+  std::string path;
+  dl_iterate_phdr(
+      [](dl_phdr_info* info, size_t, void* data) {
+        const char* slash = info->dlpi_name ? strrchr(info->dlpi_name, '/') : nullptr;
+        if (slash && strncmp(slash + 1, "libamd_smi.so", strlen("libamd_smi.so")) == 0) {
+          *static_cast<std::string*>(data) = info->dlpi_name;
+          return 1;
+        }
+        return 0;
+      },
+      &path);
+  return path.empty() ? nullptr : dlopen(path.c_str(), RTLD_NOW | RTLD_NOLOAD);
+}
+


amd-smi is new dependency?

不是。main 上 9 月的 #663（2958fce6b）就已经让 ROCm 10.1+ 走 amd_smi 了（10.1 去掉了 rocm_smi）。它是运行时 dlopen 的，不是链接期依赖，这个 PR 也没改任何构建依赖。这里只是先复用进程里已经加载的那份，避免再从 /opt/rocm 加载第二份。

jhchouuu · 2026-11-04T07:48:41Z

+def test_baked_version_matches_the_installed_rocm_sdk():
+    # A mismatch only warns at import time, from inside rocm_sdk, so nothing
+    # else in the suite would notice it.
+    if bootstrap.ROCM_VERSION_PATTERN is None:
+        pytest.skip("ROCm build metadata is generated while building the package")
+    rocm_sdk = pytest.importorskip("rocm_sdk")
+
+    major, minor, *_ = rocm_sdk.__version__.split(".")
+    assert bootstrap.ROCM_VERSION_PATTERN == f"{major}.{minor}.*"


why need this test?

will remove

jhchouuu

A few more points, beyond the inline threads. I checked the amd_smi one with this PR's Dockerfile.dev (ROCm 10.1) on MI355X.

Wheels for ROCm 7.x. On main the nightly wheel lanes are 7.2.4 (py3.10) and 7.14 (py3.12). With this PR both build against 10.1, and Dockerfile.manylinux defaults to 10.1. Releases follow the nightly lanes, so the next release would ship only 10.1-built wheels, and no job would cover 7.2.x at all; the only 7.x lane left is the 7.14 source-build gate.

Could the py3.10 lane stay on 7.2.4, as on main? Its build args and test images are the ones this diff removes, so a 7.x wheel would still be built and tested every night. The wheel artifacts are keyed by Python version, so a separate 7.x lane on the same Python would also need the ROCm version in the artifact names.

Reusing the loaded libamd_smi. This does fix it:

With main in the 10.1 image, import torch alone loads rocm_sdk's libamd_smi.so.27. The next mori communicator then dies with "amdsmi_get_socket_handles reported 0 sockets".
With this PR the process keeps one copy, and a session opened on that copy before mori's init/shut_down still sees every socket afterwards.

Two small things:

mori now prefers whatever copy is already loaded over the one it was built against. A copy from another ROCm release could have different struct layouts. Comparing amdsmi_get_lib_version() with the header's AMDSMI_LIB_VERSION_MAJOR before reusing it would rule that out.
amdsmi_init on an already-initialized library only takes a reference and ignores its flags. So reuse relies on the first user having initialized with AMDSMI_INIT_AMD_GPUS. torch does today, but a comment saying so would help.

torch 2.14 and ccoMemImport. Pinning torch 2.12 keeps CI green, but users on torch 2.14 will hit the 0x2400 offset in symm_mem.empty(). Could you open an issue for it so it is tracked? We saw the same failure when testing wheels with torch 2.14.

Working notes from the investigation added in c1d4ce8; the findings are in e4a4ccf's message and next to HSA_USE_SVM in the workflows. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

With both nightly lanes on 10.1, the next release would ship only 10.1-built wheels and no job would build or test 7.2.x. Keep the py3.10 lane on 7.2.4 as on main: its RHEL 8 build args for Dockerfile.manylinux and its rocm7.2.4 py3.10 test images in test-wheel, test-wheel-internode and test-pypi-wheel. py3.12 moves to 10.1. The wheel artifacts are keyed by Python version, so the two lanes do not collide. Co-authored-by: Cursor <cursoragent@cursor.com>

Only hosts whose amdgpu-dkms lacks 93aa919 need it, and it is not free: with SVM off, libhsakmt pins a pageable buffer again on every copy, which everyone who builds Dockerfile.dev would inherit. Of the runners, only the MI355X-AINIC pair (amdgpu-dkms 6.14.14) lacks the fix. Set it per job instead, on node1's and node2's containers: cco-unit-test and cco-internode-test in ci_cco.yml, internode-test's MI355X_AINIC entry in ci.yml, and test-wheel-internode's MI355X_AINIC entries and flydsl-compat in nightly.yml. Dockerfile.dev drops the workaround block, MALLOC_MMAP_THRESHOLD_ included: it was there for the same bug and does not prevent it. Co-authored-by: Cursor <cursoragent@cursor.com>

A libamd_smi of another major version than the amdsmi.h mori was built against may lay out its structs differently. Check amdsmi_get_lib_version() against AMDSMI_LIB_VERSION_MAJOR before reusing the copy already in the process; otherwise say so on stderr and open one by path as before. Also note that reuse relies on the first user having initialized amd_smi with AMDSMI_INIT_AMD_GPUS: a later amdsmi_init only takes a reference and ignores its flags. test_topology in the 10.1 image finds all 8 GPUs with rocm_sdk's libamd_smi.so.27 preloaded (reused, one copy mapped) and with a stub reporting major 99 preloaded (not reused, falls back). Co-authored-by: Cursor <cursoragent@cursor.com>

QizhouZhang97 and others added 8 commits October 29, 2026 07:55

oom report

c1d4ce8

QizhouZhang97 requested a review from jhchouuu November 4, 2026 07:05

QizhouZhang97 changed the title ~~Dev/migrate rocm10~~ CI: Migrate MoRI to ROCm 10.1 Nov 4, 2026

jhchouuu reviewed Nov 4, 2026

View reviewed changes

QizhouZhang97 and others added 5 commits November 4, 2026 21:48

Drop the CCO bench OOM handoff notes

905b80b

Working notes from the investigation added in c1d4ce8; the findings are in e4a4ccf's message and next to HSA_USE_SVM in the workflows. Co-authored-by: Cursor <cursoragent@cursor.com>

test(cco): drop the rocm_sdk version-match test

98efed5

Co-authored-by: Cursor <cursoragent@cursor.com>

QizhouZhang97 mentioned this pull request Nov 6, 2026

ccoMemImport rejects torch 2.14 symm_mem tensors: data starts 0x2400 bytes into the allocation #925

Open

Navigation Menu

Repository navigation

CI: Migrate MoRI to ROCm 10.1 - #871

CI: Migrate MoRI to ROCm 10.1#871
QizhouZhang97 wants to merge 13 commits into
mainfrom
dev/migrate_rocm10

QizhouZhang97 commented Oct 29, 2026 •

edited

Loading

Uh oh!

jhchouuu Nov 4, 2026

Uh oh!

QizhouZhang97 Nov 4, 2026

Uh oh!

jhchouuu Nov 4, 2026

Uh oh!

jhchouuu Nov 4, 2026

Uh oh!

QizhouZhang97 Nov 4, 2026

Uh oh!

jhchouuu Nov 4, 2026

Uh oh!

QizhouZhang97 Nov 4, 2026

Uh oh!

jhchouuu Nov 4, 2026

Uh oh!

QizhouZhang97 Nov 4, 2026

Uh oh!

jhchouuu left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

QizhouZhang97 commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhchouuu left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

QizhouZhang97 commented Oct 29, 2026 •

edited

Loading