fix(rdc): keep librdc_client loaded so rdci does not crash on exit by adam360x · Pull Request #13296 · ROCm/rocm-systems

adam360x · 2026-10-12T18:40:50Z

Motivation

On ROCm 10.2 nightly builds, rdci crashes with Segmentation fault as it exits, after printing its normal output. Every subcommand we tried crashes: discovery, group and fieldgroup in 10 of 10 runs, dmon in 7 of 10. For example, on an 8x AMD Instinct MI350X node, rdci discovery localhost -l -u lists all 8 GPUs, prints the closing table line, and then segfaults. Python clients that use standalone mode (rdc_connect() ... rdc_shutdown()) hit the same crash.

Technical Details

Root cause

~RdciSubSystem() calls rdc_shutdown(), which dlclose()s librdc_client.so through RdcLibraryLoader::unload().
librdc_client.so links gRPC statically. gRPC's background threads (event_engine pool, grpc_global_timer, lifeguard) keep running after the last channel is destroyed, so unloading the library unmaps code that those threads are still executing.
gdb on the crash: the main thread is in _dl_close_worker() called from RdcLibraryLoader::unload(), and an event_engine thread faults inside librdc_client.so.

Why it started now

The bug was latent. When TheRock built gRPC with GCC, the gRPC objects added one STB_GNU_UNIQUE symbol (std::_Sp_make_shared_tag::_S_ti()::__tag) to librdc_client.so. glibc never unloads a library that defines a unique symbol (LD_DEBUG=files shows activating NODELETE), so dlclose() did nothing.
grpc: build with amd-llvm and add ppc64le support TheRock#8423 (merged 2026-09-30) builds gRPC with amd-llvm. Clang emits no unique symbols, so dlclose() now really unloads the library. The 2026-09-30 and 2026-10-01 nightlies were built from the same rocm-systems commit, yet librdc_client.so went from 18.3 MB with one unique symbol to 14.5 MB with none. With the stock rdci, the 09-30 library exits cleanly 20 of 20 times and the 10-01 library segfaults 19 of 20 times. 10.2.0a20261001 is the first affected nightly.
RDC CI builds RDC with GCC and shared gRPC. There, librdc_client.so also stays loaded: the GCC-built gRPC libraries are never unloaded and bind symbols into librdc_client.so, so glibc cannot unload it either. CI never saw the crash.

Fix

rdc_libs/rdc_client/CMakeLists.txt: link librdc_client.so with -z nodelete (DF_1_NODELETE). The library now stays loaded no matter which compiler builds gRPC, which is how every build behaved before 10.2.0a20261001. With the flag, dlclose() neither runs the library's destructors nor unmaps it; they run at process exit, as for a linked library.
Why a link flag rather than RTLD_NODELETE in RdcLibraryLoader::load(): both fix the crash (an LD_PRELOAD shim that adds RTLD_NODELETE to that dlopen() gave 20/20 clean exits). But the constraint belongs to librdc_client.so, which carries gRPC's threads, so the flag protects every program that loads it, including ones that dlopen() it directly. The same loader also unloads the embedded librdc.so, which is safe to unload and should stay that way.
Embedded mode is not affected: librdc.so contains no gRPC. With the stock 10.2.0a20261008 build, rdc_start_embedded(), a field watch, rdc_stop_embedded() and rdc_shutdown() really unload librdc.so and librdc_rocr.so, and the process exits cleanly 30 of 30 times.

Tests (tests/ci/test_negative_resilience.py, no GPU or daemon needed)

test_client_library_is_nodelete: DT_FLAGS_1 of the installed librdc_client.so has DF_1_NODELETE. This test fails without the fix in every build configuration, including RDC CI's GCC build.
test_client_library_stays_loaded_after_shutdown: a child process runs two cycles of rdc_connect() to a closed port through librdc_bootstrap.so, a failing RPC (which starts gRPC's threads), rdc_disconnect() and rdc_shutdown(). It checks in /proc/self/maps that the tested librdc_client.so is the copy that was loaded and that it is still mapped at the end.
In the RDC CI build only the first test can fail without the fix; in a TheRock build both do (see Test Result).

No CHANGELOG entry: no release has shipped the crash (TheRock release/therock-10.1 and the 10.2 BKC branch build gRPC with GCC), and this fix lands on develop ahead of the first release that builds gRPC with amd-llvm.

As of 2026-10-12, no other open PR touches these files or rdc_libs/bootstrap/.

Not fixed here (found while investigating; pre-existing; follow-up tickets to be filed)

RdcLibraryLoader::load() reads libHandler_ before it takes its mutex, so concurrent rdc_connect() calls race.
When rdcd cannot read a GPU's market name (for example, a non-root daemon without the render group), rdc_device_get_attributes() sends an uninitialized device_name. The client cannot parse the reply (invalid UTF-8), and rdci discovery silently stops listing GPUs with exit code 0.

Issue Tracking

JIRA ID: ROCM-32616

Test Plan

Reproduce on 8x MI350X (gfx950) with the build from the ticket (10.2.0a20261008 multi-arch tarball): stock non-root rdcd -u, then the ticket's rdci commands.
A/B: build develop and this branch the way TheRock does (amd-llvm, static gRPC 1.78.1 with TheRock's options). Run each rdci 30 times against the stock daemon. Also drop only the fixed librdc_client.so into the stock install.
Run rdci subcommands (discovery -l/-v/-i, group, fieldgroup, dmon) 10 times each, stock vs fixed library.
Run tests/ci against both builds.
Build both again with GCC 13 and shared gRPC (GNU toolchain with shared gRPC like RDC CI, which uses GCC 11 and gRPC 1.67.1) and run the 2 new tests against all 4 builds.
Embedded mode: start, watch fields, stop and rdc_shutdown() 30 times with the stock build.
Cross-check on MI455X (gfx1250) with the stock 2026-10-12 nightly.

Test Result

8x MI350X, stock rdcd from the ticket build:

Client	Runs	Result
Stock `10.2.0a20261008` `rdci discovery -l -u`	30	SIGSEGV 30/30
`develop` built with the TheRock toolchain	30	SIGSEGV 30/30
This branch built the same way	30	exit 0, 30/30
Stock `rdci` with only the fixed `librdc_client.so`	30	exit 0, 30/30

Ticket commands under a pty: stock output matches the ticket and 3 of 4 runs segfault; with the fixed library 4 of 4 exit 0.
Subcommands, 10 runs each, stock vs fixed library: discovery -l, -v, -i, group -l, fieldgroup -l go from SIGSEGV 10/10 to exit 0 10/10; dmon -c 2 (root daemon) from SIGSEGV 7/10 to exit 0 10/10; dmon against a closed port from SIGSEGV 3/10 to its normal connection error (exit 8) 10/10.
readelf -d: the develop build has no FLAGS_1; this branch has Flags: NODELETE, with both LLD (TheRock toolchain) and GNU ld (GCC 13 with shared gRPC, built like RDC CI).
New tests, 3 runs per build: TheRock toolchain, develop fails both (the residency probe finds the library unloaded or segfaults) and this branch passes both. GCC with shared gRPC, develop fails the NODELETE test and passes the residency test (the GCC-built gRPC libraries keep librdc_client.so loaded); this branch passes both.
RDC CI rdc_test (GCC 11, shared gRPC 1.67.1, no GPU) on 42a45b3: both new tests PASSED (59 passed, 17 skipped).
tests/ci on MI350X, 2 runs per build: the develop build had 15 and 14 failures, this branch 9 and 9. Fixed: the 2 new tests, plus test_discovery_lists_gpus, test_discovery_shows_attributes and test_dmon_invalid_field_id, which segfaulted on exit, and test_wrong_port_connection_refused, which segfaulted in 1 of the 2 runs. The 9 failures left are the same on both builds and unrelated: 8 tests call rdci with arguments it rejects (names where it expects numeric ids, such as dmon -f 100,140 where -f takes a field-group id, or field 140, which does not exist), and the rdcd restart test collides with the session daemon's lock file.
MI455X, stock 2026-10-12 nightly: 30/30 SIGSEGV. The 2026-09-19 nightly client against the same daemon: 30/30 exit 0.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

- Link librdc_client.so with -z nodelete. rdc_shutdown() dlclose()s it, but the gRPC inside leaves background threads running after the last channel is destroyed; unmapping the library under them made every rdci command end in SIGSEGV. - The bug was latent while gRPC was built with GCC: a GNU-unique symbol from gRPC made glibc pin the library. gRPC built with clang has none, so dlclose() really unloaded it. - Add no-GPU tests: the library carries the NODELETE flag and stays loaded after rdc_shutdown(). Signed-off-by: Adam360x <Adam.pryor@amd.com>

therock-pr-bot · 2026-10-12T18:41:39Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

Copilot

🟢 Approval recommended

The focused linker change addresses the documented unload race and is covered by configuration and runtime regression tests.

0 open findings

What changed in this PR

Prevents RDC client shutdown crashes by keeping the statically linked gRPC client library resident.

Changes:

Links librdc_client.so with DF_1_NODELETE.
Adds ELF flag and post-shutdown residency tests.

File	Description
`projects/rdc/rdc_libs/rdc_client/CMakeLists.txt`	Applies the `nodelete` linker option.
`projects/rdc/tests/ci/test_negative_resilience.py`	Verifies the linker flag and library lifetime.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- Run the probe with the tested lib directory first in LD_LIBRARY_PATH and check residency by full path, so another installed copy of librdc_client.so cannot satisfy the check. - Name the ELF constants and mention the new tests in the module docstring. Signed-off-by: Adam360x <Adam.pryor@amd.com>

- Run two connect / RPC / rdc_shutdown() cycles in the probe: the client library now survives rdc_shutdown() and is reused. - Check /proc/self/maps for the tested librdc_client.so: it must be the copy the bootstrap loaded, and still be mapped at the end. - Note why the NODELETE test is needed next to the residency test. Signed-off-by: Adam360x <Adam.pryor@amd.com>

Copilot

🟢 Approval recommended

The targeted linker change addresses the documented unload race and is covered by configuration-independent regression tests.

0 open findings

🧠 Review effort: Balanced

adam360x · 2026-10-12T20:46:01Z

Same root cause as #13298 (ROCM-32420; also explains ROCM-32219, the rdci topo segfault on MI350X), which came out of a parallel investigation of the same crash. #13298 passes RTLD_NODELETE at the dlopen() in rdc_connect() instead of linking with -z nodelete. It adds A/B results on 8x MI350X and MI455X with the 10.2.0-20261007 packages and shows that released ROCm 10.0.0 and 10.1.0 are not affected. Only one of the two PRs should merge.

exit (#13296) ## Motivation On ROCm 10.2 nightly builds, `rdci` crashes with `Segmentation fault` as it exits, after printing its normal output. Every subcommand we tried crashes: `discovery`, `group` and `fieldgroup` in 10 of 10 runs, `dmon` in 7 of 10. For example, on an 8x AMD Instinct MI350X node, `rdci discovery localhost -l -u` lists all 8 GPUs, prints the closing table line, and then segfaults. Python clients that use standalone mode (`rdc_connect()` ... `rdc_shutdown()`) hit the same crash. Signed-off-by: Adam360x <Adam.pryor@amd.com> [rocm-systems] ROCm/rocm-systems#13296 (commit 57099d7)

systems-assistant · 2026-10-14T12:02:55Z

TheRock Submodule Bump Activity

Newest first

2026-10-14 — Included in TheRock via ROCm/TheRock#8996.

## Motivation QA reports that `rdci link -u --xgmi-link-status` segfaults on 8x MI300X with TheRock 10.2.0-20261001 and -20261008 and prints no link table, and points at the uninitialized `amdsmi_xgmi_link_status_t` in `rdc_link_status_get()`. The reported crash is the `rdci` exit crash that #13296 and #13298 fixed on develop (merged 2026-10-13). `rdci link` prints its table with `\n` and no flush, so when stdout is a pipe, as in the QA harness, the crash at exit discards the whole table. The amdsmi call the ticket suspects succeeded on the QA host: `amd-smi xgmi -l` makes the same call for each GPU and prints N/A for a GPU whose call fails, and it printed D for every link. The code path the ticket points at is still wrong, and this PR fixes it. `amdsmi_get_gpu_xgmi_link_status()` can fail without writing its output (when it cannot read the GPU metrics or the sockets) or after writing part of it (`AMDSMI_STATUS_UNEXPECTED_DATA`). RDC copied the struct anyway: - A GPU whose query failed reported the previous GPU's links, left on the stack. - Any other leftover link count was copied as is. Above 8 it reads past the end of amdsmi's `status[8]`, and above 16 it writes past `link_states[16]`. With an injected count of 4096, `rdcd` crashed (see Test Result). `rdci link` also hid failures. It printed nothing and exited 0 when it could not get the link status, for example when `rdcd` was down or had crashed. `rdci discovery` and `rdci topo` exit 8 in the same case.

## Motivation Two RDC bugs found while fixing the `rdci` exit crash (#13296): - **Library unloads.** `rdc_connect()` and `rdc_start_embedded()` shared one loader in `librdc_bootstrap`, and each load first unloaded the library already loaded. So `rdc_stop_embedded()` crashed (SIGSEGV) after a standalone handle was opened, a second handle could unload the first one's library, and concurrent `rdc_connect()` calls failed at random. - **Unset GPU names.** When a GPU's name could not be read (a GPU index that does not exist, or `rdcd` without GPU access), `rdc_device_get_attributes()` returned unset bytes and `rdcd` sent them to clients. `rdci discovery` then printed only a header and exited 0. GCC builds, as in RDC CI, hide the crashes; clang builds, as in TheRock, show them. Signed-off-by: Adam360x <Adam.pryor@amd.com>

## Motivation QA reported that `rdci topo -u -i 6` and `-i 7` print the XGMI table and then crash with SIGSEGV on 8x MI350X with the ROCm 10.2.0 nightly 20261001, and that the test harness, which pipes stdout, captured empty output for GPU 7. That crash is the `rdci` exit crash that #13296 and #13298 fixed on develop (merged 2026-10-13). It is not specific to GPUs 6 and 7 (see Test Result), and this PR does not touch it. Verifying it showed that `rdci topo` reports the links of the wrong GPUs. `rdc_device_topology_get()` ignored the GPU index when it filled in the links: for every GPU i it looped over all peers and overwrote `link_infos[i]` each time, so entry i ended up holding the link from GPU i to the last GPU. `rdci topo` prints entry i as the link to GPU i. On a fully connected XGMI mesh, such as the MI350X host, every link is the same and the table looks right. On a system whose GPU pairs are not all linked the same way, for example two XGMI hives joined by PCIe, the table is wrong. `rdci topo` also exits 0 when the GPU index is out of range (after printing an error) and when the topology query fails (after printing nothing). ## Technical Details - `rdc_libs/rdc/src/RdcTopologyLinkImpl.cc`: `rdc_device_topology_get()` fills `link_infos[i]` with the link from the requested GPU to GPU i, making the four AMD SMI link queries once per peer instead of once per GPU pair. The requested GPU's own entry gets N/A values: the max values that `rdc.h` documents for N/A data, and an undefined link type. A GPU count above `RDC_MAX_NUM_DEVICES` now returns `RDC_ST_MAX_LIMIT`. Only an `assert` guarded it before, which Release builds drop, so a larger count would have overrun a 128-entry stack array and the result. A GPU whose handle lookup fails now returns `RDC_ST_NOT_FOUND` instead of `RDC_ST_OK`. - `RdcTopologyLinkImpl.cc`: link types RDC does not define (AMD SMI's not applicable, unknown, NUMA and XNUMA) and failed lookups are `RDC_IOLINK_TYPE_UNDEFINED`; the cast passed values 3-6 to API and gRPC callers. `rdc.h` documents what `rdc_device_topology_get()` fills in, the requested GPU's own entry and its statuses. - `rdci/src/RdciTopologyLinkSubSystem.cc`: an out-of-range `-i` exits with `RDC_ST_BAD_PARAMETER` (5) instead of 0. A failed topology query prints `rdci Error: Fail to get the topology of GPU <N>` and exits with its status instead of printing nothing and exiting 0. - `tests/unit/test_topology_link.cc`, `tests/unit/CMakeLists.txt`: new GPU-less `rdc_topology_tests`. librdc cannot be loaded without a GPU (it initializes AMD SMI when loaded), so the test builds `RdcTopologyLinkImpl.cc` itself against fake AMD SMI calls that model two XGMI hives joined by PCIe. It links only `librdc_bootstrap` (for the logger) and gtest, and is installed with the RDC test package. - `tests/ci/test_gtest_runner.py`: run `rdc_topology_tests` in RDC CI. `tests/ci/test_rdc_core.py`: GPU tests that `rdci topo -i N` exits 0 and lists every other GPU for each GPU, and that an out-of-range index exits nonzero with an error. - `CHANGELOG.md`: entry under Resolved Issues. No proto or struct changes. #13384 fixes `rdc_link_status_get()` in the same file and adds a similar test binary. The two PRs change different functions and merge cleanly in either order; both test binaries pass on the merged code. ## Issue Tracking JIRA ID: ROCM-32219 The crash in this ticket is fixed by #13296 and #13298. This PR fixes the `rdci topo` output and exit codes found while verifying that fix. ## Test Plan - Ticket flow on 8x MI350X (Ubuntu 24.04; the QA host was not reachable) with the full 10.2.0a20261001 nightly tarball: its `rdcd -u`, then `rdci topo -u -i N` for every GPU, 20 runs with stdout piped as in the QA harness and 20 on a pty, with the nightly `rdci` and with `rdci` built from develop (TheRock-like build: amd-llvm, static gRPC 1.78.1). - Same host, TheRock-like builds of develop and of this branch (`rdcd`, `librdc`, `rdci`): `rdci topo -u -i 0` to `-i 7` with and without an `LD_PRELOAD` shim in `rdcd` that makes `amdsmi_topo_get_link_type()` report GPUs 0-3 and GPUs 4-7 as two XGMI hives joined by PCIe, and `rdci topo -u -i 8`. - `rdc_topology_tests` against this branch and against develop's `RdcTopologyLinkImpl.cc` (GCC 11, plain and ASan+UBSan), and the installed amd-llvm binary from `share/rdc/rdctst_tests`. - `pytest tests/ci/test_rdc_core.py::TestTopology tests/ci/test_gtest_runner.py::TestGoogleTestSuite::test_rdc_topology`, run as root with `ROCM_DIR` at the develop install, this branch's install, and the 20261001 nightly. - This branch: `rdci topo -u -i N` for every GPU, 10 piped and 10 pty runs each. - `pre-commit run --config projects/rdc/.pre-commit-config.yaml` (codespell, clang-format, ruff-format, gersemi). ## Test Result - Ticket flow: the 20261001 nightly exited 139 in 320/320 runs on all 8 GPUs, with empty piped output in 159/160; develop's `rdci` exited 0 with all 7 rows in 320/320. - Two-hive shim: | `rdcd` | `rdci topo -i 0` | `-i 1` to `-i 3` | `-i 4` to `-i 7` | |---|---|---|---| | develop | links to GPUs 1-3 PCIe, to GPUs 4-7 XGMI (inverted) | wrong | right by coincidence | | this PR | links to GPUs 1-3 XGMI, to GPUs 4-7 PCIe | right | right | Without the shim both builds report XGMI for every link, as `amd-smi topology` does. A `rdci topo` call takes 7-18 ms with either build (wall time of the `rdci` process, including connect). `rdci topo -u -i 8` exits 0 with develop and 5 with this PR. - `rdc_topology_tests`: 4/4 pass with this PR (no ASan or UBSan reports; the installed amd-llvm binary also passes). With the old link-type cast, the failed-lookup test fails (type 4 instead of 0). Against develop's code 2 of 4 fail (rows hold other GPU pairs' links, and a failed lookup returns `RDC_ST_OK`), and the 129-GPU test hits an ASan stack-buffer-overflow in the old index array. - pytest: this PR 3 passed. Develop fails the out-of-range test (exit 0) and the wrapper (no binary). The 20261001 nightly fails both GPU tests (exit -11). - This branch: 160/160 `rdci topo` runs exited 0 with all rows. - pre-commit passed. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Signed-off-by: Adam360x <Adam.pryor@amd.com>

## Motivation QA reported that `rdcd` aborts during `rdci diag -u -r 2 -g 1` on AMD Instinct MI455X (gfx1250) with ROCm build #265: ``` F0921 ... queue_controller.cpp:147] Could not find agent - 129205111608944 ``` The checks of the ROCr diagnostic module (`librdc_rocr.so`: Compute Queue ready, System memory check) build test objects that call `hsa_init()` and `hsa_shut_down()` themselves. Nothing else held a reference, so ROCr was unloaded and loaded again between them. `rdcd` always loads rocprofiler-sdk (the profiler module registers a tool at the first `hsa_init()`, even with `RDC_DISABLE_ROCP=1`), and rocprofiler-sdk does not support a ROCr reload. Depending on its version, the next queue creation aborts `rdcd` (`Could not find agent`) or fails, or the next profiler field read aborts it (`HSA Queue is not initialized`). ROCm 10.2.0 and develop are affected too, and not only on MI455X: with a 10.2.0 nightly on MI350X the checks pass, but the next profiler field read aborts `rdcd`. Keeping ROCr loaded exposed a second rocprofiler-sdk problem: it never destroys a queue it intercepted. With a queue per check, each `rdci diag` run left two queues and about 70 MB of VRAM in `rdcd`, until queue creation failed after 126 runs (MI455X allows 254 queues per process). So the checks now share one queue per GPU. ## Technical Details - **ROCr lifetime** (`RdcDiagnosticLib.cc`): the module takes one `hsa_init()` reference at its first check and releases it in `rdc_diag_destroy()`; the test objects' calls only move the count. If that `hsa_init()` fails, the run fails (`ROCr initialization failed`) before any check is built, and the next run retries. - **Shared queue**: `AcquireDiagQueue()` creates one 128-packet queue per GPU, destroyed by `rdc_diag_destroy()` (the System memory check used to create a 131072-packet queue per run). A mutex runs one check at a time, since they share the queues (and `MemoryAccess.cc` a global kernel-argument pointer). The unused `CreateQueue()` helper is removed. - **Cleanup** (`ComputeQueueTest.cc`, `MemoryAccess.cc`, `base_rocr_utils.cc`, `RdcRocrBase.*`): both checks release their signals, memory, file descriptors and loaded code objects on every path, including Skip and failure (`ScopeExit` guard in `common.h`). A copy that fails after it started no longer leaks its signal. - **gfx1250 kernels**: without them both checks reported Skip on MI455X. They are code object v4 (ROCm 10.1 compiler: `clang -x cl -Xclang -finclude-default-header -target amdgcn-amd-amdhsa -mcpu=gfx1250 -mcode-object-version=4 -O3`), because the checks fill in the v4 kernel arguments; the default v6 fails the System memory check. - **Tests**: new GPU-less `rdc_rocr_lifetime_tests` runs the module against fake HSA calls that count loads, references and objects: ROCr is loaded once, one queue per GPU, a failed `hsa_init()` loads nothing and is retried, and no check leaves objects behind. It is built with the module, installed with the RDC test package, and run in RDC CI by `tests/ci/test_gtest_runner.py`. `rdc_diag_kernel_tests` requires v4 gfx1250 kernels. - `CHANGELOG.md` entries. No public API, proto or struct changes (the changed headers are not installed). Trade-off: after the first `rdci diag`, `rdcd` keeps ROCr and one queue per GPU until it stops (MI455X: 3 KFD queues and about 149 MB of VRAM, flat across runs after the first), as it already does after the first profiler read. Changing the GPU partition mode then needs `rdcd` stopped, and a second `rdci diag` waits for the first. #13570 and #13874 also edit `CHANGELOG.md`; trial merges show the three PRs merge cleanly in any order. Not in this PR: - rocprofiler-sdk: support a ROCr reload and destroy the queues it intercepts (ROCM-33402). A process that creates and destroys a queue in a loop aborts after about 250 iterations under `rocprofv3 --pmc`. - Kernel arguments: the shipped gfx950 kernels (code object v6) and the gfx940-gfx942 binary search kernel (v5) declare 280-288 bytes of kernel arguments, while the checks fill in the 80-byte v4 layout. Release builds pass both checks on MI350X with them, but an assertion-enabled build aborts `rdcd` on the Compute Queue check's existing kernel-argument size assertion. v4 rebuilds match and pass on MI350X. ## Issue Tracking JIRA ID: ROCM-31379 ## Test Plan - 1x MI455X (gfx1250), Ubuntu 26.04: QA's build #265, ROCm 10.2.0, and a 10.2.0 BKC build of 2026-10-13. Each experiment runs its own `rdcd -u` in a private network namespace: N `rdci diag -u -r 2 -g 1` runs, a profiler `rdci dmon` (fields 805/806), one more diag; also profiler first. KFD queues and VRAM from `/sys/class/kfd/kfd/proc/<pid>/`. The module is built from this branch against each ROCm. - 8x MI350X (gfx950), Ubuntu 24.04, 10.2.0 nightly of 2026-10-22: on GPU 0, the stock module and this PR's module (release and assertion-enabled builds, shipped gfx950 kernels and v4 rebuilds); on a group of all 8 GPUs, this PR's release build with the shipped kernels, in both orders. - `rdc_rocr_lifetime_tests` and `rdc_diag_kernel_tests` (GCC 11) on this branch, against develop's module, and with each part of the fix removed; the CI wrapper; `pre-commit`. ## Test Result | MI455X | `rdci diag` runs | KFD queues / VRAM, run 1 → last | profiler after | `rdcd` exit | |---|---|---|---|---| | #265 stock | aborts in run 1 (`Could not find agent`) | - | - | 134 | | 10.2.0 stock | run 1 Skip (no gfx1250 kernels), run 2 fails queue creation | - | aborts | 134 | | BKC 10-13 stock | aborts in run 1 | - | - | 134 | | 10.2.0, ROCr kept loaded, a queue per check | 126 Pass, then Fail | 4 → 254 / 0.39 → 9.3 GB | aborts | 134 | | 10.2.0, this PR | 141/141 Pass | 3 → 3 / 149 → 149 MB | works | 0 | | #265, this PR | 21/21 Pass | 3 → 3 / 149 → 149 MB | works | 0 | | BKC 10-13, this PR | 21/21 Pass | 3 → 3 / 287 → 149 MB | works | 0 | - Pass = both checks Pass. Profiler read first: 3/3 on #265 and on the BKC build. Stock builds run clean with `HSA_TOOLS_DISABLE_REGISTER=1` (rocprofiler-sdk not registered with ROCr). On the BKC build `rdci` itself exits with SIGSEGV after printing (the client exit crash fixed on develop by #13296), which does not affect `rdcd`. - 8x MI350X: stock passes 3/3 diags on GPU 0, then the profiler read aborts `rdcd` (`HSA Queue is not initialized`, exit 134). This PR (release build, shipped kernels): 4/4 on GPU 0 and 11/11 on all 8 GPUs, every GPU passing both checks, including a run after the profiler read; 3/3 on all 8 GPUs with the profiler read first. KFD queues stay flat across runs (24 with all 8 GPUs) and `rdcd` exits 0. - gfx1250 kernels: v4 passes both checks; v6 fails the System memory check in a release build. - `rdc_rocr_lifetime_tests` 5/5 pass and 4 of 5 fail against develop's module; each of 14 mutants of the fix, and reverting either review fix, makes a test fail. The mutex is not unit-tested. `rdc_diag_kernel_tests`, the CI wrapper and pre-commit pass. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Signed-off-by: Adam360x <Adam.pryor@amd.com>

adam360x requested a review from a team as a code owner October 12, 2026 18:40

Copilot AI balanced review requested due to automatic review settings October 12, 2026 18:40

github-actions Bot added the project: rdc label Oct 12, 2026

Copilot started reviewing on behalf of adam360x October 12, 2026 18:41

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

systems-assistant Bot added the organization: ROCm label Oct 12, 2026

adam360x added 2 commits October 12, 2026 13:52

adam360x requested a balanced review from Copilot October 12, 2026 19:16

Copilot started reviewing on behalf of adam360x October 12, 2026 19:17

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

adam360x mentioned this pull request Oct 12, 2026

fix(rdc): fix library reload crash and uninitialized GPU names #13308

Merged

1 task done

adam360x mentioned this pull request Oct 12, 2026

fix(rdc): stop rdci crashing with SIGSEGV on exit #13298

Merged

1 task done

bill-shuzhou-liu approved these changes Oct 13, 2026

View reviewed changes

adam360x merged commit 57099d7 into develop Oct 13, 2026
76 of 80 checks passed

adam360x deleted the users/adam360x/ROCM-32616 branch October 13, 2026 14:58

This was referenced Oct 13, 2026

fix(rdc): don't report XGMI links amdsmi did not return #13384

Merged

fix(rdc): report the requested GPU's links in rdci topo #13386

Merged

adam360x mentioned this pull request Oct 23, 2026

fix(rdc): keep ROCr loaded across rdci diag checks #13878

Merged

1 task done

Navigation Menu

Repository navigation

fix(rdc): keep librdc_client loaded so rdci does not crash on exit - #13296

fix(rdc): keep librdc_client loaded so rdci does not crash on exit#13296
adam360x merged 3 commits into
developfrom
users/adam360x/ROCM-32616

adam360x commented Oct 12, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

adam360x commented Oct 12, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adam360x commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 12, 2026

✅ All Policy Checks Passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

adam360x commented Oct 12, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 14, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adam360x commented Oct 12, 2026 •

edited

Loading