Repository navigation
fix(rdc): stop rdci crashing with SIGSEGV on exit - #13298
Conversation
rdci (and any standalone-mode client) crashed after printing its results on TheRock builds. Commands whose output was piped could lose all of it, since the crash came before stdout was flushed. - Load librdc_client.so with RTLD_NODELETE. gRPC keeps detached EventEngine threads (thread-pool workers and their watchdog) running after the channel is destroyed, and rdc_shutdown() dlclose()d the library, so the next wakeup executed unmapped code. - GCC builds never unloaded the library: their STB_GNU_UNIQUE symbols make glibc pin it. TheRock builds with clang, which emits none, and links gRPC statically into librdc_client.so. - Add rdc_client_tests: connect to an address with no server, fail one RPC, shut down, then check that the library is still loaded and that the process outlives the watchdog's next wakeup. Signed-off-by: Adam360x <Adam.pryor@amd.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
There was a problem hiding this comment.
🟢 Approval recommended
The focused lifecycle fix preserves existing defaults and includes targeted regression coverage.
0 open findings
What changed in this PR
Prevents standalone RDC clients from crashing during shutdown by keeping the statically linked gRPC client library mapped.
Changes:
- Adds configurable
dlopenflags and usesRTLD_NODELETEforlibrdc_client.so. - Adds a shutdown regression test and build configuration.
- Documents the resolved
rdcicrash.
| File | Description |
|---|---|
projects/rdc/CHANGELOG.md |
Records the resolved crash. |
projects/rdc/include/rdc_lib/RdcLibraryLoader.h |
Supports custom dlopen flags. |
projects/rdc/rdc_libs/bootstrap/src/RdcBootStrap.cc |
Keeps the client library mapped after shutdown. |
projects/rdc/tests/unit/CMakeLists.txt |
Builds and registers the regression test. |
projects/rdc/tests/unit/test_client_shutdown.cc |
Verifies safe shutdown and library retention. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Fail instead of skipping when librdc_client.so cannot be loaded: the test is built only with the client library, so a load failure means a broken setup. - Reconnect after rdc_shutdown() in a second pass, wait once at the end, and name the wait after the watchdog wakeup it covers. - Say that a client library pinned by an STB_GNU_UNIQUE symbol passes with or without the fix; only clang-only builds can fail. - Run the installed rdc_client_tests from the RDC CI pytest suite. - Drop the CHANGELOG entry. Released ROCm 10.0.0 and 10.1.0 do not crash: their client library has a GCC-built gRPC object whose unique symbol keeps it loaded. The crash exists only in 10.2 development builds made after gRPC moved to clang. Signed-off-by: Adam360x <Adam.pryor@amd.com>
- Reword the STB_GNU_UNIQUE note; the spell-fix hook had turned "statics" into "statistics". - Say why the CI wrapper fails rather than skips when rdc_client_tests is missing: rdctst and rdc_client_tests are both standalone-only. Signed-off-by: Adam360x <Adam.pryor@amd.com>
|
Same root cause as #13296 (ROCM-32616), which came out of a parallel investigation of the same crash. Both PRs keep
This PR also carries the hardware A/B results (8x MI350X, MI455X), the check that released ROCm 10.0.0 and 10.1.0 are not affected, and the related tickets ROCM-32420 and ROCM-32219. |
TheRock Submodule Bump ActivityNewest first
|
## Motivation QA reports that `rdci link -u --xgmi-link-status` segfaults on 8x MI300X with TheRock 10.2.0-20261001 and -20261008 and prints no link table, and points at the uninitialized `amdsmi_xgmi_link_status_t` in `rdc_link_status_get()`. The reported crash is the `rdci` exit crash that #13296 and #13298 fixed on develop (merged 2026-10-13). `rdci link` prints its table with `\n` and no flush, so when stdout is a pipe, as in the QA harness, the crash at exit discards the whole table. The amdsmi call the ticket suspects succeeded on the QA host: `amd-smi xgmi -l` makes the same call for each GPU and prints N/A for a GPU whose call fails, and it printed D for every link. The code path the ticket points at is still wrong, and this PR fixes it. `amdsmi_get_gpu_xgmi_link_status()` can fail without writing its output (when it cannot read the GPU metrics or the sockets) or after writing part of it (`AMDSMI_STATUS_UNEXPECTED_DATA`). RDC copied the struct anyway: - A GPU whose query failed reported the previous GPU's links, left on the stack. - Any other leftover link count was copied as is. Above 8 it reads past the end of amdsmi's `status[8]`, and above 16 it writes past `link_states[16]`. With an injected count of 4096, `rdcd` crashed (see Test Result). `rdci link` also hid failures. It printed nothing and exited 0 when it could not get the link status, for example when `rdcd` was down or had crashed. `rdci discovery` and `rdci topo` exit 8 in the same case.
## Motivation QA reported that `rdci topo -u -i 6` and `-i 7` print the XGMI table and then crash with SIGSEGV on 8x MI350X with the ROCm 10.2.0 nightly 20261001, and that the test harness, which pipes stdout, captured empty output for GPU 7. That crash is the `rdci` exit crash that #13296 and #13298 fixed on develop (merged 2026-10-13). It is not specific to GPUs 6 and 7 (see Test Result), and this PR does not touch it. Verifying it showed that `rdci topo` reports the links of the wrong GPUs. `rdc_device_topology_get()` ignored the GPU index when it filled in the links: for every GPU i it looped over all peers and overwrote `link_infos[i]` each time, so entry i ended up holding the link from GPU i to the last GPU. `rdci topo` prints entry i as the link to GPU i. On a fully connected XGMI mesh, such as the MI350X host, every link is the same and the table looks right. On a system whose GPU pairs are not all linked the same way, for example two XGMI hives joined by PCIe, the table is wrong. `rdci topo` also exits 0 when the GPU index is out of range (after printing an error) and when the topology query fails (after printing nothing). ## Technical Details - `rdc_libs/rdc/src/RdcTopologyLinkImpl.cc`: `rdc_device_topology_get()` fills `link_infos[i]` with the link from the requested GPU to GPU i, making the four AMD SMI link queries once per peer instead of once per GPU pair. The requested GPU's own entry gets N/A values: the max values that `rdc.h` documents for N/A data, and an undefined link type. A GPU count above `RDC_MAX_NUM_DEVICES` now returns `RDC_ST_MAX_LIMIT`. Only an `assert` guarded it before, which Release builds drop, so a larger count would have overrun a 128-entry stack array and the result. A GPU whose handle lookup fails now returns `RDC_ST_NOT_FOUND` instead of `RDC_ST_OK`. - `RdcTopologyLinkImpl.cc`: link types RDC does not define (AMD SMI's not applicable, unknown, NUMA and XNUMA) and failed lookups are `RDC_IOLINK_TYPE_UNDEFINED`; the cast passed values 3-6 to API and gRPC callers. `rdc.h` documents what `rdc_device_topology_get()` fills in, the requested GPU's own entry and its statuses. - `rdci/src/RdciTopologyLinkSubSystem.cc`: an out-of-range `-i` exits with `RDC_ST_BAD_PARAMETER` (5) instead of 0. A failed topology query prints `rdci Error: Fail to get the topology of GPU <N>` and exits with its status instead of printing nothing and exiting 0. - `tests/unit/test_topology_link.cc`, `tests/unit/CMakeLists.txt`: new GPU-less `rdc_topology_tests`. librdc cannot be loaded without a GPU (it initializes AMD SMI when loaded), so the test builds `RdcTopologyLinkImpl.cc` itself against fake AMD SMI calls that model two XGMI hives joined by PCIe. It links only `librdc_bootstrap` (for the logger) and gtest, and is installed with the RDC test package. - `tests/ci/test_gtest_runner.py`: run `rdc_topology_tests` in RDC CI. `tests/ci/test_rdc_core.py`: GPU tests that `rdci topo -i N` exits 0 and lists every other GPU for each GPU, and that an out-of-range index exits nonzero with an error. - `CHANGELOG.md`: entry under Resolved Issues. No proto or struct changes. #13384 fixes `rdc_link_status_get()` in the same file and adds a similar test binary. The two PRs change different functions and merge cleanly in either order; both test binaries pass on the merged code. ## Issue Tracking JIRA ID: ROCM-32219 The crash in this ticket is fixed by #13296 and #13298. This PR fixes the `rdci topo` output and exit codes found while verifying that fix. ## Test Plan - Ticket flow on 8x MI350X (Ubuntu 24.04; the QA host was not reachable) with the full 10.2.0a20261001 nightly tarball: its `rdcd -u`, then `rdci topo -u -i N` for every GPU, 20 runs with stdout piped as in the QA harness and 20 on a pty, with the nightly `rdci` and with `rdci` built from develop (TheRock-like build: amd-llvm, static gRPC 1.78.1). - Same host, TheRock-like builds of develop and of this branch (`rdcd`, `librdc`, `rdci`): `rdci topo -u -i 0` to `-i 7` with and without an `LD_PRELOAD` shim in `rdcd` that makes `amdsmi_topo_get_link_type()` report GPUs 0-3 and GPUs 4-7 as two XGMI hives joined by PCIe, and `rdci topo -u -i 8`. - `rdc_topology_tests` against this branch and against develop's `RdcTopologyLinkImpl.cc` (GCC 11, plain and ASan+UBSan), and the installed amd-llvm binary from `share/rdc/rdctst_tests`. - `pytest tests/ci/test_rdc_core.py::TestTopology tests/ci/test_gtest_runner.py::TestGoogleTestSuite::test_rdc_topology`, run as root with `ROCM_DIR` at the develop install, this branch's install, and the 20261001 nightly. - This branch: `rdci topo -u -i N` for every GPU, 10 piped and 10 pty runs each. - `pre-commit run --config projects/rdc/.pre-commit-config.yaml` (codespell, clang-format, ruff-format, gersemi). ## Test Result - Ticket flow: the 20261001 nightly exited 139 in 320/320 runs on all 8 GPUs, with empty piped output in 159/160; develop's `rdci` exited 0 with all 7 rows in 320/320. - Two-hive shim: | `rdcd` | `rdci topo -i 0` | `-i 1` to `-i 3` | `-i 4` to `-i 7` | |---|---|---|---| | develop | links to GPUs 1-3 PCIe, to GPUs 4-7 XGMI (inverted) | wrong | right by coincidence | | this PR | links to GPUs 1-3 XGMI, to GPUs 4-7 PCIe | right | right | Without the shim both builds report XGMI for every link, as `amd-smi topology` does. A `rdci topo` call takes 7-18 ms with either build (wall time of the `rdci` process, including connect). `rdci topo -u -i 8` exits 0 with develop and 5 with this PR. - `rdc_topology_tests`: 4/4 pass with this PR (no ASan or UBSan reports; the installed amd-llvm binary also passes). With the old link-type cast, the failed-lookup test fails (type 4 instead of 0). Against develop's code 2 of 4 fail (rows hold other GPU pairs' links, and a failed lookup returns `RDC_ST_OK`), and the 129-GPU test hits an ASan stack-buffer-overflow in the old index array. - pytest: this PR 3 passed. Develop fails the out-of-range test (exit 0) and the wrapper (no binary). The 20261001 nightly fails both GPU tests (exit -11). - This branch: 160/160 `rdci topo` runs exited 0 with all rows. - pre-commit passed. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Signed-off-by: Adam360x <Adam.pryor@amd.com>
## Motivation QA reported that `rdci topo -u -i 6` and `-i 7` print the XGMI table and then crash with SIGSEGV on 8x MI350X with the ROCm 10.2.0 nightly 20261001, and that the test harness, which pipes stdout, captured empty output for GPU 7. That crash is the `rdci` exit crash that #13296 and #13298 fixed on develop (merged 2026-10-13). It is not specific to GPUs 6 and 7 (see Test Result), and this PR does not touch it. Verifying it showed that `rdci topo` reports the links of the wrong GPUs. `rdc_device_topology_get()` ignored the GPU index when it filled in the links: for every GPU i it looped over all peers and overwrote `link_infos[i]` each time, so entry i ended up holding the link from GPU i to the last GPU. `rdci topo` prints entry i as the link to GPU i. On a fully connected XGMI mesh, such as the MI350X host, every link is the same and the table looks right. On a system whose GPU pairs are not all linked the same way, for example two XGMI hives joined by PCIe, the table is wrong. `rdci topo` also exits 0 when the GPU index is out of range (after printing an error) and when the topology query fails (after printing nothing). ## Technical Details - `rdc_libs/rdc/src/RdcTopologyLinkImpl.cc`: `rdc_device_topology_get()` fills `link_infos[i]` with the link from the requested GPU to GPU i, making the four AMD SMI link queries once per peer instead of once per GPU pair. The requested GPU's own entry gets N/A values: the max values that `rdc.h` documents for N/A data, and an undefined link type. A GPU count above `RDC_MAX_NUM_DEVICES` now returns `RDC_ST_MAX_LIMIT`. Only an `assert` guarded it before, which Release builds drop, so a larger count would have overrun a 128-entry stack array and the result. A GPU whose handle lookup fails now returns `RDC_ST_NOT_FOUND` instead of `RDC_ST_OK`. - `RdcTopologyLinkImpl.cc`: link types RDC does not define (AMD SMI's not applicable, unknown, NUMA and XNUMA) and failed lookups are `RDC_IOLINK_TYPE_UNDEFINED`; the cast passed values 3-6 to API and gRPC callers. `rdc.h` documents what `rdc_device_topology_get()` fills in, the requested GPU's own entry and its statuses. - `rdci/src/RdciTopologyLinkSubSystem.cc`: an out-of-range `-i` exits with `RDC_ST_BAD_PARAMETER` (5) instead of 0. A failed topology query prints `rdci Error: Fail to get the topology of GPU <N>` and exits with its status instead of printing nothing and exiting 0. - `tests/unit/test_topology_link.cc`, `tests/unit/CMakeLists.txt`: new GPU-less `rdc_topology_tests`. librdc cannot be loaded without a GPU (it initializes AMD SMI when loaded), so the test builds `RdcTopologyLinkImpl.cc` itself against fake AMD SMI calls that model two XGMI hives joined by PCIe. It links only `librdc_bootstrap` (for the logger) and gtest, and is installed with the RDC test package. - `tests/ci/test_gtest_runner.py`: run `rdc_topology_tests` in RDC CI. `tests/ci/test_rdc_core.py`: GPU tests that `rdci topo -i N` exits 0 and lists every other GPU for each GPU, and that an out-of-range index exits nonzero with an error. - `CHANGELOG.md`: entry under Resolved Issues. No proto or struct changes. #13384 fixes `rdc_link_status_get()` in the same file and adds a similar test binary. The two PRs change different functions and merge cleanly in either order; both test binaries pass on the merged code. ## Issue Tracking JIRA ID: ROCM-32219 The crash in this ticket is fixed by #13296 and #13298. This PR fixes the `rdci topo` output and exit codes found while verifying that fix. ## Test Plan - Ticket flow on 8x MI350X (Ubuntu 24.04; the QA host was not reachable) with the full 10.2.0a20261001 nightly tarball: its `rdcd -u`, then `rdci topo -u -i N` for every GPU, 20 runs with stdout piped as in the QA harness and 20 on a pty, with the nightly `rdci` and with `rdci` built from develop (TheRock-like build: amd-llvm, static gRPC 1.78.1). - Same host, TheRock-like builds of develop and of this branch (`rdcd`, `librdc`, `rdci`): `rdci topo -u -i 0` to `-i 7` with and without an `LD_PRELOAD` shim in `rdcd` that makes `amdsmi_topo_get_link_type()` report GPUs 0-3 and GPUs 4-7 as two XGMI hives joined by PCIe, and `rdci topo -u -i 8`. - `rdc_topology_tests` against this branch and against develop's `RdcTopologyLinkImpl.cc` (GCC 11, plain and ASan+UBSan), and the installed amd-llvm binary from `share/rdc/rdctst_tests`. - `pytest tests/ci/test_rdc_core.py::TestTopology tests/ci/test_gtest_runner.py::TestGoogleTestSuite::test_rdc_topology`, run as root with `ROCM_DIR` at the develop install, this branch's install, and the 20261001 nightly. - This branch: `rdci topo -u -i N` for every GPU, 10 piped and 10 pty runs each. - `pre-commit run --config projects/rdc/.pre-commit-config.yaml` (codespell, clang-format, ruff-format, gersemi). ## Test Result - Ticket flow: the 20261001 nightly exited 139 in 320/320 runs on all 8 GPUs, with empty piped output in 159/160; develop's `rdci` exited 0 with all 7 rows in 320/320. - Two-hive shim: | `rdcd` | `rdci topo -i 0` | `-i 1` to `-i 3` | `-i 4` to `-i 7` | |---|---|---|---| | develop | links to GPUs 1-3 PCIe, to GPUs 4-7 XGMI (inverted) | wrong | right by coincidence | | this PR | links to GPUs 1-3 XGMI, to GPUs 4-7 PCIe | right | right | Without the shim both builds report XGMI for every link, as `amd-smi topology` does. A `rdci topo` call takes 7-18 ms with either build (wall time of the `rdci` process, including connect). `rdci topo -u -i 8` exits 0 with develop and 5 with this PR. - `rdc_topology_tests`: 4/4 pass with this PR (no ASan or UBSan reports; the installed amd-llvm binary also passes). With the old link-type cast, the failed-lookup test fails (type 4 instead of 0). Against develop's code 2 of 4 fail (rows hold other GPU pairs' links, and a failed lookup returns `RDC_ST_OK`), and the 129-GPU test hits an ASan stack-buffer-overflow in the old index array. - pytest: this PR 3 passed. Develop fails the out-of-range test (exit 0) and the wrapper (no binary). The 20261001 nightly fails both GPU tests (exit -11). - This branch: 160/160 `rdci topo` runs exited 0 with all rows. - pre-commit passed. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Signed-off-by: Adam360x <Adam.pryor@amd.com>
Motivation
rdcicrashes with SIGSEGV (exit code 139, or -11 in Python harnesses) as it exits, after it has printed its results. QA reports it as an intermittentrdci discovery -l -ufailure on 8x MI300X with ROCm 10.2 nightlies. The crash comes before stdout is flushed, so a subcommand whose output is piped can lose part or all of it (rdci topo -u -i 0printed onlySegmentation faultin the same QA log). Everyrdcisubcommand and any other standalone-mode client oflibrdc_bootstrapis affected, on any GPU. Released ROCm 10.0.0 and 10.1.0 are not affected.Technical Details
Root cause:
rdc_shutdown()dlclose()slibrdc_client.so, which has gRPC 1.78.1 linked in statically. gRPC keeps detached EventEngine threads (thread-pool workers plus a "lifeguard" watchdog that wakes at least once a second) running after the channel is destroyed. In 1.78.1grpc_shutdown()does not stop the default EventEngine and these threads are created non-joinable, so they can be neither stopped nor joined before the unload. A thread that wakes after the unmap executes unmapped code. gdb on the 10.2.0-20261007 nightly: main thread insidedlclose(),lifeguardthread faulting in__tls_get_addr()for the unloaded module.STB_GNU_UNIQUEsymbol, and the GCC-built gRPC objects in it carried one (std::_Sp_make_shared_tag::_S_ti()::__tag); The ROCm 10.0.0 and 10.1.0 libraries and the nightlies checked up to 20260930 carry it, andLD_DEBUG=bindingson 10.1.0 shows glibc markinglibrdc_client.soNODELETE for it.dlclose()unmaps it. Nightlies 20260930 and 20261001 build the same rocm-systems commit (0bf70ef, the build commit both nightlies' amd-smi packages report); the 20260930rdciexits cleanly 30/30 on lsttdev6, the 20261001 one crashes 30/30.rdcicrashes 20/20 on lsttdev6. A run crashes only if a gRPC thread wakes between the unmap and process exit.Changes:
rdc_libs/bootstrap/src/RdcBootStrap.cc: loadlibrdc_client.sowithRTLD_LAZY | RTLD_NODELETE, sordc_shutdown()no longer unmaps it. This makes explicit what the unique symbol used to do by accident. A laterrdc_connect()reuses the same mapping.include/rdc_lib/RdcLibraryLoader.h: the templatedload()takes optionaldlopen_flags; the default staysRTLD_LAZY, so every other caller (librdc.soin embedded mode, the modules) is unchanged.tests/unit/test_client_shutdown.cc,tests/unit/CMakeLists.txt: newrdc_client_tests, built whenrdc_clientis. Two passes of: connect to a socket with no server, fail one RPC so gRPC starts its threads,rdc_disconnect(),rdc_shutdown(), check thatlibrdc_client.sois still loaded (RTLD_NOLOAD). Then it waits 1.5 s, longer than the watchdog's 1 s maximum sleep. Needs no GPU and nordcd; ctest pointsLD_LIBRARY_PATHat the freshly built client. A client library pinned by a unique symbol passes with or without the fix (RDC CI builds RDC and gRPC with GCC); TheRock's clang-built library fails without it.tests/ci/test_gtest_runner.py: run the installedrdc_client_testsin RDC CI.Alternatives considered:
librdc_client.sowith-z nodelete: same effect, but the dlopen flag sits next to the unload it guards and needs no build change.dlclose()inrdc_shutdown(): would also stop unloadinglibrdc.soin embedded mode, which does not need it.rdci: keeps the output but still exits 139.Embedded mode (
librdc.so, no gRPC) is unchanged. A start, field watch, stop andrdc_shutdown()cycle exits cleanly withlibrdc.sounloaded: 10/10 on MI455X with the full 10.2 nightly, 20/20 on lsttdev6. Restarting embedded mode in the same process afterrdc_shutdown()hangs on MI455X with the nightly's ownlibrdc_bootstrap.so(3/3, main thread spinning inRdcWatchTableImpl::rdc_field_watch) but not with local clang builds of it: develop, this branch, and a variant that also pinslibrdc.soare each 3/3 clean. This change neither causes nor fixes that hang; it needs its own investigation. Follow-up ticket: TBD.Issue Tracking
JIRA ID: ROCM-32420
The
rdci toposegfault reported separately on MI350X (ROCM-32219) matches this crash: pipedrdci topo -u -i 0on the stock nightly prints nothing and exits 139; with this change it prints the full table and exits 0.Test Plan
amdrocm-rdc10.2,amdrocm-amdsmi10.2,amdrocm-sysdeps10.2): the nightlyrdcd -uon a spare port, the nightlyrdciandlibrdc_client.so, and one of threelibrdc_bootstrap.sobuilds: stock (nightly), develop (this branch's parent, rebuilt) and this branch (LD_PRELOAD). Only the bootstrap differs between develop and this branch. Runs interleaved in blocks of 20, stdout redirected like the QA harness.rdcisubcommand once per build with stdout piped: lines printed and exit code.amdrocm-rdcpackages and nightlies 20260915 to 20261001, checked for the unique symbol and run against the samerdcd.rdc_client_testsbuilt from this branch against the nightly libraries, stock and fixed bootstrap; the CI wrapper run with pytest against an install tree of each.-DBUILD_TESTS=ON(GCC 11, gRPC 1.67.1),ctest -R rdc_client_tests.pre-commit run --config projects/rdc/.pre-commit-config.yaml(codespell, clang-format 22.1.0, ruff-format, gersemi 0.26.0).Test Result
rdci discovery --host localhost:<port> -l -u, SIGSEGV count:group -l0/139 -> 2/0,fieldgroup -l0/139 -> 2/0,link -l0/139 -> 19/0,topo -i 012/139 -> 12/0,discovery -i13/139 -> 13/0,discovery -l --json1/139 -> 1/0,dmon -i 0 -e 100 -c 12/139 -> 2/0. On MI455X, pipedtopo -i 0printed 0 lines (139) before and the full table after, the QA symptom;discovery -llost its closing dashes line before, as in the QA SLES log. The MI455X host's installed 10.2.0 nightlyrdcicrashed 18/20 against the samerdcd.rdc_client_testsagainst the nightly libraries: stock bootstrap 0/20 and 0/10 pass on lsttdev6 (theRTLD_NOLOADcheck fails, then SIGSEGV), 0/3 on MI350X and on MI455X; fixed bootstrap 50/50 and 10/10 on lsttdev6, 3/3 on MI350X and on MI455X. The pytest wrapper fails with rc -11 on the stock install tree and passes on the fixed one.ctest -R rdc_client_testspassed (1.5 s). pre-commit passed./var/crashon the test hosts.Submission Checklist