Skip to content

fix(rdc): stop rdci crashing with SIGSEGV on exit - #13298

Merged
adam360x merged 3 commits into
developfrom
users/adam360x/ROCM-32420
Oct 13, 2026
Merged

adam360x merged 3 commits into
developfrom
users/adam360x/ROCM-32420

Conversation

@adam360x

@adam360x adam360x commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

rdci crashes with SIGSEGV (exit code 139, or -11 in Python harnesses) as it exits, after it has printed its results. QA reports it as an intermittent rdci discovery -l -u failure on 8x MI300X with ROCm 10.2 nightlies. The crash comes before stdout is flushed, so a subcommand whose output is piped can lose part or all of it (rdci topo -u -i 0 printed only Segmentation fault in the same QA log). Every rdci subcommand and any other standalone-mode client of librdc_bootstrap is affected, on any GPU. Released ROCm 10.0.0 and 10.1.0 are not affected.

Technical Details

Root cause:

  • rdc_shutdown() dlclose()s librdc_client.so, which has gRPC 1.78.1 linked in statically. gRPC keeps detached EventEngine threads (thread-pool workers plus a "lifeguard" watchdog that wakes at least once a second) running after the channel is destroyed. In 1.78.1 grpc_shutdown() does not stop the default EventEngine and these threads are created non-joinable, so they can be neither stopped nor joined before the unload. A thread that wakes after the unmap executes unmapped code. gdb on the 10.2.0-20261007 nightly: main thread inside dlclose(), lifeguard thread faulting in __tls_get_addr() for the unloaded module.
  • Until recently the library was never actually unloaded. glibc never unloads a library that has an STB_GNU_UNIQUE symbol, and the GCC-built gRPC objects in it carried one (std::_Sp_make_shared_tag::_S_ti()::__tag); The ROCm 10.0.0 and 10.1.0 libraries and the nightlies checked up to 20260930 carry it, and LD_DEBUG=bindings on 10.1.0 shows glibc marking librdc_client.so NODELETE for it.
  • TheRock builds gRPC with amd-llvm since grpc: build with amd-llvm and add ppc64le support TheRock#8423 (2026-09-30). From nightly 20261001 on, the library has no unique symbol and dlclose() unmaps it. Nightlies 20260930 and 20261001 build the same rocm-systems commit (0bf70ef, the build commit both nightlies' amd-smi packages report); the 20260930 rdci exits cleanly 30/30 on lsttdev6, the 20261001 one crashes 30/30.
  • QA's "PASS on 20261005, FAIL on 20261006 and 20261007" is chance: the RDC libraries are byte-identical in those three nightlies, and the 20261005 rdci crashes 20/20 on lsttdev6. A run crashes only if a gRPC thread wakes between the unmap and process exit.

Changes:

  • rdc_libs/bootstrap/src/RdcBootStrap.cc: load librdc_client.so with RTLD_LAZY | RTLD_NODELETE, so rdc_shutdown() no longer unmaps it. This makes explicit what the unique symbol used to do by accident. A later rdc_connect() reuses the same mapping.
  • include/rdc_lib/RdcLibraryLoader.h: the templated load() takes optional dlopen_flags; the default stays RTLD_LAZY, so every other caller (librdc.so in embedded mode, the modules) is unchanged.
  • tests/unit/test_client_shutdown.cc, tests/unit/CMakeLists.txt: new rdc_client_tests, built when rdc_client is. Two passes of: connect to a socket with no server, fail one RPC so gRPC starts its threads, rdc_disconnect(), rdc_shutdown(), check that librdc_client.so is still loaded (RTLD_NOLOAD). Then it waits 1.5 s, longer than the watchdog's 1 s maximum sleep. Needs no GPU and no rdcd; ctest points LD_LIBRARY_PATH at the freshly built client. A client library pinned by a unique symbol passes with or without the fix (RDC CI builds RDC and gRPC with GCC); TheRock's clang-built library fails without it.
  • tests/ci/test_gtest_runner.py: run the installed rdc_client_tests in RDC CI.
  • No CHANGELOG entry: no released ROCm has the crash.

Alternatives considered:

  • Linking librdc_client.so with -z nodelete: same effect, but the dlopen flag sits next to the unload it guards and needs no build change.
  • Never calling dlclose() in rdc_shutdown(): would also stop unloading librdc.so in embedded mode, which does not need it.
  • Shutting gRPC down before the unload: not possible with 1.78.1 (see above).
  • Flushing stdout in rdci: keeps the output but still exits 139.

Embedded mode (librdc.so, no gRPC) is unchanged. A start, field watch, stop and rdc_shutdown() cycle exits cleanly with librdc.so unloaded: 10/10 on MI455X with the full 10.2 nightly, 20/20 on lsttdev6. Restarting embedded mode in the same process after rdc_shutdown() hangs on MI455X with the nightly's own librdc_bootstrap.so (3/3, main thread spinning in RdcWatchTableImpl::rdc_field_watch) but not with local clang builds of it: develop, this branch, and a variant that also pins librdc.so are each 3/3 clean. This change neither causes nor fixes that hang; it needs its own investigation. Follow-up ticket: TBD.

Issue Tracking

JIRA ID: ROCM-32420

The rdci topo segfault reported separately on MI350X (ROCM-32219) matches this crash: piped rdci topo -u -i 0 on the stock nightly prints nothing and exits 139; with this change it prints the full table and exits 0.

Test Plan

  • A/B on hardware with the 10.2.0-20261007 nightly packages (amdrocm-rdc10.2, amdrocm-amdsmi10.2, amdrocm-sysdeps10.2): the nightly rdcd -u on a spare port, the nightly rdci and librdc_client.so, and one of three librdc_bootstrap.so builds: stock (nightly), develop (this branch's parent, rebuilt) and this branch (LD_PRELOAD). Only the bootstrap differs between develop and this branch. Runs interleaved in blocks of 20, stdout redirected like the QA harness.
  • Each rdci subcommand once per build with stdout piped: lines printed and exit code.
  • Releases and bisect: released 10.0.0 and 10.1.0 amdrocm-rdc packages and nightlies 20260915 to 20261001, checked for the unique symbol and run against the same rdcd.
  • rdc_client_tests built from this branch against the nightly libraries, stock and fixed bootstrap; the CI wrapper run with pytest against an install tree of each.
  • RDC configure and build with -DBUILD_TESTS=ON (GCC 11, gRPC 1.67.1), ctest -R rdc_client_tests.
  • pre-commit run --config projects/rdc/.pre-commit-config.yaml (codespell, clang-format 22.1.0, ruff-format, gersemi 0.26.0).

Test Result

rdci discovery --host localhost:<port> -l -u, SIGSEGV count:

Host stock develop this PR
lsttdev6, RX 6800 XT + RX 7900 XT, Ubuntu 22.04 230/230 30/30 (GCC), 50/50 (clang) 0/100 (GCC), 0/50 (clang)
smci350-rck-g03-b20-31, 8x MI350X, Ubuntu 24.04 200/200 200/200 0/200
heliosr-1b114-a06-3, MI455X, Ubuntu 26.04 198/200 195/200 0/200
same MI455X host, no server listening (error exit) 101/400 120/400 0/400 (all exit 8, the normal error code)
  • Releases and nightlies (lsttdev6, unique symbol present / crashes): 10.0.0 yes / 0 of 30, 10.1.0 yes / 0 of 30, nightlies 20260915, 20260925, 20260927, 20260929 and 20260930 yes (20260930: 0 of 30), 20261001 no / 30 of 30, 20261005 to 20261007 no.
  • Subcommands on MI350X, stock -> this PR (lines printed, exit code): group -l 0/139 -> 2/0, fieldgroup -l 0/139 -> 2/0, link -l 0/139 -> 19/0, topo -i 0 12/139 -> 12/0, discovery -i 13/139 -> 13/0, discovery -l --json 1/139 -> 1/0, dmon -i 0 -e 100 -c 1 2/139 -> 2/0. On MI455X, piped topo -i 0 printed 0 lines (139) before and the full table after, the QA symptom; discovery -l lost its closing dashes line before, as in the QA SLES log. The MI455X host's installed 10.2.0 nightly rdci crashed 18/20 against the same rdcd.
  • rdc_client_tests against the nightly libraries: stock bootstrap 0/20 and 0/10 pass on lsttdev6 (the RTLD_NOLOAD check fails, then SIGSEGV), 0/3 on MI350X and on MI455X; fixed bootstrap 50/50 and 10/10 on lsttdev6, 3/3 on MI350X and on MI455X. The pytest wrapper fails with rc -11 on the stock install tree and passes on the fixed one.
  • Local build: ctest -R rdc_client_tests passed (1.5 s). pre-commit passed.
  • No crash reports left in /var/crash on the test hosts.

Submission Checklist

rdci (and any standalone-mode client) crashed after printing its
results on TheRock builds. Commands whose output was piped could lose
all of it, since the crash came before stdout was flushed.

- Load librdc_client.so with RTLD_NODELETE. gRPC keeps detached
  EventEngine threads (thread-pool workers and their watchdog) running
  after the channel is destroyed, and rdc_shutdown() dlclose()d the
  library, so the next wakeup executed unmapped code.
- GCC builds never unloaded the library: their STB_GNU_UNIQUE symbols
  make glibc pin it. TheRock builds with clang, which emits none, and
  links gRPC statically into librdc_client.so.
- Add rdc_client_tests: connect to an address with no server, fail one
  RPC, shut down, then check that the library is still loaded and that
  the process outlives the watchdog's next wakeup.

Signed-off-by: Adam360x <Adam.pryor@amd.com>
@adam360x
adam360x requested review from a team as code owners October 12, 2026 19:09
Copilot AI balanced review requested due to automatic review settings October 12, 2026 19:09
@therock-pr-bot

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused lifecycle fix preserves existing defaults and includes targeted regression coverage.

0 open findings

What changed in this PR

Prevents standalone RDC clients from crashing during shutdown by keeping the statically linked gRPC client library mapped.

Changes:

  • Adds configurable dlopen flags and uses RTLD_NODELETE for librdc_client.so.
  • Adds a shutdown regression test and build configuration.
  • Documents the resolved rdci crash.
File Description
projects/​rdc/​CHANGELOG.md Records the resolved crash.
projects/​rdc/​include/​rdc_lib/​RdcLibraryLoader.h Supports custom dlopen flags.
projects/​rdc/​rdc_libs/​bootstrap/​src/​RdcBootStrap.cc Keeps the client library mapped after shutdown.
projects/​rdc/​tests/​unit/​CMakeLists.txt Builds and registers the regression test.
projects/​rdc/​tests/​unit/​test_client_shutdown.cc Verifies safe shutdown and library retention.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- Fail instead of skipping when librdc_client.so cannot be loaded:
  the test is built only with the client library, so a load failure
  means a broken setup.
- Reconnect after rdc_shutdown() in a second pass, wait once at the
  end, and name the wait after the watchdog wakeup it covers.
- Say that a client library pinned by an STB_GNU_UNIQUE symbol passes
  with or without the fix; only clang-only builds can fail.
- Run the installed rdc_client_tests from the RDC CI pytest suite.
- Drop the CHANGELOG entry. Released ROCm 10.0.0 and 10.1.0 do not
  crash: their client library has a GCC-built gRPC object whose unique
  symbol keeps it loaded. The crash exists only in 10.2 development
  builds made after gRPC moved to clang.

Signed-off-by: Adam360x <Adam.pryor@amd.com>
- Reword the STB_GNU_UNIQUE note; the spell-fix hook had turned
  "statics" into "statistics".
- Say why the CI wrapper fails rather than skips when rdc_client_tests
  is missing: rdctst and rdc_client_tests are both standalone-only.

Signed-off-by: Adam360x <Adam.pryor@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The targeted lifetime fix preserves embedded-mode behavior and includes focused regression and package-level coverage.

0 open findings

🧠 Review effort: Balanced

@adam360x

Copy link
Copy Markdown
Contributor Author

Same root cause as #13296 (ROCM-32616), which came out of a parallel investigation of the same crash. Both PRs keep librdc_client.so loaded after rdc_shutdown(), so only one of them should merge:

This PR also carries the hardware A/B results (8x MI350X, MI455X), the check that released ROCm 10.0.0 and 10.1.0 are not affected, and the related tickets ROCM-32420 and ROCM-32219.

@adam360x
adam360x merged commit 49b5618 into develop Oct 13, 2026
76 of 78 checks passed
@adam360x
adam360x deleted the users/adam360x/ROCM-32420 branch October 13, 2026 15:36
@systems-assistant

Copy link
Copy Markdown
Contributor

TheRock Submodule Bump Activity

Newest first

adam360x added a commit that referenced this pull request Oct 14, 2026
## Motivation

QA reports that `rdci link -u --xgmi-link-status` segfaults on 8x MI300X
with TheRock 10.2.0-20261001 and -20261008 and prints no link table, and
points at the uninitialized `amdsmi_xgmi_link_status_t` in
`rdc_link_status_get()`.

The reported crash is the `rdci` exit crash that #13296 and #13298 fixed
on develop (merged 2026-10-13). `rdci link` prints its table with `\n`
and no flush, so when stdout is a pipe, as in the QA harness, the crash
at exit discards the whole table. The amdsmi call the ticket suspects
succeeded on the QA host: `amd-smi xgmi -l` makes the same call for each
GPU and prints N/A for a GPU whose call fails, and it printed D for
every link.

The code path the ticket points at is still wrong, and this PR fixes it.
`amdsmi_get_gpu_xgmi_link_status()` can fail without writing its output
(when it cannot read the GPU metrics or the sockets) or after writing
part of it (`AMDSMI_STATUS_UNEXPECTED_DATA`). RDC copied the struct
anyway:

- A GPU whose query failed reported the previous GPU's links, left on
the stack.
- Any other leftover link count was copied as is. Above 8 it reads past
the end of amdsmi's `status[8]`, and above 16 it writes past
`link_states[16]`. With an injected count of 4096, `rdcd` crashed (see
Test Result).

`rdci link` also hid failures. It printed nothing and exited 0 when it
could not get the link status, for example when `rdcd` was down or had
crashed. `rdci discovery` and `rdci topo` exit 8 in the same case.
adam360x added a commit that referenced this pull request Oct 21, 2026
## Motivation

QA reported that `rdci topo -u -i 6` and `-i 7` print the XGMI table and
then crash with SIGSEGV on 8x MI350X with the ROCm 10.2.0 nightly
20261001, and that the test harness, which pipes stdout, captured empty
output for GPU 7.

That crash is the `rdci` exit crash that #13296 and #13298 fixed on
develop (merged 2026-10-13). It is not specific to GPUs 6 and 7 (see
Test Result), and this PR does not touch it.

Verifying it showed that `rdci topo` reports the links of the wrong
GPUs. `rdc_device_topology_get()` ignored the GPU index when it filled
in the links: for every GPU i it looped over all peers and overwrote
`link_infos[i]` each time, so entry i ended up holding the link from GPU
i to the last GPU. `rdci topo` prints entry i as the link to GPU i. On a
fully connected XGMI mesh, such as the MI350X host, every link is the
same and the table looks right. On a system whose GPU pairs are not all
linked the same way, for example two XGMI hives joined by PCIe, the
table is wrong. `rdci topo` also exits 0 when the GPU index is out of
range (after printing an error) and when the topology query fails (after
printing nothing).

## Technical Details

- `rdc_libs/rdc/src/RdcTopologyLinkImpl.cc`: `rdc_device_topology_get()`
fills `link_infos[i]` with the link from the requested GPU to GPU i,
making the four AMD SMI link queries once per peer instead of once per
GPU pair. The requested GPU's own entry gets N/A values: the max values
that `rdc.h` documents for N/A data, and an undefined link type. A GPU
count above `RDC_MAX_NUM_DEVICES` now returns `RDC_ST_MAX_LIMIT`. Only
an `assert` guarded it before, which Release builds drop, so a larger
count would have overrun a 128-entry stack array and the result. A GPU
whose handle lookup fails now returns `RDC_ST_NOT_FOUND` instead of
`RDC_ST_OK`.
- `RdcTopologyLinkImpl.cc`: link types RDC does not define (AMD SMI's
not applicable, unknown, NUMA and XNUMA) and failed lookups are
`RDC_IOLINK_TYPE_UNDEFINED`; the cast passed values 3-6 to API and gRPC
callers. `rdc.h` documents what `rdc_device_topology_get()` fills in,
the requested GPU's own entry and its statuses.
- `rdci/src/RdciTopologyLinkSubSystem.cc`: an out-of-range `-i` exits
with `RDC_ST_BAD_PARAMETER` (5) instead of 0. A failed topology query
prints `rdci Error: Fail to get the topology of GPU <N>` and exits with
its status instead of printing nothing and exiting 0.
- `tests/unit/test_topology_link.cc`, `tests/unit/CMakeLists.txt`: new
GPU-less `rdc_topology_tests`. librdc cannot be loaded without a GPU (it
initializes AMD SMI when loaded), so the test builds
`RdcTopologyLinkImpl.cc` itself against fake AMD SMI calls that model
two XGMI hives joined by PCIe. It links only `librdc_bootstrap` (for the
logger) and gtest, and is installed with the RDC test package.
- `tests/ci/test_gtest_runner.py`: run `rdc_topology_tests` in RDC CI.
`tests/ci/test_rdc_core.py`: GPU tests that `rdci topo -i N` exits 0 and
lists every other GPU for each GPU, and that an out-of-range index exits
nonzero with an error.
- `CHANGELOG.md`: entry under Resolved Issues.

No proto or struct changes.

#13384 fixes `rdc_link_status_get()` in the same file and adds a similar
test binary. The two PRs change different functions and merge cleanly in
either order; both test binaries pass on the merged code.

## Issue Tracking

JIRA ID: ROCM-32219

The crash in this ticket is fixed by #13296 and #13298. This PR fixes
the `rdci topo` output and exit codes found while verifying that fix.

## Test Plan

- Ticket flow on 8x MI350X (Ubuntu 24.04; the QA host was not reachable)
with the full 10.2.0a20261001 nightly tarball: its `rdcd -u`, then `rdci
topo -u -i N` for every GPU, 20 runs with stdout piped as in the QA
harness and 20 on a pty, with the nightly `rdci` and with `rdci` built
from develop (TheRock-like build: amd-llvm, static gRPC 1.78.1).
- Same host, TheRock-like builds of develop and of this branch (`rdcd`,
`librdc`, `rdci`): `rdci topo -u -i 0` to `-i 7` with and without an
`LD_PRELOAD` shim in `rdcd` that makes `amdsmi_topo_get_link_type()`
report GPUs 0-3 and GPUs 4-7 as two XGMI hives joined by PCIe, and `rdci
topo -u -i 8`.
- `rdc_topology_tests` against this branch and against develop's
`RdcTopologyLinkImpl.cc` (GCC 11, plain and ASan+UBSan), and the
installed amd-llvm binary from `share/rdc/rdctst_tests`.
- `pytest tests/ci/test_rdc_core.py::TestTopology
tests/ci/test_gtest_runner.py::TestGoogleTestSuite::test_rdc_topology`,
run as root with `ROCM_DIR` at the develop install, this branch's
install, and the 20261001 nightly.
- This branch: `rdci topo -u -i N` for every GPU, 10 piped and 10 pty
runs each.
- `pre-commit run --config projects/rdc/.pre-commit-config.yaml`
(codespell, clang-format, ruff-format, gersemi).

## Test Result

- Ticket flow: the 20261001 nightly exited 139 in 320/320 runs on all 8
GPUs, with empty piped output in 159/160; develop's `rdci` exited 0 with
all 7 rows in 320/320.
- Two-hive shim:

| `rdcd` | `rdci topo -i 0` | `-i 1` to `-i 3` | `-i 4` to `-i 7` |
|---|---|---|---|
| develop | links to GPUs 1-3 PCIe, to GPUs 4-7 XGMI (inverted) | wrong
| right by coincidence |
| this PR | links to GPUs 1-3 XGMI, to GPUs 4-7 PCIe | right | right |

Without the shim both builds report XGMI for every link, as `amd-smi
topology` does. A `rdci topo` call takes 7-18 ms with either build (wall
time of the `rdci` process, including connect). `rdci topo -u -i 8`
exits 0 with develop and 5 with this PR.
- `rdc_topology_tests`: 4/4 pass with this PR (no ASan or UBSan reports;
the installed amd-llvm binary also passes). With the old link-type cast,
the failed-lookup test fails (type 4 instead of 0). Against develop's
code 2 of 4 fail (rows hold other GPU pairs' links, and a failed lookup
returns `RDC_ST_OK`), and the 129-GPU test hits an ASan
stack-buffer-overflow in the old index array.
- pytest: this PR 3 passed. Develop fails the out-of-range test (exit 0)
and the wrapper (no binary). The 20261001 nightly fails both GPU tests
(exit -11).
- This branch: 160/160 `rdci topo` runs exited 0 with all rows.
- pre-commit passed.

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

---------

Signed-off-by: Adam360x <Adam.pryor@amd.com>
dgaliffiAMD pushed a commit that referenced this pull request Oct 21, 2026
## Motivation

QA reported that `rdci topo -u -i 6` and `-i 7` print the XGMI table and
then crash with SIGSEGV on 8x MI350X with the ROCm 10.2.0 nightly
20261001, and that the test harness, which pipes stdout, captured empty
output for GPU 7.

That crash is the `rdci` exit crash that #13296 and #13298 fixed on
develop (merged 2026-10-13). It is not specific to GPUs 6 and 7 (see
Test Result), and this PR does not touch it.

Verifying it showed that `rdci topo` reports the links of the wrong
GPUs. `rdc_device_topology_get()` ignored the GPU index when it filled
in the links: for every GPU i it looped over all peers and overwrote
`link_infos[i]` each time, so entry i ended up holding the link from GPU
i to the last GPU. `rdci topo` prints entry i as the link to GPU i. On a
fully connected XGMI mesh, such as the MI350X host, every link is the
same and the table looks right. On a system whose GPU pairs are not all
linked the same way, for example two XGMI hives joined by PCIe, the
table is wrong. `rdci topo` also exits 0 when the GPU index is out of
range (after printing an error) and when the topology query fails (after
printing nothing).

## Technical Details

- `rdc_libs/rdc/src/RdcTopologyLinkImpl.cc`: `rdc_device_topology_get()`
fills `link_infos[i]` with the link from the requested GPU to GPU i,
making the four AMD SMI link queries once per peer instead of once per
GPU pair. The requested GPU's own entry gets N/A values: the max values
that `rdc.h` documents for N/A data, and an undefined link type. A GPU
count above `RDC_MAX_NUM_DEVICES` now returns `RDC_ST_MAX_LIMIT`. Only
an `assert` guarded it before, which Release builds drop, so a larger
count would have overrun a 128-entry stack array and the result. A GPU
whose handle lookup fails now returns `RDC_ST_NOT_FOUND` instead of
`RDC_ST_OK`.
- `RdcTopologyLinkImpl.cc`: link types RDC does not define (AMD SMI's
not applicable, unknown, NUMA and XNUMA) and failed lookups are
`RDC_IOLINK_TYPE_UNDEFINED`; the cast passed values 3-6 to API and gRPC
callers. `rdc.h` documents what `rdc_device_topology_get()` fills in,
the requested GPU's own entry and its statuses.
- `rdci/src/RdciTopologyLinkSubSystem.cc`: an out-of-range `-i` exits
with `RDC_ST_BAD_PARAMETER` (5) instead of 0. A failed topology query
prints `rdci Error: Fail to get the topology of GPU <N>` and exits with
its status instead of printing nothing and exiting 0.
- `tests/unit/test_topology_link.cc`, `tests/unit/CMakeLists.txt`: new
GPU-less `rdc_topology_tests`. librdc cannot be loaded without a GPU (it
initializes AMD SMI when loaded), so the test builds
`RdcTopologyLinkImpl.cc` itself against fake AMD SMI calls that model
two XGMI hives joined by PCIe. It links only `librdc_bootstrap` (for the
logger) and gtest, and is installed with the RDC test package.
- `tests/ci/test_gtest_runner.py`: run `rdc_topology_tests` in RDC CI.
`tests/ci/test_rdc_core.py`: GPU tests that `rdci topo -i N` exits 0 and
lists every other GPU for each GPU, and that an out-of-range index exits
nonzero with an error.
- `CHANGELOG.md`: entry under Resolved Issues.

No proto or struct changes.

#13384 fixes `rdc_link_status_get()` in the same file and adds a similar
test binary. The two PRs change different functions and merge cleanly in
either order; both test binaries pass on the merged code.

## Issue Tracking

JIRA ID: ROCM-32219

The crash in this ticket is fixed by #13296 and #13298. This PR fixes
the `rdci topo` output and exit codes found while verifying that fix.

## Test Plan

- Ticket flow on 8x MI350X (Ubuntu 24.04; the QA host was not reachable)
with the full 10.2.0a20261001 nightly tarball: its `rdcd -u`, then `rdci
topo -u -i N` for every GPU, 20 runs with stdout piped as in the QA
harness and 20 on a pty, with the nightly `rdci` and with `rdci` built
from develop (TheRock-like build: amd-llvm, static gRPC 1.78.1).
- Same host, TheRock-like builds of develop and of this branch (`rdcd`,
`librdc`, `rdci`): `rdci topo -u -i 0` to `-i 7` with and without an
`LD_PRELOAD` shim in `rdcd` that makes `amdsmi_topo_get_link_type()`
report GPUs 0-3 and GPUs 4-7 as two XGMI hives joined by PCIe, and `rdci
topo -u -i 8`.
- `rdc_topology_tests` against this branch and against develop's
`RdcTopologyLinkImpl.cc` (GCC 11, plain and ASan+UBSan), and the
installed amd-llvm binary from `share/rdc/rdctst_tests`.
- `pytest tests/ci/test_rdc_core.py::TestTopology
tests/ci/test_gtest_runner.py::TestGoogleTestSuite::test_rdc_topology`,
run as root with `ROCM_DIR` at the develop install, this branch's
install, and the 20261001 nightly.
- This branch: `rdci topo -u -i N` for every GPU, 10 piped and 10 pty
runs each.
- `pre-commit run --config projects/rdc/.pre-commit-config.yaml`
(codespell, clang-format, ruff-format, gersemi).

## Test Result

- Ticket flow: the 20261001 nightly exited 139 in 320/320 runs on all 8
GPUs, with empty piped output in 159/160; develop's `rdci` exited 0 with
all 7 rows in 320/320.
- Two-hive shim:

| `rdcd` | `rdci topo -i 0` | `-i 1` to `-i 3` | `-i 4` to `-i 7` |
|---|---|---|---|
| develop | links to GPUs 1-3 PCIe, to GPUs 4-7 XGMI (inverted) | wrong
| right by coincidence |
| this PR | links to GPUs 1-3 XGMI, to GPUs 4-7 PCIe | right | right |

Without the shim both builds report XGMI for every link, as `amd-smi
topology` does. A `rdci topo` call takes 7-18 ms with either build (wall
time of the `rdci` process, including connect). `rdci topo -u -i 8`
exits 0 with develop and 5 with this PR.
- `rdc_topology_tests`: 4/4 pass with this PR (no ASan or UBSan reports;
the installed amd-llvm binary also passes). With the old link-type cast,
the failed-lookup test fails (type 4 instead of 0). Against develop's
code 2 of 4 fail (rows hold other GPU pairs' links, and a failed lookup
returns `RDC_ST_OK`), and the 129-GPU test hits an ASan
stack-buffer-overflow in the old index array.
- pytest: this PR 3 passed. Develop fails the out-of-range test (exit 0)
and the wrapper (no binary). The 20261001 nightly fails both GPU tests
(exit -11).
- This branch: 160/160 `rdci topo` runs exited 0 with all rows.
- pre-commit passed.

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

---------

Signed-off-by: Adam360x <Adam.pryor@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants