Repository navigation
fix(rdc): keep ROCr loaded across rdci diag checks - #13878
Merged
Merged
Conversation
rdcd aborted during "rdci diag" on AMD Instinct MI455X (gfx1250):
F0921 ... queue_controller.cpp:147] Could not find agent - 129205111608944
Each check of the ROCr diagnostic module (Compute Queue ready, System
memory check) builds test objects that call hsa_init() when they are
built and hsa_shut_down() when they are destroyed. Nothing else held a
reference, so every test object unloaded the ROCr runtime and the next
one loaded it again. rdcd also loads rocprofiler-sdk, because the
profiler module registers a tool with it, and rocprofiler-sdk keeps the
agents and queues of the first ROCr load it sees. After a reload its
queue interception no longer knows the GPU agent: depending on the
rocprofiler-sdk version and on where the new agents land in memory, the
next queue creation aborts rdcd ("Could not find agent"), queue creation
fails with HSA_STATUS_ERROR_OUT_OF_RESOURCES, or the next profiler field
read aborts rdcd ("HSA Queue is not initialized"). With ROCr's
rocprofiler registration turned off (HSA_TOOLS_DISABLE_REGISTER=1) the
same runs are clean.
- The module takes one hsa_init() reference at its first check and
releases it in rdc_diag_destroy(), so the test objects' hsa_init() and
hsa_shut_down() calls only move the reference count. A failed
hsa_init() is retried at the next check.
- The checks share one queue per GPU, which the module creates at the
first check that needs it and destroys with its reference, and the
module runs one check at a time. rocprofiler-sdk does not destroy a
queue it intercepted when the application destroys it, so with ROCr
kept loaded every check's own queue stayed in rdcd: on MI455X with
ROCm 10.2.0 each run added two queues and about 70 MB of VRAM, and
after 126 runs queue creation failed and the next profiler field read
aborted rdcd.
- With ROCr no longer unloaded after each check, nothing else releases
what a check leaves behind. Both checks now release their signals,
memory and loaded code objects on every path, including when they
fail or are skipped; before, a skipped check left its queue and memory
allocated.
- RDC shipped no diagnostic kernels for gfx1250, so both checks reported
Skip there. Add them, built as code object v4: the checks fill in the
kernel arguments of the v4 layout, and a later code object version
takes the work-group size from hidden arguments the checks never
write, which fails the System memory check's comparison.
Tests:
- rdc_rocr_lifetime_tests (new, GPU-less): compiles the module against
fake HSA runtime calls and checks that repeated checks load ROCr once
and create one queue, that rdc_diag_destroy() releases both, that a
failed hsa_init() is retried, and that skipped, passing and failing
checks leave no signal, memory or code object behind. Three of its
four tests fail on the old module. It runs in the rdc_test CI job
through test_gtest_runner.py.
- rdc_diag_kernel_tests checks that the gfx1250 kernels ship and are
code object v4.
Signed-off-by: Adam360x <Adam.pryor@amd.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Initialization failure can still trigger ROCr reloads, and asynchronous copy failures can leak signals.
2 open issues
What changed in this PR
This PR stabilizes RDC diagnostics by retaining ROCr between checks, reusing GPU queues, and improving resource cleanup.
Changes:
- Retain the runtime and serialize diagnostics using shared per-GPU queues.
- Release diagnostic resources on success and failure paths.
- Add lifetime regression tests, CI integration, and gfx1250 kernel checks.
| File | Description |
|---|---|
| projects/rdc/tests/unit/test_rocr_lifetime.cc | Adds GPU-less lifetime and cleanup tests. |
| projects/rdc/tests/unit/test_diag_kernels.cc | Checks gfx1250 kernel packaging and version. |
| projects/rdc/tests/unit/CMakeLists.txt | Builds and installs lifetime tests. |
| projects/rdc/tests/ci/test_gtest_runner.py | Runs lifetime tests in CI. |
| projects/rdc/rdc_libs/rdc_modules/rdc_rocr/RdcRocrBase.cc | Initializes owned kernel handles. |
| projects/rdc/rdc_libs/rdc_modules/rdc_rocr/RdcDiagnosticLib.cc | Retains ROCr, shares queues, and serializes checks. |
| projects/rdc/rdc_libs/rdc_modules/rdc_rocr/MemoryAccess.cc | Adds scoped cleanup and queue reuse. |
| projects/rdc/rdc_libs/rdc_modules/rdc_rocr/ComputeQueueTest.cc | Improves cleanup and reuses module queues. |
| projects/rdc/rdc_libs/rdc_modules/rdc_rocr/base_rocr_utils.cc | Tracks and releases loaded kernel resources. |
| projects/rdc/include/rdc_modules/rdc_rocr/RdcRocrBase.h | Adds kernel-resource ownership fields. |
| projects/rdc/include/rdc_modules/rdc_rocr/ComputeQueueTest.h | Tracks executable and reader handles. |
| projects/rdc/include/rdc_modules/rdc_rocr/common.h | Adds scope guard and shared-queue declaration. |
| projects/rdc/CHANGELOG.md | Documents fixes and runtime-lifetime trade-offs. |
Review effort: Balanced
bill-shuzhou-liu
approved these changes
Oct 22, 2026
Follow-ups for the ROCr diagnostic module's lifetime fix: - When the module's own hsa_init() failed, the checks still ran and called hsa_init() themselves. If a check's call succeeded, that check could hold the last reference and unload ROCr when it ended, so the next run loaded it again: the reload that rocprofiler-sdk cannot handle. A run now fails with "ROCr initialization failed" before it builds a check, and the next run tries again. - The Compute Queue check's memory copy destroyed its signal only when the copy did not start. When a started copy fails, the runtime sets the signal negative and the check threw without destroying it. With ROCr kept loaded nothing else releases it, so destroy it there too. - Remove CreateQueue(), which nothing calls. Checks take their queue from AcquireDiagQueue(): rocprofiler-sdk never frees a queue it intercepted, so a queue per check stays in rdcd. rdc_rocr_lifetime_tests: the failed-init test now requires that the failed run loads, unloads and shuts down nothing and that the retry loads ROCr once; a new test fails a started copy and requires that its signal is destroyed. Each fails against the previous version of the file it covers. Signed-off-by: Adam360x <Adam.pryor@amd.com>
This was referenced Oct 27, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
QA reported that
rdcdaborts duringrdci diag -u -r 2 -g 1on AMD Instinct MI455X (gfx1250) with ROCm build #265:The checks of the ROCr diagnostic module (
librdc_rocr.so: Compute Queue ready, System memory check) build test objects that callhsa_init()andhsa_shut_down()themselves. Nothing else held a reference, so ROCr was unloaded and loaded again between them.rdcdalways loads rocprofiler-sdk (the profiler module registers a tool at the firsthsa_init(), even withRDC_DISABLE_ROCP=1), and rocprofiler-sdk does not support a ROCr reload. Depending on its version, the next queue creation abortsrdcd(Could not find agent) or fails, or the next profiler field read aborts it (HSA Queue is not initialized).ROCm 10.2.0 and develop are affected too, and not only on MI455X: with a 10.2.0 nightly on MI350X the checks pass, but the next profiler field read aborts
rdcd.Keeping ROCr loaded exposed a second rocprofiler-sdk problem: it never destroys a queue it intercepted. With a queue per check, each
rdci diagrun left two queues and about 70 MB of VRAM inrdcd, until queue creation failed after 126 runs (MI455X allows 254 queues per process). So the checks now share one queue per GPU.Technical Details
RdcDiagnosticLib.cc): the module takes onehsa_init()reference at its first check and releases it inrdc_diag_destroy(); the test objects' calls only move the count. If thathsa_init()fails, the run fails (ROCr initialization failed) before any check is built, and the next run retries.AcquireDiagQueue()creates one 128-packet queue per GPU, destroyed byrdc_diag_destroy()(the System memory check used to create a 131072-packet queue per run). A mutex runs one check at a time, since they share the queues (andMemoryAccess.cca global kernel-argument pointer). The unusedCreateQueue()helper is removed.ComputeQueueTest.cc,MemoryAccess.cc,base_rocr_utils.cc,RdcRocrBase.*): both checks release their signals, memory, file descriptors and loaded code objects on every path, including Skip and failure (ScopeExitguard incommon.h). A copy that fails after it started no longer leaks its signal.clang -x cl -Xclang -finclude-default-header -target amdgcn-amd-amdhsa -mcpu=gfx1250 -mcode-object-version=4 -O3), because the checks fill in the v4 kernel arguments; the default v6 fails the System memory check.rdc_rocr_lifetime_testsruns the module against fake HSA calls that count loads, references and objects: ROCr is loaded once, one queue per GPU, a failedhsa_init()loads nothing and is retried, and no check leaves objects behind. It is built with the module, installed with the RDC test package, and run in RDC CI bytests/ci/test_gtest_runner.py.rdc_diag_kernel_testsrequires v4 gfx1250 kernels.CHANGELOG.mdentries. No public API, proto or struct changes (the changed headers are not installed).Trade-off: after the first
rdci diag,rdcdkeeps ROCr and one queue per GPU until it stops (MI455X: 3 KFD queues and about 149 MB of VRAM, flat across runs after the first), as it already does after the first profiler read. Changing the GPU partition mode then needsrdcdstopped, and a secondrdci diagwaits for the first.#13570 and #13874 also edit
CHANGELOG.md; trial merges show the three PRs merge cleanly in any order.Not in this PR:
rocprofv3 --pmc.rdcdon the Compute Queue check's existing kernel-argument size assertion. v4 rebuilds match and pass on MI350X.Issue Tracking
JIRA ID: ROCM-31379
Test Plan
rdcd -uin a private network namespace: Nrdci diag -u -r 2 -g 1runs, a profilerrdci dmon(fields 805/806), one more diag; also profiler first. KFD queues and VRAM from/sys/class/kfd/kfd/proc/<pid>/. The module is built from this branch against each ROCm.rdc_rocr_lifetime_testsandrdc_diag_kernel_tests(GCC 11) on this branch, against develop's module, and with each part of the fix removed; the CI wrapper;pre-commit.Test Result
rdci diagrunsrdcdexitCould not find agent)HSA_TOOLS_DISABLE_REGISTER=1(rocprofiler-sdk not registered with ROCr). On the BKC buildrdciitself exits with SIGSEGV after printing (the client exit crash fixed on develop by fix(rdc): keep librdc_client loaded so rdci does not crash on exit #13296), which does not affectrdcd.rdcd(HSA Queue is not initialized, exit 134). This PR (release build, shipped kernels): 4/4 on GPU 0 and 11/11 on all 8 GPUs, every GPU passing both checks, including a run after the profiler read; 3/3 on all 8 GPUs with the profiler read first. KFD queues stay flat across runs (24 with all 8 GPUs) andrdcdexits 0.rdc_rocr_lifetime_tests5/5 pass and 4 of 5 fail against develop's module; each of 14 mutants of the fix, and reverting either review fix, makes a test fail. The mutex is not unit-tested.rdc_diag_kernel_tests, the CI wrapper and pre-commit pass.Submission Checklist