Repository navigation
Conversation
A new job, rdc_sanitize, has two legs. One builds RDC with GCC's -fsanitize=address and the other with -fsanitize=undefined, and each runs the installed gtests that need no GPU (all but rdc_diag_kernel_tests, which has no install RPATH), the fake-GPU suite and tests/ci against its build. GCC links the sanitizer runtime as a shared library, so rdcd, rdci, the gtests, the fake amd-smi and Python, which loads RDC's libraries through ctypes and gets the runtime through LD_PRELOAD, share one runtime. Every process writes its reports to one directory. A later step prints them, lists the distinct errors in the job summary and fails the job if there are any, if a report shows up instead in output the tests keep (such as rdcd's logs), or if the directory is missing because no test ran. The legs report what the other jobs cannot see, such as a read past a stack array or an invalid enum value that does not crash. The sanitizers get separate builds because in a process with both of GCC's runtimes, UBSan's reports go to stderr instead of to log_path (both runtimes export __sanitizer_set_report_path, and ASan's copy wins). The tests capture that stderr, so with one combined build the log directory had one report while rdcd's captured output held 270 UBSan errors. Before the tests, a step checks that rdcd, rdci, librdc, librdc_bootstrap and the fake amd-smi load the leg's runtime, that librdc calls the leg's checks, and that a program with a known bug leaves a report at log_path, so a build without the sanitizer fails instead of passing quietly. ASan's leak detection is off: Python under the runtime does not free everything at exit (151 allocations for "import ctypes, json"), and rdcd and rdci inherit its options. verify_asan_link_order is off because the tests preload the fake amd-smi, which links the runtime, ahead of it into rdcd; with the check on, rdcd exits at once. The legs build the same gRPC 1.67.1 as rdc_build and cache the install; each leg's ccache has its own key. Both are restored and saved in separate steps, since actions/cache saves nothing when a job fails. The legs name gcc and g++, build amd-smi with E-SMI and without a sanitizer, and build RDC without the runtime, profiler and RVS modules, which need ROCm's GPU runtime; rdcd runs without them. Each gtest has a time limit and writes a JUnit report, and the JUnit results go to the job summary as in the other test jobs. The legs do not gate until the repo variable RDC_SANITIZE_GATING is 'true'. On develop with the fake-GPU suite, in a GPU-less Ubuntu 22.04 container rather than the job's image, both legs' test runs pass every test: the 8 gtest programs the job runs (43 cases), 106 fake-GPU tests (24 skipped), and tests/ci but for test_prometheus_help, which needs prometheus_client (the job installs it). The ASan leg reports nothing. The UBSan leg reports, in 89 processes: a reference bound to a null pointer when RdcTelemetryModule passes &v[0] of an empty vector; an invalid rdc_field_type_t in fetch_smi_field() and the cache, because RDC_HEALTH_EEPROM_CONFIG_VALID reports success without setting a type or value, which then come from RdcSmiLib's uninitialized values array; the link type and P2P flag that rdc_device_topology_get() reads after a failed amd-smi call; and an invalid rdc_field_t where rdci dmon, the client library and rdcd load the field ID 99999 that a tests/ci test passes on purpose. GCC rather than clang for now: in local builds, ROCm's clang 22 with -shared-libsan reported ODR violations in the gtests (each string literal registered twice), and with clang's static runtime every process that loads RDC's instrumented libraries without being an instrumented executable (Python, setpriv, llvm-symbolizer) needed the runtime preloaded. A clang build like TheRock's is a follow-up. Signed-off-by: Adam360x <Adam.pryor@amd.com>
Add the rdc_clang job: RDC built with the toolchain TheRock uses, AMD clang with gRPC 1.78.1 linked statically. It checks that RDC was compiled by AMD clang and that no RDC binary or library needs a gRPC, protobuf or Abseil library; runs the gtests, the fake-GPU suite and the other no-GPU tests; and checks that the gtests ran and that the pytest suites still collect their tests. It also runs: - rdc_race_tests: a ThreadSanitizer test binary into which each race test compiles the RDC sources it drives, built with RDC_BUILD_RACE_TESTS (off by default) and never installed. Its only test so far checks that ThreadSanitizer reports a planted race. - stress tests: a test marked `stress` runs only with RDC_TEST_STRESS=1, which this job sets, and with RDC_CI_REQUIRE_STRESS=1 a run in which no stress test ran fails. The first one starts rdcd, reads one sample of eight fields and stops rdcd, 50 times. The installed fake-GPU suite gains a conftest.py and fake_gpu_stress.py for this. The job's container runs without Docker's default seccomp profile: on hosts with 32 bits of mmap entropy ThreadSanitizer turns address randomization off for itself, which that profile blocks. The job does not gate until the repository variable RDC_CLANG_GATING is 'true'. Signed-off-by: Adam360x <Adam.pryor@amd.com>
github-actions
Bot
added
project: rdc
github actions
This was referenced Oct 22, 2026
fix(rdc): rdci options, output and exit statuses, and the names and policy values RDC accepts
#14058
Draft
joshnguy-amd
force-pushed
the
users/adam360x/rdc-ci-sanitizers
branch
from
November 4, 2026 21:54
db0b967 to
ec58d80
Compare
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
TheRock builds RDC with AMD clang and gRPC linked statically, while RDC CI builds it only with GCC and shared gRPC, so problems specific to that build reach TheRock first. RDC also had no way to test for data races.
Technical Details
rdc_clang: RDC built with AMD clang and gRPC 1.78.1 linked statically. It checks that RDC was compiled by AMD clang and that no RDC binary or library needs a gRPC, protobuf or Abseil library, runs the gtests, the fake-GPU suite and the other no-GPU tests, and checks that the gtests ran and the pytest suites still collect their tests.rdc_race_tests: a ThreadSanitizer binary into which each race test compiles the RDC sources it drives (RDC_BUILD_RACE_TESTS, off by default, never installed). Its only test so far checks that ThreadSanitizer reports a planted race.stressruns only withRDC_TEST_STRESS=1, which this job sets. The first startsrdcd, reads one sample of eight fields and stopsrdcd, 50 times.RDC_CLANG_GATINGistrue.CI-only; no CHANGELOG entry.
Issue Tracking
ISSUE ID : #14107
Part of a series of RDC fixes, which #14107 lists. No JIRA ticket.
Test Plan
Ran the GCC fake-GPU suite and
tests/cion this branch in a GPU-less container, and a clang build with the race tests (ThreadSanitizer) of the whole series.Test Result
Fake-GPU suite: 106 passed, 25 skipped.
tests/ci: 92 passed, 6 skipped. Race tests: 8 of 8 pass with no ThreadSanitizer warning (whole series).Submission Checklist