test(rdc): run rdcd and rdci in CI against a fake amd-smi by adam360x · Pull Request #14041 · ROCm/rocm-systems

adam360x · 2026-10-22T18:17:19Z

Motivation

RDC CI has no GPU, so rdcd and rdci never ran in CI, and RDC's handling of amd-smi failures was untested.

Technical Details

A test-only fake amd-smi (tests/fake_amdsmi, installed with the tests, never on a library path) answers every amdsmi_* call RDC makes for simulated GPUs and can fail any of them.
The fake matches amd-smi where RDC depends on it: a NULL list to amdsmi_get_gpu_compute_process_gpus is invalid, and amdsmi_init accepts the CPU flag. RDC runs under the fake with RDC_DISABLE_ROCP set, so the profiler module stays off.
The fake's handles are generation-encoded numbers. A handle kept across the amdsmi_shut_down() that drops the last init reference is unknown, as with amd-smi: AMDSMI_STATUS_NOT_FOUND for a processor, AMDSMI_STATUS_INVAL for a socket.
RDC_FAKE_FAIL_ONCE, RDC_FAKE_UNFILLED and RDC_FAKE_DELAY_MS fail a call once, return success without writing outputs, or slow a call down.
RDC_FAKE_PROCS, RDC_FAKE_PROCS_GROW and RDC_FAKE_COUNTER_RESET model compute processes, processes that start between two calls, and counters that restart after a GPU reset.
RDC_FAKE_CPER_COUNT, RDC_FAKE_EVENTS, RDC_FAKE_NUM_FREQ and RDC_FAKE_VERSION_BUILD_NULL break amd-smi's output contract on purpose, so tests can check that RDC does not trust its library blindly.
A malformed knob value aborts instead of silently disabling the fault.
A GPU-less suite, tests/ci/test_fake_gpu*.py:
- compares each GPU's values in rdci output with the fake's value table,
- runs embedded mode in a child process,
- fails each amd-smi call RDC reaches, with and without poisoned outputs, and checks that rdcd survives and exits cleanly,
- checks that the fake implements every amdsmi_* symbol RDC imports and that no runtime file ships or references it,
- checks the knobs above and the handle rules against the fake directly (test_fake_gpu_fake.py).
CI runs the installed suite in its own pytest run. Without a GPU, the daemon tests in tests/ci now run against the fake too, including the rdcd that the quick-restart test starts.
rdc_test's timeout grows from 30 to 45 minutes; the suite takes a few minutes.

Test-only; no CHANGELOG entry.

Issue Tracking

JIRA ID: AILITOOLS-364

The base for a series of RDC fixes whose tests use this suite; #14107 lists them.

Test Plan

Built RDC with tests from this branch merged with develop and ran the fake-GPU suite and tests/ci in a GPU-less container, as rdc_test runs them.

Test Result

RDC CI (rdc_test): fake-GPU suite 135 passed, 24 skipped; tests/ci 96 passed, 4 skipped.

Known: on develop, rdcd can crash with SIGSEGV as it exits after SIGTERM. The suite's fault tests stop rdcd often; under heavy load they hit it 3 times in about 1,400 stops. The start/stop fix in this series (#14048) orders rdcd's stop.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

RDC CI has no GPU, so rdcd and rdci never ran there. Add a test-only fake amd-smi (tests component, never on a library path) that answers every amdsmi_* call RDC makes for simulated GPUs and can fail any of them, and a GPU-less suite that: - compares each GPU's values in rdci output with the fake's value table, - runs embedded mode in a child process, - fails each amd-smi call RDC reaches, with and without poisoned outputs, and checks that rdcd survives and exits cleanly, - checks that the fake implements every amdsmi_* symbol RDC imports and that no runtime file ships or references it. CI runs the installed copy of the suite (every tests/ci/test_fake_gpu*.py) in its own pytest run; without a GPU, the daemon tests in tests/ci now run against the fake as well, including the rdcd that the quick-restart test starts. rdc_test's timeout grows from 30 to 45 minutes; the suite takes about two minutes. Signed-off-by: Adam360x <Adam.pryor@amd.com>

Give the fake amd-smi the behaviors the remaining RDC fixes need to reproduce their bugs in CI: - Handles are generation-encoded numbers. The amdsmi_shut_down() that drops the last init reference ends the generation, so a handle kept across that shutdown is unknown, as to amd-smi: AMDSMI_STATUS_NOT_FOUND for a processor, AMDSMI_STATUS_INVAL for a socket. - RDC_FAKE_FAIL_ONCE, RDC_FAKE_UNFILLED and RDC_FAKE_DELAY_MS fail a call once, return success without writing outputs, or slow a call down; amdsmi_init honors the ones that apply to it. - RDC_FAKE_PROCS, RDC_FAKE_PROCS_GROW and RDC_FAKE_COUNTER_RESET model compute processes, processes that start between two calls, and counters that restart after a GPU reset. Process lists follow amd-smi's rules for short lists and count-only queries. - RDC_FAKE_CPER_COUNT, RDC_FAKE_EVENTS, RDC_FAKE_NUM_FREQ and RDC_FAKE_VERSION_BUILD_NULL can also break amd-smi's output contract on purpose (counts past what was written, an empty frequency table, a NULL build string), so tests can check that RDC does not trust its library blindly. A malformed value of these knobs aborts instead of silently disabling the fault. tests/ci/test_fake_gpu_fake.py checks each new knob and the handle rules against the fake directly. Signed-off-by: Adam360x <Adam.pryor@amd.com>

therock-pr-bot · 2026-10-22T18:18:34Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-22T18:18:37Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

🟡 Changes recommended

Inconsistent four-partition resource profiles and sanitizer preload ordering need correction before the affected test configurations are reliable.

2 open issues

What changed in this PR

This PR adds GPU-less end-to-end coverage for RDC, allowing CI to exercise rdcd, rdci, and embedded mode using a test-only fake amd-smi.

Changes:

Adds a configurable fake amd-smi with per-GPU readings and fault injection.
Adds tests for reported values, daemon resilience, API coverage, and packaging isolation.
Runs the installed suite separately in CI and enables existing daemon tests without a GPU.

File	Description
`projects/rdc/tests/fake_amdsmi/README.md`	Documents fake configuration and local testing.
`projects/rdc/tests/fake_amdsmi/fake_values.inc`	Defines shared per-GPU readings.
`projects/rdc/tests/fake_amdsmi/fake_amdsmi.cc`	Implements simulated GPUs and fault injection.
`projects/rdc/tests/fake_amdsmi/CMakeLists.txt`	Builds and installs the test-only library and suite.
`projects/rdc/tests/ci/test_negative_resilience.py`	Preserves the daemon environment during restart testing.
`projects/rdc/tests/ci/test_fake_gpu.py`	Adds GPU-less integration and fault tests.
`projects/rdc/tests/ci/fake_gpu_golden.py`	Converts fake readings into expected RDC values.
`projects/rdc/tests/ci/conftest.py`	Enables fake-backed daemon fixtures without GPUs.
`projects/rdc/CMakeLists.txt`	Includes the fake library when tests are enabled.
`.github/workflows/rdc-ci.yml`	Runs the installed suite and adds timeout support.

_{Review effort: Balanced}

The fake amd-smi listed partition profiles for 1, 2 and 8 partitions and gave every other count the eight-partition profile's index, so with RDC_FAKE_PARTITIONS=4 RDC read one XCC and a shared decoder per partition instead of two XCCs and a decoder of its own. List SPX, DPX, QPX and CPX, and the configured count as a mode of its own when it is none of those, and give the active profile its mode's index. test_discovery_shows_each_partitions_resources checks rdci discovery -i for 2, 3, 4 and 8 partitions; the 3 and 4 cases failed before. Signed-off-by: Adam360x <Adam.pryor@amd.com>

They were PR #14042, stacked on this one. Both change the same test-only fake amd-smi, so they are reviewed and merged as one PR. Signed-off-by: Adam360x <Adam.pryor@amd.com>

Address the review comments on #14041: - amdsmi_get_gpu_compute_process_gpus returns AMDSMI_STATUS_INVAL for a NULL list, as amd-smi does, instead of reporting a count. RDC's own NULL-count call then fails here as it does on real hardware. - amdsmi_init accepts AMDSMI_INIT_AMD_CPUS and finds no CPUs, as amd-smi does without the HSMP driver, so rdcd takes its production init path rather than the disable-CPUs retry. - The GPU-less daemon, harness and embedded launches set RDC_DISABLE_ROCP, so the profiler module is not loaded into the hermetic tests. Co-Authored-By: Claude <noreply@anthropic.com>

rdc_test_gpu depended on a gfx94x label resolved from therock-ci-config and on RDC_GPU_TESTS_ENABLED, so it has never run. Run tests/ci against a real GPU on lstt-ci-runner, the self-hosted runners AMDSMI CI uses, and drop the ci_config job that only resolved the old label. The job now skips fork PRs because it runs on a privileged self-hosted runner, and its container options match AMDSMI's GPU jobs. rdc_test stays GPU-less. Co-Authored-By: Claude <noreply@anthropic.com>

This reverts commit a4f99f8.

Signed-off-by: Adam360x <Adam.pryor@amd.com>

> Stacked on #14041 as a draft: it shows only its own commit. When #14041 merges, this PR moves to develop, where RDC CI runs, and becomes ready for review. ## Motivation RDC's tests run without sanitizers, so a read past a buffer or an invalid enum value that does not crash goes unseen. ## Technical Details - A new job, `rdc_sanitize`, has two legs: GCC `-fsanitize=address` and `-fsanitize=undefined`, one build each. With both runtimes in one process, UBSan's reports go to stderr instead of `log_path`. - Each leg runs the installed gtests that need no GPU (all but `rdc_diag_kernel_tests`, which has no install RPATH), the fake-GPU suite (#14041) and `tests/ci` against its build. Every process writes its reports to one directory. - A later step prints them, lists the distinct errors in the job summary, and fails the leg if there are any, if a report lands in output the tests keep (such as `rdcd`'s logs) instead, or if no test ran. - Before the tests, a step checks that the binaries load the leg's runtime, that librdc calls its checks, and that a program with a known bug leaves a report at `log_path`. - ASan's leak detection is off (Python under the runtime does not free everything at exit, and `rdcd` and `rdci` inherit its options), and so is `verify_asan_link_order` (the tests preload the fake amd-smi, which links the runtime, into `rdcd`). - The legs build gRPC 1.67.1 as `rdc_build` does and cache it. They do not gate until the repo variable `RDC_SANITIZE_GATING` is `true`. CI-only; no CHANGELOG entry. ## Issue Tracking ISSUE ID : #14107 Part of a series of RDC fixes, which #14107 lists. No JIRA ticket. ## Test Plan Ran each leg's build and test steps on this branch in a GPU-less Ubuntu 22.04 container. ## Test Result Both legs pass every test: the 8 gtest programs the job runs (43 cases), 106 fake-GPU tests (24 skipped), and `tests/ci` (92 passed, 6 skipped; `test_prometheus_help` needs `prometheus_client`, which the job installs). The ASan leg reports nothing. The UBSan leg reports, in 89 processes, findings that are on develop today: - a reference bound to a null pointer when `RdcTelemetryModule` passes `&v[0]` of an empty vector (the telemetry fix in this series, #14045, removes it), - an invalid `rdc_field_type_t` in `fetch_smi_field()` and the cache, because `RDC_HEALTH_EEPROM_CONFIG_VALID` reports success without setting a type or value, - the link type and P2P flag that `rdc_device_topology_get()` reads after a failed amd-smi call, - an invalid `rdc_field_t` where `rdci dmon`, the client library and `rdcd` load the field ID 99999 that a `tests/ci` test passes on purpose. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

systems-assistant · 2026-11-08T06:00:42Z

TheRock Submodule Bump Activity

Newest first

2026-11-08 — Included in TheRock via ROCm/TheRock#9793.

adam360x added 2 commits October 22, 2026 13:14

Copilot AI balanced review requested due to automatic review settings October 22, 2026 18:17

adam360x requested review from a team as code owners October 22, 2026 18:17

github-actions Bot added project: rdc github actions Pull requests that update GitHub Actions code labels Oct 22, 2026

therock-pr-bot Bot added the Not ready to Review label Oct 22, 2026

Copilot started reviewing on behalf of adam360x October 22, 2026 18:19

View session

systems-assistant Bot added the organization: ROCm label Oct 22, 2026

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Comment thread projects/rdc/tests/ci/test_fake_gpu.py Resolved

Comment thread projects/rdc/tests/fake_amdsmi/fake_amdsmi.cc Resolved

adam360x mentioned this pull request Oct 22, 2026

RDC: tracking the GPU-less test suite, sanitizer CI and the fixes they found #14107

Open

Merge the fake amd-smi's handle generations and fault knobs

6adc211

They were PR #14042, stacked on this one. Both change the same test-only fake amd-smi, so they are reviewed and merged as one PR. Signed-off-by: Adam360x <Adam.pryor@amd.com>

adam360x mentioned this pull request Oct 23, 2026

fix(rdc): apply a GPU group's config setting to every GPU or none #14075

Draft

1 task done

bill-shuzhou-liu approved these changes Oct 23, 2026

View reviewed changes

Comment thread projects/rdc/tests/fake_amdsmi/fake_amdsmi.cc Resolved

Comment thread projects/rdc/tests/fake_amdsmi/fake_amdsmi.cc Resolved

Comment thread projects/rdc/tests/ci/conftest.py Resolved

joshnguy-amd mentioned this pull request Nov 3, 2026

test(rdc): make the fake amd-smi match amd-smi on three calls #14980

Merged

1 task done

joshnguy-amd and others added 6 commits November 4, 2026 09:56

Revert "ci(rdc): run rdc_test_gpu on the lstt-ci-runner GPU runners"

48d5b3c

This reverts commit a4f99f8.

Merge branch 'develop' into users/adam360x/rdc-fake-gpu-tests

42f8ac1

Signed-off-by: Adam360x <Adam.pryor@amd.com>

test(rdc): apply ruff format to the fake-GPU suite

cbe733a

Signed-off-by: Adam360x <Adam.pryor@amd.com>

Merge branch 'develop' into users/adam360x/rdc-fake-gpu-tests

abbfbda

test(rdc): check that the fake amd-smi accepts the CPU init flag

ec1479c

Signed-off-by: Adam360x <Adam.pryor@amd.com>

adam360x removed the Not ready to Review label Nov 4, 2026

joshnguy-amd added the ci:skip Skip all CI builds/tests for this PR label Nov 5, 2026

Merge branch 'develop' into users/adam360x/rdc-fake-gpu-tests

aadd315

joshnguy-amd added ci:skip Skip all CI builds/tests for this PR and removed ci:skip Skip all CI builds/tests for this PR labels Nov 5, 2026

joshnguy-amd merged commit 9bcc270 into develop Nov 5, 2026
80 of 102 checks passed

joshnguy-amd deleted the users/adam360x/rdc-fake-gpu-tests branch November 5, 2026 16:51

Navigation Menu

Repository navigation

test(rdc): run rdcd and rdci in CI against a fake amd-smi - #14041

test(rdc): run rdcd and rdci in CI against a fake amd-smi#14041
joshnguy-amd merged 12 commits into
developfrom
users/adam360x/rdc-fake-gpu-tests

adam360x commented Oct 22, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

systems-assistant Bot commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

adam360x commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

systems-assistant Bot commented Nov 8, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adam360x commented Oct 22, 2026 •

edited

Loading

therock-pr-bot Bot commented Oct 22, 2026 •

edited

Loading