Repository navigation
test(amdsmi): stop partition tests staging NPS modes - #12472
koushikbillakanti-amd wants to merge 6 commits into
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
db33f17 to
88bedf9
Compare
charis-poag-amd
left a comment
There was a problem hiding this comment.
Thanks for digging into this. Looking at the history here, I think the underlying defect (ROCM-29523) is already resolved independent of this PR:
amdsmi_gpu_driver_reload() was removed entirely in #8575 (706209f4f54, 2026-07-28), along with this test's calls to it via ReloadDriverWithMessages. Since that commit, TestMemoryPartitionReadWrite can no longer trigger a real driver reload — it only stages a mode and prints an instructional message. The exact failure mode ROCM-29523 describes (~72 in-test reloads scrambling QPX→DPX topology, masked by a PASS) has been structurally impossible on develop since before the ticket was even filed.
Given that, what's left in this PR is test hygiene (dropping the now-dead staging loop, hard-failing on drift instead of skipping, the 19 new DI-backed unit tests) rather than an active defect fix. I'd suggest:
- Close this PR and ROCM-29523 with a note that the root cause (the reload API itself) was already removed in #8575, so the reported random-partition-drift can no longer occur via this path.
- If there's still appetite for keeping the cleanup/hardening parts (drift assertions, unit tests), those can land separately as a hygiene PR, decoupled from the "fixes a defect" framing.
If instead we want to keep real cross-reload NPS coverage for QA/validation rather than dropping it, we already have an established pattern for this: test_memory_partition_cycle.py stages every NPS mode, does the actual modprobe -r amdgpu && modprobe amdgpu reload, re-enumerates, verifies, and restores — gated behind AMDSMI_ALLOW_DESTRUCTIVE_TESTS=1 so it's opt-in and never runs by accident in CI. Mirroring that same env-var gate in the C++ functional test (rather than removing the NPS-cycling coverage outright) would give QA a way to still exercise real reload behavior on demand, without the always-on risk that caused ROCM-29523.
Open to it either way — just flagging that as the one path I see for not losing coverage, if there's a reason to keep this open beyond the hygiene changes.
88bedf9 to
f99a2d3
Compare
4f7c402 to
5d61f99
Compare
7ec900a to
77d1f98
Compare
Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
TestMemoryPartitionReadWrite lost its NPS coverage when the driver reload API was removed, and only requests the active mode by default. With AMDSMI_ALLOW_DESTRUCTIVE_TESTS=1 it now sets each other supported mode on every GPU, reloads amdgpu once per mode, and verifies the mode on each GPU. It then restores the starting mode and each GPU's accelerator profile, retrying BUSY and failing when the profile does not come back. The reload runs modprobe from a fixed path with fixed arguments, without a shell. The test shuts down the library first so its own descriptors do not hold the driver, refuses to start when another process holds amdgpu, and applies a mode left staged by a failed reload before restoring. Without the flag the test is unchanged: it never reloads and requests only the active mode. Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
77d1f98 to
192e227
Compare
Signed-off-by: Koushik Billakanti <koushik.billakanti@amd.com>
Motivation
On develop, the memory partition tests request NPS modes other than the active one: the C++ test asks for each supported mode from NPS1 to NPS8, and so do the Python setter and benchmark tests and the CLI
setsweep. Each request stays staged, and asking for the active mode again doesn't cancel it. The tests pass, and the next amdgpu reload by anyone brings the GPUs up in a different mode. #8575 also removed the reload from the C++ test, so it lost its NPS coverage.Technical Details
AMDSMI_ALLOW_DESTRUCTIVE_TESTS=1brings the NPS coverage back toamdsmitst: it applies each supported mode with a driver reload, then restores the starting mode and accelerator profiles.test_memory_partition_cycle.pyuses the same flag, and both follow the rules intest-design.md. Before staging anything they check for modprobe, CAP_SYS_MODULE and an idle amdgpu, and any failure that leaves a mode staged names it...._same_mode, and the CLI sweep no longer accepts INVAL for--memory-partition.Test-only.
amd_smi_drm_exstill requests NPS1 to NPS8 in the same CI step; that's a follow-up.JIRA ID
JIRA ID: ROCM-29523
Test Plan
From
projects/amdsmi, built withBUILD_TESTS=ON:amdsmitst --gtest_filter='GpuUnit.MemoryPartition*'and--gtest_filter='*Unit*'python3 -m unittest unit.gpu.test_memory_partition_staging, fromtests/pythonTestMemoryPartitionReadWritewith and without the flag, comparingamd-smi partition --currentbefore and afterTest Result
Submission Checklist