Repository navigation
rccl-tests: share GIN-SDMA pytest harness - #12730
Conversation
AllGather, AllToAll, and Broadcast were each carrying their own MPI launcher and connectivity-gate retry. One module keeps those from drifting, and the GPU-free guards now cover rocminfo failure and a data error that also looks like a connectivity abort. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
| Test | Topology |
|---|---|
regression-allgather-fp32_all_gather_perf |
4n_8ppn |
regression-alltoall-fp32_alltoall_perf |
4n_8ppn |
regression-bf16-sum_all_reduce_perf |
4n_8ppn |
regression-bf16-sum_reduce_scatter_perf |
4n_8ppn |
regression-reduce-scatter-fp32-sum_reduce_scatter_perf |
4n_8ppn |
Every failure is 4-node. All 1-node and 2-node variants of those same tests pass, and the rccl-debug stack passes 193/199. The OCI cluster ran the identical five 4-node tests and passed all of them.
The failures track a single node
In all five Ruby failures, the same node reports the primary error and the others report a cascade from it:
cv350-rck-g03-e14-08: Test NCCL failure ... 'unhandled system error (run with NCCL_DEBUG=INFO for details)'
cv350-rck-g03-e13-08: Test NCCL failure ... 'remote process exited or there was a network error'
cv350-rck-g03-e20-08: Test NCCL failure ... 'remote process exited or there was a network error'
The Slurm allocations line up with the topology pattern exactly:
| Topology | Nodes | Result |
|---|---|---|
1n_8ppn |
c07-18 |
pass |
2n_8ppn |
c09-18, e07-08 |
pass |
4n_8ppn |
c07-18, e13-08, e14-08, e20-08 |
fail |
cv350-rck-g03-e14-08 appears only in the failing allocation, and it is the primary failure in the develop baseline as well. This looks like a node or inter-node fabric problem on ruby64 that affects any RCCL PR landing on that allocation. Diagnosing it properly needs NCCL_DEBUG=INFO, which the log asks for and the perf harness doesn't set.
This PR isn't in the failing code path
The change touches four pytest files only — gin_sdma_harness.py, test_AllGather.py, test_AllToAll.py, test_Broadcast.py. The failing tests are rccl_test perf regressions launched directly via mpirun, not through pytest.
Requests
- Can someone confirm whether
cv350-rck-g03-e14-08is a known-bad node, and drain or repair it if so? I've re-run the Ruby leg in case a fresh allocation avoids it. - If it can't be fixed quickly, this PR would need a gate override given the baseline parity above.
- Worth considering: coco already collects the baseline stack and prints its results beside the PR's, but nothing compares them — the gate fails on any test failure rather than on new failures. Diffing PR against baseline and failing only on regressions would let a PR like this pass without human intervention when a cluster node is broken.
Other red checks (none are required on develop)
therock-pr-bot— systemic 15-minute timeout waiting for apre-commitcheck-run that never runs on PRs targetingdevelop.cleanup— the unrelated Docs Preview Cleanup workflow, failing with[@octokit/auth-app] appId option is required(missing secret in that workflow).mci/rocm-libraries/precheckin(rccl-tests)—ABORTEDwithTimeout has been exceededon the4gfx908stage; retriggerable.
Second confirmation:
|
| #12730 (Oct 2) | #12705 (Oct 3) | |
|---|---|---|
4n_8ppn allocation |
c07-18, e13-08, e14-08, e20-08 |
c17-08, c17-18, e14-08, e20-08 |
| Failing tests | the same five | the same five |
| Primary error node | cv350-rck-g03-e14-08 |
cv350-rck-g03-e14-08 |
| PR stack | passed=17, failed=5, skipped=3 |
passed=17, failed=5, skipped=3 |
| develop baseline | passed=17, failed=5, skipped=3 |
passed=17, failed=5, skipped=3 |
| OCI cluster | success | success |
| 1-node / 2-node | all pass | all pass |
In #12705 the healthy-node control is even cleaner: c17-08 is in both the passing 1-node allocation and the failing 4-node allocation, so that node is fine — the only thing that changes is the presence of e14-08.
Two unrelated PRs, two days, two different allocations, one node in common, with the develop-only baseline failing identically in both. This is a cluster problem on ruby64, not a regression from either change.
Note that two of two 4-node allocations drew e14-08, which suggests a small or reserved node pool rather than random selection — so re-running may not be enough to dodge it. I have a Ruby re-run in flight on this PR to test exactly that.
abouteiller
left a comment
abouteiller
left a comment
There was a problem hiding this comment.
Diff came out identical to previously approved #11894
TheRock Submodule Bump ActivityNewest first
|
> **Replaces #12488.** Same change; only the head branch moved. > > #12488 was opened from a fork, where the required `RCCL PR gate` check can never pass. Fork pull requests receive no secrets, so the `RCCL coco PR` workflow skips its `Setup` job, the coco smoke never runs, and the gate fails closed in a few seconds. This branch lives in `ROCm/rocm-systems`, so the gate dispatches coco on its own. ## Summary - Follow-up to #12730, which landed `gin_sdma_harness.py` and ported AllGather, AllToAll, and Broadcast. That PR left `test_AllToAllDevtime.py` for this one. Rebased onto `develop` after #12730 merged, so the diff is now this one file. - Replace the local MPI launcher, `-x` env assembly, fixed-size `-D 3` argv, hang handling, and `rocminfo` GPU count with `gin_sdma_harness`. `detect_ngpus()` raises when `rocminfo` is missing, so it runs only when this module is enabled and `RCCL_TESTS_A2A_NP` is unset. - Keep the device-api CLI and test names: `-B 1`, `-B 2`, and `-B 2 -H 1`, plus `NCCL_GIN_TYPE` default `2`. `run-device-api-ci.sh` still runs this file directly. - Retry connectivity-gate aborts through `run_with_conn_gate_retry`, with the same `RCCL_TESTS_A2A_CONN_RETRIES` knob (default 5) as `test_AllToAll.py`. Data-check failures are not retried. - Fail the devtime smoke unless the release-build out-of-place `#wrong` column and the `Out of bounds values` count are both zero. `#wrong=` is printed only under `DEBUG_PRINT`, so a line such as `Out of bounds values : 3 FAILED` used to pass. In-place `#wrong` stays `N/A` because AllToAll does not check in-place. - The row regex requires both `#wrong` groups, as `BCAST_ROW_RE` does, so a skipped out-of-place row does not match instead of reading the in-place column. The `-B 2` test checks the `WARN --device_timing=2` line before the datacheck, so that warning names the cause. `_wrong_count`, `_NUM`, `_WRONG`, and `_OOB_RE` now come from `gin_sdma_harness`. `_A2A_OOP_WRONG_RE` stays local because `BCAST_ROW_RE` cannot match the AllToAll root of `-1`. JIRA ID : AICOMRCCL-2221 ## Test plan - [x] `pytest projects/rccl-tests/test/test_AllToAllDevtime.py --collect-only` collects the three GPU smokes with the module disabled, and again with `RCCL_TESTS_GIN_SDMA_DEVTIME=1 RCCL_TESTS_A2A_NP=2` (smci350-rck-g03-d07-40) - [x] `pytest projects/rccl-tests/test/test_AllToAllDevtime.py -k 'not Mode'`: 8 datacheck guards passed, and the three GPU smokes stay skipped with the module disabled. Deleting the unchecked-`N/A` branch, or reverting the regex to a single `#wrong` group, turns the matching guard red. - [x] Device-api job: `projects/rccl/tools/ci/lib/device-api.sbatch` on Ruby `cv350-rck-g03-e22-08` (8× gfx950, ROCm 10.2.0 TheRock), head `c1f875fe47`. Builds OpenMPI, RCCL, and rccl-tests with `ENABLE_DEVICE_API=1`, then runs `run-device-api-ci.sh`. The benchmark matrix passed, and the devtime smoke passed 11 of 11, including the three GPU tests `-B 1`, `-B 2`, and `-B 2 -H 1` (69.96s). An earlier run on `388c680765` (`cv350-rck-g03-e07-18`) also passed. - [ ] `RCCL PR gate` (Ruby and OCI coco smoke) on head `56b28e24c6`. It passed on `1b28af1f56`. The first OCI run on `56b28e24c6` failed one unrelated test: the 2-node BF16 AllReduce regression on ROCm 10.1 hit `LLVM ERROR: IO failure on output stream: Bad address` (see the CI comment below). Re-running. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
gin_sdma_harness.py(MPI launcher, connectivity-gate retry, Broadcast debug-file capture).detect_ngpus()raisesRuntimeErrorwhenrocminfois missing or fails, so GPU-free collection still works on a host with ROCm on PATH and no/dev/kfd.#wrong, so deleting the data-failure branch fails the test.test_AllToAllDevtime.pyis intentionally not ported here. It is the pytest device-api CI runs, and that move is a follow-up on AICOMRCCL-2221.Same commit as #11894 (
17d7a8680e). This pull request is opened fromROCm/rocm-systemsso the RCCL coco PR workflow can run on thepull_requestevent.JIRA ID : AICOMRCCL-2221
Test plan
pytest projects/rccl-tests/test -k GinSdmacollects from the rccl-tests root and fromtest/(38 tests, reported on rccl-tests: share GIN-SDMA pytest harness #11894)Made with Cursor