Repository navigation
Conversation
AllGather, AllToAll, and Broadcast were each carrying their own MPI launcher and connectivity-gate retry. One module keeps those from drifting, and the GPU-free guards now cover rocminfo failure and a data error that also looks like a connectivity abort. Co-authored-by: Cursor <cursoragent@cursor.com>
…CL-2221) test_AllToAllDevtime.py was the remaining copy of the MPI launcher and GPU detection. The device-api job is the only consumer, so the -B 1, -B 2, and -B 2 -H 1 flags and the test names stay the same. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
3 of 4 tasks
Contributor
Author
|
Superseded by #12744 and closing in favour of it. Same commits, same content; only the head branch moved. Why. This PR was opened from a fork, and the required No further work is needed on this branch. Closing rather than leaving it open so it does not keep consuming CI or show up as a duplicate for reviewers; it stays readable and can be reopened. |
dlamd1dai
added a commit
that referenced
this pull request
Oct 18, 2026
> **Replaces #12488.** Same change; only the head branch moved. > > #12488 was opened from a fork, where the required `RCCL PR gate` check can never pass. Fork pull requests receive no secrets, so the `RCCL coco PR` workflow skips its `Setup` job, the coco smoke never runs, and the gate fails closed in a few seconds. This branch lives in `ROCm/rocm-systems`, so the gate dispatches coco on its own. ## Summary - Follow-up to #12730, which landed `gin_sdma_harness.py` and ported AllGather, AllToAll, and Broadcast. That PR left `test_AllToAllDevtime.py` for this one. Rebased onto `develop` after #12730 merged, so the diff is now this one file. - Replace the local MPI launcher, `-x` env assembly, fixed-size `-D 3` argv, hang handling, and `rocminfo` GPU count with `gin_sdma_harness`. `detect_ngpus()` raises when `rocminfo` is missing, so it runs only when this module is enabled and `RCCL_TESTS_A2A_NP` is unset. - Keep the device-api CLI and test names: `-B 1`, `-B 2`, and `-B 2 -H 1`, plus `NCCL_GIN_TYPE` default `2`. `run-device-api-ci.sh` still runs this file directly. - Retry connectivity-gate aborts through `run_with_conn_gate_retry`, with the same `RCCL_TESTS_A2A_CONN_RETRIES` knob (default 5) as `test_AllToAll.py`. Data-check failures are not retried. - Fail the devtime smoke unless the release-build out-of-place `#wrong` column and the `Out of bounds values` count are both zero. `#wrong=` is printed only under `DEBUG_PRINT`, so a line such as `Out of bounds values : 3 FAILED` used to pass. In-place `#wrong` stays `N/A` because AllToAll does not check in-place. - The row regex requires both `#wrong` groups, as `BCAST_ROW_RE` does, so a skipped out-of-place row does not match instead of reading the in-place column. The `-B 2` test checks the `WARN --device_timing=2` line before the datacheck, so that warning names the cause. `_wrong_count`, `_NUM`, `_WRONG`, and `_OOB_RE` now come from `gin_sdma_harness`. `_A2A_OOP_WRONG_RE` stays local because `BCAST_ROW_RE` cannot match the AllToAll root of `-1`. JIRA ID : AICOMRCCL-2221 ## Test plan - [x] `pytest projects/rccl-tests/test/test_AllToAllDevtime.py --collect-only` collects the three GPU smokes with the module disabled, and again with `RCCL_TESTS_GIN_SDMA_DEVTIME=1 RCCL_TESTS_A2A_NP=2` (smci350-rck-g03-d07-40) - [x] `pytest projects/rccl-tests/test/test_AllToAllDevtime.py -k 'not Mode'`: 8 datacheck guards passed, and the three GPU smokes stay skipped with the module disabled. Deleting the unchecked-`N/A` branch, or reverting the regex to a single `#wrong` group, turns the matching guard red. - [x] Device-api job: `projects/rccl/tools/ci/lib/device-api.sbatch` on Ruby `cv350-rck-g03-e22-08` (8× gfx950, ROCm 10.2.0 TheRock), head `c1f875fe47`. Builds OpenMPI, RCCL, and rccl-tests with `ENABLE_DEVICE_API=1`, then runs `run-device-api-ci.sh`. The benchmark matrix passed, and the devtime smoke passed 11 of 11, including the three GPU tests `-B 1`, `-B 2`, and `-B 2 -H 1` (69.96s). An earlier run on `388c680765` (`cv350-rck-g03-e07-18`) also passed. - [ ] `RCCL PR gate` (Ruby and OCI coco smoke) on head `56b28e24c6`. It passed on `1b28af1f56`. The first OCI run on `56b28e24c6` failed one unrelated test: the 2-node BF16 AllReduce regression on ROCm 10.1 hit `LLVM ERROR: IO failure on output stream: Bad address` (see the CI comment below). Re-running. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gin_sdma_harness.pyis not ondevelopyet, so this branch is stacked on that PR. The only new commit is thetest_AllToAllDevtime.pyport; the other files in the develop diff are rccl-tests: share GIN-SDMA pytest harness #11894.-xenv assembly, fixed-size-D 3argv, hang handling, androcminfoGPU count withgin_sdma_harness.detect_ngpus()now raises whenrocminfois missing, and it runs only when this module is enabled andRCCL_TESTS_A2A_NPis unset.-B 1,-B 2, and-B 2 -H 1, plusNCCL_GIN_TYPEdefault2.run-device-api-ci.shstill runs this file directly.JIRA ID : AICOMRCCL-2221
Test plan
pytest projects/rccl-tests/test/test_AllToAllDevtime.py --collect-onlycollects the three tests with the module disabled, and again withRCCL_TESTS_GIN_SDMA_DEVTIME=1 RCCL_TESTS_A2A_NP=2(smci350-rck-g03-d07-40)-B 1,-B 2, and-B 2 -H 1(run-device-api-ci.shdevtime smoke)Made with Cursor