Repository navigation
ci(rccl): build device-api with rocSHMEM GIN - #12752
Merged
Merged
Conversation
3 tasks
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
ARGUS
Bottom line: Worth settling before approval: device-api.sbatch:161 builds the non-GIN rccl-tests binaries against the --rocshmem-gin librccl.so, which leaves its rocSHMEM host symbols for the executable to supply.
Comment @argus review to re-run, or @argus re-review after pushing.
dlamd1dai
force-pushed
the
users/dondai/aic2221-device-api-ci
branch
from
October 12, 2026 03:53
2b6f1bf to
91bd83c
Compare
Contributor
Author
|
@argus re-review |
Contributor
Author
|
@argus re-review |
Contributor
Author
|
@argus re-review d5084ba addresses the latest notes:
|
The device-api job compiled RCCL with --rocshmem-gin but never installed it, so the cmake config glob died before any benchmark, and rccl-tests did not RDC-link the binaries that load that library. Co-authored-by: Cursor <cursoragent@cursor.com>
GIN type 6 selects rocSHMEM GDA, so gin-d3-allgather never reached the SDMA kernel this job builds. Point it at type 7 and keep the longer rocSHMEM build from dying on a 2h SLURM limit or a missing rccl.env. Co-authored-by: Cursor <cursoragent@cursor.com>
…221) Restore the ENABLE_DEVICE_API install.sh build and LSA matrix so this job still covers device-API-without-rocSHMEM, and match gin.sbatch's build-log handling for the remaining Argus nits. Co-authored-by: Cursor <cursoragent@cursor.com>
…-2221) Keep the ENABLE_DEVICE_API install.sh build for the symmetric matrix, and do not run or require all_reduce_perf -D 1/-D 2 in this job. Co-authored-by: Cursor <cursoragent@cursor.com>
A single || on the test brace group ignored set -e for the non-GIN run, so only the GIN matrix status was retained. Capture each invocation separately. Co-authored-by: Cursor <cursoragent@cursor.com>
…-2221) Put all_reduce_perf -D 1/-D 2 back on the non-GIN matrix, where that binary builds, and run the five symmetric binaries against the rocSHMEM GIN build so the RDC link is exercised. Keep a single --rocshmem-gin librccl.so. Co-authored-by: Cursor <cursoragent@cursor.com>
The -x check only proves each RDC-linked binary was built under make -k. The rocSHMEM GIN path is reached only by the -D 3/-D 4 suites. Co-authored-by: Cursor <cursoragent@cursor.com>
…221) The runner writes that summary to stdout, and a non-GIN failure is followed by the GIN matrix, so the job log has to show test_out rather than only test_err. The build-phase comment now matches the || subshell, where set -e does not abort intermediate steps. Co-authored-by: Cursor <cursoragent@cursor.com>
…221) Symmetric kernels exist only for AllReduce, AllGather, and ReduceScatter, so the non-GIN matrix runs those at -R 2 and the device-api GIN matrix keeps the host-proxy alltoall suites. Anvil SDMA allgather moves to gin-tests.json, and --rocshmem-gin builds rocSHMEM instead of a separate prebuild. Co-authored-by: Cursor <cursoragent@cursor.com>
dlamd1dai
force-pushed
the
users/dondai/aic2221-device-api-ci
branch
from
October 19, 2026 17:22
774335b to
41731d7
Compare
Kapil-Shyam-Pawar
approved these changes
Oct 19, 2026
…OMRCCL-2221) The comment named the CI matrices and gin.sbatch, which change on their own schedule. Describe the binaries by their GIN device paths instead. Co-authored-by: Cursor <cursoragent@cursor.com>
abouteiller
approved these changes
Oct 22, 2026
1 task done
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
device-api.sbatchinstalls RCCL with--prefix, matchinggin.sbatch, solib/cmake/rccl/*config.cmakeexists.--rocshmem-ginbuilds the sibling rocSHMEM checkout intoprojects/rccl/ext/rocshmem. This job does not callbuild-rocshmem.sh.device-api-nongin-tests.json,ENABLE_DEVICE_API=1 ./install.shbuild) runs AllReduce, AllGather and ReduceScatter at-R 2, which are the collectives that have symmetric kernels, plus LSAall_reduce_perf -D 1and-D 2.device-api-tests.json, cmakeENABLE_DEVICE_API+ENABLE_ROCSHMEM_GINbuild) is host-proxy alltoall:gin-d3,gin-d4, andgin-d3-nicfusionfromdevelop, all withNCCL_GIN_TYPE=2.NCCL_GIN_TYPE=7) moves togin-tests.jsonasrccl-gin-rocshmem-sdma-ag-d3, next to the existing SDMA alltoall and broadcast cases. The gin build fails ifall_gather_perfwas not produced, and the gin test results remain non-gating.alltoall_perf,all_gather_perfandbroadcast_perf.all_reduce_perfstill does not build in that configuration, so it stays on the non-GIN matrix.FAILED RUNSsummary fromtest_out.Independent of #11894 (pytest harness).
test_AllToAllDevtime.pyis unchanged.JIRA ID : AICOMRCCL-2221
Known failure on ROCm 10.2 nightlies (not caused by this PR)
On ROCm 10.2 nightlies (HIP 7.17.26405 to 7.17.26412), communicator teardown in symmetric and device-API runs fails in
ncclCuMemFree. One rank hitsalloc.h:789/:809 HIP failure 'invalid argument'and another segfaults inhipMemUnmap. Thedevelopnightly of RCCL cluster CI fails the same way without this PR (run 38567603219, Oct 14). The likely root cause is the HIP VMM import/release regression ROCM-33022. RCCL-side tracking is in AICOMRCCL-3065.On this PR it appeared as:
774335be8d, wheredevice-api-symmetric_reduce_perfexited with rc 139 onrocm-10.2-latestonly (the 10.1-ga stacks passed);symmetricandgin-d4failures in the Oct 12 device-api run on91bd83c1f4.If it recurs, it is this bug, not a regression from this change. This PR touches only CI scripts and the rccl-tests CMake, not RCCL source.
Test plan
RCCL cluster CI on
41731d7d77: run 39293555415.librccl.so, finds the RCCL cmake config, and installs rocSHMEM atprojects/rccl/ext/rocshmemall_reduce_perf,all_gather_perfandreduce_scatter_perfat-R 2, plusall_reduce_perf -D 1and-D 2alltoall_perf -D 3and-D 4(gin-d3,gin-d4,gin-d3-nicfusion;NCCL_GIN_TYPE=2)all_gather_perfand runsrccl-gin-rocshmem-sdma-ag-d3(test result is non-gating)