Use NCCL wheels from PyPI for CUDA 12 builds by divyegala · Pull Request #2629 · rapidsai/raft

divyegala · 2025-04-08T20:55:48Z

This PR attempts to de-vendor NCCL DSOs that we bundle along with our wheel builds.

This reverts commit 5290e87.

divyegala · 2025-04-16T03:24:52Z

conda/recipes/libraft/recipe.yaml

+      - nccl ${{ nccl_version }}
+      - ucxx ${{ ucxx_version }}


libraft recipe installs the distributed component

raft/conda/recipes/libraft/recipe.yaml

Line 94 in 902ed56

cmake --install cpp/build --component distributed

which has a build and runtime dependency on targets from NCCL and ucxx as seen here

raft/cpp/CMakeLists.txt

Lines 392 to 406 in 902ed56

rapids_export_package(

BUILD ucxx raft-distributed-exports COMPONENTS ucxx python GLOBAL_TARGETS ucxx::ucxx ucxx::python

)

rapids_export_package(

INSTALL ucxx raft-distributed-exports COMPONENTS ucxx python GLOBAL_TARGETS ucxx::ucxx

ucxx::python

)

rapids_export_package(BUILD NCCL raft-distributed-exports)

rapids_export_package(INSTALL NCCL raft-distributed-exports)

# ucx is a requirement for raft_distributed, but its config is not safe to be found multiple times,

# so rather than exporting a package dependency on it above we rely on consumers to find it

# themselves. Once https://github.com/rapidsai/ucxx/issues/173 is resolved we can export it above

# again.

target_link_libraries(raft_distributed INTERFACE ucx::ucp ucxx::ucxx NCCL::NCCL)

I think these changes are needed and I believe the reason it did not error out until now is because distributed is a header-only target with INTERFACE dependencies so if consumer libraries brought in NCCL themselves it would be okay.

Happy to revert the changes, however, if this is considered an improper way of working with header only packages.

@divyegala I think these changes to the rattler-build recipe are reasonable, but can we break them out into a separate PR so we've got a cleaner history of when and why this was added?

Agreed, this seems fine but doesn't make sense to bundle with the NCCL wheels work.

Reverted the commit, opened new PR #2636

This reverts commit f67abbe.

bdice · 2025-04-16T18:37:55Z

ci/build_wheel.sh

 fi

+RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
+if [[ ${RAPIDS_CUDA_MAJOR} == "12" ]]; then


Is there a reason we do this only for CUDA 12?

Yes, there's no NCCL wheel distributions available for CUDA 11 since April 2024 and even then, no arm64 binaries. I think we'll have to continue to vendor NCCL binaries with CUDA 11.

Great. That is fine! I just didn't know the constraints off the top of my head.

bdice · 2025-04-16T18:39:17Z

dependencies.yaml

+    common:
+      - output_types: conda
+        packages:
+          - &nccl_unsuffixed nccl>=2.19


Is 2.19 a high enough lower bound? The latest is 2.26. I'm pretty sure we rely on newer versions than 2.19 for Blackwell support and other features. I don't know the exact bound to use here but we should do some validation with the oldest NCCL release we claim to support.

That's a good question. I think we could bump the version of NCCL as a follow-on to this PR depending on the . I just used the version constraint that was already present. I just checked our ci-wheel images and the system version on those is also 2.26.

Alrighty! If this matches other bounds already being used in RAFT, we can merge in the current state.

gforsyth

divyegala · 2025-04-16T18:56:15Z

/merge

Reference PR in RAFT rapidsai/raft#2629. The size of `libcuvs-cu12` wheel goes down from about 1.2G to 854M. Authors: - Divye Gala (https://github.com/divyegala) Approvers: - Bradley Dice (https://github.com/bdice) URL: #827

use nccl wheel

7e37261

divyegala added feature request New feature or request non-breaking Non-breaking change labels Apr 8, 2025

divyegala self-assigned this Apr 8, 2025

divyegala requested review from a team as code owners April 8, 2025 20:55

divyegala requested a review from gforsyth April 8, 2025 20:55

github-actions bot added python ci labels Apr 8, 2025

divyegala added 5 commits April 8, 2025 14:21

delete nccl dep in pyproject from cuda 11 build

1df2dc7

delete nccl in libraft cuda 11 requirements

da99d61

use bash var

2916f7e

try

9319ddb

packages list empty

bd06d58

github-actions bot removed the python label Apr 8, 2025

attempt to use nccl from runtime path

881bf17

divyegala requested a review from a team as a code owner April 8, 2025 22:55

github-actions bot added CMake python labels Apr 8, 2025

use system nccl at build time

ad3c7aa

github-actions bot added the cpp label Apr 9, 2025

divyegala added 5 commits April 9, 2025 11:49

fix

3863d3b

style check

1e7fa83

keep exporting nccl

9de1626

style check

fabab7b

don't skip tests and add nccl to run deps

426fca7

divyegala requested a review from a team as a code owner April 9, 2025 21:33

divyegala added 3 commits April 9, 2025 15:23

tests are still being skipped

5290e87

Revert "tests are still being skipped"

602ca4f

This reverts commit 5290e87.

revert test skip

0d94ab6

divyegala added 8 commits April 15, 2025 21:17

Merge remote-tracking branch 'upstream/branch-25.06' into nccl-wheel

dfd88a5

delete system nccl for cuda 12 dask wheel tests

7aabd32

check arg

a373adc

message

82ef5a6

append property

62c8bb6

fix style

bcccb25

libraft.so does not link to nccl; correct rpath for raft-dask objects

54bb266

simplify cmake options

132aa73

github-actions bot removed the cpp label Apr 16, 2025

attempt to fix conda recipe of libraft with nccl, ucxx

f67abbe

divyegala commented Apr 16, 2025

View reviewed changes

divyegala added 2 commits April 16, 2025 09:37

Revert "attempt to fix conda recipe of libraft with nccl, ucxx"

e5031fb

This reverts commit f67abbe.

Merge branch 'branch-25.06' into nccl-wheel

74069d0

divyegala removed the request for review from a team April 16, 2025 18:24

bdice reviewed Apr 16, 2025

View reviewed changes

bdice approved these changes Apr 16, 2025

View reviewed changes

gforsyth approved these changes Apr 16, 2025

View reviewed changes

rapids-bot bot merged commit 022a4f1 into rapidsai:branch-25.06 Apr 16, 2025
80 checks passed

divyegala mentioned this pull request Apr 16, 2025

Use NCCL wheels from PyPI for CUDA 12 builds rapidsai/cuvs#827

Merged

bdice mentioned this pull request Apr 23, 2025

feat: add libwholegraph wheel rapidsai/cugraph-gnn#182

Merged

jameslamb mentioned this pull request Jan 6, 2026

build and test against CUDA 13.1.0 #2912

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use NCCL wheels from PyPI for CUDA 12 builds#2629

Use NCCL wheels from PyPI for CUDA 12 builds#2629
rapids-bot[bot] merged 27 commits intorapidsai:branch-25.06from
divyegala:nccl-wheel

divyegala commented Apr 8, 2025

Uh oh!

divyegala Apr 16, 2025 •

edited

Loading

Uh oh!

gforsyth Apr 16, 2025

Uh oh!

bdice Apr 16, 2025

Uh oh!

divyegala Apr 16, 2025

Uh oh!

bdice Apr 16, 2025

Uh oh!

divyegala Apr 16, 2025

Uh oh!

bdice Apr 16, 2025

Uh oh!

bdice Apr 16, 2025

Uh oh!

divyegala Apr 16, 2025

Uh oh!

bdice Apr 16, 2025

Uh oh!

gforsyth left a comment

Uh oh!

divyegala commented Apr 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	rapids_export_package(
	BUILD ucxx raft-distributed-exports COMPONENTS ucxx python GLOBAL_TARGETS ucxx::ucxx ucxx::python
	)
	rapids_export_package(
	INSTALL ucxx raft-distributed-exports COMPONENTS ucxx python GLOBAL_TARGETS ucxx::ucxx
	ucxx::python
	)
	rapids_export_package(BUILD NCCL raft-distributed-exports)
	rapids_export_package(INSTALL NCCL raft-distributed-exports)

	# ucx is a requirement for raft_distributed, but its config is not safe to be found multiple times,
	# so rather than exporting a package dependency on it above we rely on consumers to find it
	# themselves. Once https://github.com/rapidsai/ucxx/issues/173 is resolved we can export it above
	# again.
	target_link_libraries(raft_distributed INTERFACE ucx::ucp ucxx::ucxx NCCL::NCCL)

Conversation

divyegala commented Apr 8, 2025

Uh oh!

divyegala Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gforsyth left a comment

Choose a reason for hiding this comment

Uh oh!

divyegala commented Apr 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

divyegala Apr 16, 2025 •

edited

Loading