feat: add libwholegraph wheel by gforsyth · Pull Request #182 · rapidsai/cugraph-gnn

gforsyth · 2025-04-16T19:52:47Z

Contributes to rapidsai/build-planning#33

Resolves rapidsai/rapids-wheels-planning#59

This PR proposes packaging libwholegraph as a wheel, which is then re-used by pylibwholegraph, cugraph-dgl, and cugraph-pyg.

It should avoid recompiling RAFT and RMM, as those are now both handled by their respective wheels.

The wholegraph library is now compiled as a shared library.

* Remove `USE_NCCL_WHEEL` arg where I was getting ahead of myself * Add notes about what things do * set tests and benchmarks to OFF * make sure name passed to `project` matches name passed to `rapids_cuda_init_architectures`

copy-pr-bot · 2025-04-16T19:52:50Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

gforsyth · 2025-04-16T19:53:34Z

/ok to test

gforsyth · 2025-04-16T19:56:57Z

/ok to test

gforsyth · 2025-04-16T20:03:30Z

/ok to test

gforsyth · 2025-04-16T20:10:38Z

/ok to test

don't need to expand over Python versions, only CUDA and ARCH

gforsyth · 2025-04-17T13:05:41Z

/ok to test

gforsyth · 2025-04-17T15:44:49Z

/ok to test

jameslamb

Hope you don't mind me crashing in, but I was interested when I saw @alexbarghi-nv post that this was up. Put up some comments for your consideration.

Leaving a "comment" review so my review doesn't block merging in case you get approvals between now and the next time I'm able to look at this.

python/libwholegraph/libwholegraph/load.py

python/libwholegraph/pyproject.toml

python/pylibwholegraph/pyproject.toml

python/libwholegraph/CMakeLists.txt

ci/build_wheel.sh

ci/build_wheel_libwholegraph.sh

Co-authored-by: James Lamb <jaylamb20@gmail.com>

This reverts commit eed6b84.

…ists.txt

.github/workflows/build.yaml

.github/workflows/pr.yaml

ci/build_wheel.sh

ci/build_wheel_pylibwholegraph.sh

python/libwholegraph/libwholegraph/load.py

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

bdice · 2025-04-22T02:37:22Z

The pip devcontainer build is looking for libwholegraph-cu12, but it should just look for the unsuffixed package iirc. I think you need some kind of cuda_suffixed: "true" thing like we do here: https://github.com/rapidsai/cudf/blob/bd588a8949de004685ffbf953d817c003006a3e6/dependencies.yaml#L1020-L1033

Also you might need to update the devcontainers like we did here for cuml: rapidsai/devcontainers#442. Get that PR ready first, and we'll merge it just before we merge this one (to avoid an admin-merge cycle).

gforsyth · 2025-04-22T12:42:31Z

The pip devcontainer build is looking for libwholegraph-cu12, but it should just look for the unsuffixed package iirc.

The pip devcontainer build is looking for libwholegraph-cu12, but it should just look for the unsuffixed package iirc. I think you need some kind of cuda_suffixed: "true" thing like we do here

There is an unsuffixed fallback in the dependencies.yaml -- I think it should be looking for the cu12 variant because it's running pip under cuda 12.8.

I think it's failing because there are no libwholegraph wheels until this gets merged.

jameslamb · 2025-04-22T12:59:14Z

I think it's failing because there are no libwholegraph wheels until this gets merged.

That's not it. libwholegraph wheels should never be downloaded in devcontainers builds here. There's code in the devcontainers setup to install dependencies, and it filters out of that anything that it knows it's building directly from source. Here's where that happens: https://github.com/rapidsai/devcontainers/blob/branch-25.06/features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh#L182-L191

@bdice 's advice in #182 (comment) is the fix... get a libwholegraph entry into the rapids-build-utils manifest in the devcontainers repo, and that should fix the build here.

gforsyth · 2025-04-22T16:27:00Z

devcontainers are currently broken by ucxx -- should that block this from going in?

bdice · 2025-04-22T16:46:56Z

devcontainers are currently broken by ucxx -- should that block this from going in?

Replied here: rapidsai/devcontainers#494 (comment)

alexbarghi-nv

Approved; mostly deferring to the ops team on the finer details of the build process.

bdice · 2025-04-23T14:02:54Z

All looks fine here with one required follow-up: we're shipping a ~350MB wheel but the conda package is only 18MB. The difference appears to be in NCCL. We need to stop shipping NCCL in this wheel as @divyegala recently did with RAFT and cuVS.

@gforsyth Here's what I propose:

Merge feat(cugraph-gnn): add libwholegraph Python build devcontainers#494 now
Rerun pip devcontainers and merge this PR immediately afterward
Create follow-up for NCCL devendoring

gforsyth · 2025-04-23T14:04:55Z

Yeah, I have a note down for the NCCL devendoring followup

xref: rapidsai/build-planning#33 xref: rapidsai/cugraph-gnn#182 Add `libwholegraph` wheels to the build manifest. Currently pointing at my fork in `cugraph-gnn` for testing. --------- Co-authored-by: Bradley Dice <bdice@bradleydice.com>

gforsyth · 2025-04-23T15:49:29Z

/merge

Followup to #182, related to #386 (review) Proposes using `--no-build-isolation` for `libwholegraph` wheel builds. `--no-build-isolation` preserves all of the source file paths, which should lead to better `sccache` hit rate and faster builds. All of the other `lib{something}` wheels in RAPIDS are built this way, see rapidsai/build-planning#108 Also switches `cugraph-pyg` builds to smaller runners (cpu4). That's a pure-Python package that shouldn't need many resources to build. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #388

gforsyth added 13 commits April 16, 2025 14:55

Stub out libwholegraph python library

81b24de

feat(dependencies): add entries for libwholegraph

9190163

feat(cmake): cargo-cult cmakelists for libwholegraph

03585fe

fix: make VERSION a symlink

8871971

fix(libwholegraph): add dynamic version resolution

162eb88

add nccl wheel conditional

1a0f140

refactor(wheel): add libraft and librmm to build dependencies

263d94d

fix(libwholegraph): CMakeLists.txt tweaks to build

330a8b2

* Remove `USE_NCCL_WHEEL` arg where I was getting ahead of myself * Add notes about what things do * set tests and benchmarks to OFF * make sure name passed to `project` matches name passed to `rapids_cuda_init_architectures`

fix(build_wheel): exclude librapids_logger.so

beb461a

chore(libwholegraph): add rapids_logger as runtime dependency

d97dd75

fix(libwholegraph): update rapids_config.cmake location

0d3af50

chore(libwholegraph): cleanup CMakeLists.txt

98da9cb

refactor(pylibwholegraph): add dependency on libwholegraph

9319112

gforsyth added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Apr 16, 2025

ci: add job to build libwholegraph wheel

081a3ac

gforsyth force-pushed the wholegraph_wheel branch from 5751242 to 081a3ac Compare April 16, 2025 19:56

gforsyth added 2 commits April 16, 2025 16:09

ci: add the libwholegraph wheel script

df2714b

chore: temporarily stop conda CI jobs

a70942a

gforsyth force-pushed the wholegraph_wheel branch from f933c99 to a70942a Compare April 16, 2025 20:09

chore: disable devcontainer build

b23b057

gforsyth added 2 commits April 16, 2025 16:43

ci: add matrix filter for libwholegraph wheel build

7ea78de

don't need to expand over Python versions, only CUDA and ARCH

refactor(wholegraph): shared library, not static

eed6b84

jameslamb reviewed Apr 21, 2025

View reviewed changes

gforsyth and others added 5 commits April 21, 2025 15:53

chore: apply suggestions from review

b440dd2

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Revert "refactor(wholegraph): shared library, not static"

d8c1940

This reverts commit eed6b84.

refactor: set BUILD_SHARED_LIBS=ON instead of changes to cpp/CMakeL…

c14900f

…ists.txt

fix(pylibwholegraph): add libwholegraph as runtime dependency

29a7b15

chore: fix formatting

cd1d7e0

bdice reviewed Apr 21, 2025

View reviewed changes

gforsyth and others added 3 commits April 21, 2025 16:10

fix: set package type to cpp

a2437b1

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

refactor(build_wheel): take package type as argument

e0d22c4

chore: remove bash debugging

bee0e10

gforsyth mentioned this pull request Apr 22, 2025

feat(cugraph-gnn): add libwholegraph Python build rapidsai/devcontainers#494

Merged

alexbarghi-nv approved these changes Apr 23, 2025

View reviewed changes

bdice approved these changes Apr 23, 2025

View reviewed changes

rapids-bot bot merged commit 2dd02f9 into rapidsai:branch-25.06 Apr 23, 2025
68 checks passed

gforsyth deleted the wholegraph_wheel branch April 23, 2025 15:49

gforsyth mentioned this pull request Apr 23, 2025

refactor: add libwholegraph wheel for 25.06 rapidsai/rapids-metadata#52

Merged

gforsyth mentioned this pull request Apr 30, 2025

fix(libwholegraph): set build_type to "nightly" and also publish wheel #193

Merged

This was referenced Aug 22, 2025

libwholegraph: stop vendoring NCCL in wheels #281

Closed

Update rapids-dependency-file-generator #285

Merged

jameslamb mentioned this pull request Jan 16, 2026

libwholegraph: build wheels without build isolation #388

Merged

Conversation

gforsyth commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Apr 16, 2025

Uh oh!

gforsyth commented Apr 16, 2025

Uh oh!

gforsyth commented Apr 16, 2025

Uh oh!

gforsyth commented Apr 16, 2025

Uh oh!

gforsyth commented Apr 16, 2025

Uh oh!

gforsyth commented Apr 17, 2025

Uh oh!

gforsyth commented Apr 17, 2025

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bdice commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gforsyth commented Apr 22, 2025

Uh oh!

jameslamb commented Apr 22, 2025

Uh oh!

gforsyth commented Apr 22, 2025

Uh oh!

bdice commented Apr 22, 2025

Uh oh!

alexbarghi-nv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdice commented Apr 23, 2025

Uh oh!

gforsyth commented Apr 23, 2025

Uh oh!

gforsyth commented Apr 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gforsyth commented Apr 16, 2025 •

edited

Loading

bdice commented Apr 22, 2025 •

edited

Loading

alexbarghi-nv left a comment •

edited

Loading