Skip to content

wheels: fix handling of cuda-tile dependency - #5646

Closed
ChuckHastings wants to merge 2 commits into
rapidsai:release/26.10from
ChuckHastings:fix_cuda_tile_dependency
Closed

ChuckHastings wants to merge 2 commits into
rapidsai:release/26.10from
ChuckHastings:fix_cuda_tile_dependency

Conversation

@ChuckHastings

@ChuckHastings ChuckHastings commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

(heavily edited by @jameslamb on Sep 8, 2026)

Contributes to rapidsai/build-planning#324

Wheel builds have been failing like this:

  -- Found Python: /pyenv/versions/3.14.7/bin/python (found version "3.14.7") found components: Interpreter
  CMake Error at build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61 (message):
    cuda.tile (cuTile Python) is required to build cuTile embedded kernels.
    Install cutile-python and cuda-tileiras (conda), or cuda-tile[tileiras]
    (pip).

    Interpreter: /pyenv/versions/3.14.7/bin/python

    Import error: Traceback (most recent call last):

      File "<string>", line 1, in <module>
        import cuda.tile

    ModuleNotFoundError: No module named 'cuda'
  Call Stack (most recent call first):
    build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:229 (_cutile_kernels_setup)
    build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/CMakeLists.txt:1161 (generate_cutile_kernels)


  -- Configuring incomplete, errors occurred!

(build link

Because:

  1. cuVS now needs cuda-tile (Python library) at build time, since Generate cubins from cutile at build time and embed them in C++ DSO NVIDIA/cuvs#2548
  2. libcugraph wheels build cuVS from source

This PR fixes that by adding a build-time dependency on cuda-tile for libcugraph.

Notes for Reviewers

Why no conda changes?

libcugraph conda builds do not compile cuVS from source, they link against pre-built packages.

@copy-pr-bot

copy-pr-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ChuckHastings ChuckHastings added bug Something isn't working improvement Improvement / enhancement to an existing function labels Sep 4, 2026
@ChuckHastings
ChuckHastings marked this pull request as ready for review September 4, 2026 18:02
@ChuckHastings
ChuckHastings requested a review from a team as a code owner September 4, 2026 18:02
@ChuckHastings ChuckHastings added non-breaking Non-breaking change and removed improvement Improvement / enhancement to an existing function labels Sep 4, 2026
Comment thread dependencies.yaml Outdated

@jameslamb jameslamb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Started a thread with my comments from our offline conversation, so it's here in the GitHub history.

Comment thread dependencies.yaml Outdated
- common_build
- cpp_build
- cuda
- cutile_python

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

corrects the broken nightly builds.

I went to the build history to figure out what that means specifically. It looks like the failure in CI is CUDA 13 libcugraph wheel builds

  -- Found Python: /pyenv/versions/3.14.7/bin/python (found version "3.14.7") found components: Interpreter
  CMake Error at build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61 (message):
    cuda.tile (cuTile Python) is required to build cuTile embedded kernels.
    Install cutile-python and cuda-tileiras (conda), or cuda-tile[tileiras]
    (pip).

    Interpreter: /pyenv/versions/3.14.7/bin/python

    Import error: Traceback (most recent call last):

      File "<string>", line 1, in <module>
        import cuda.tile

    ModuleNotFoundError: No module named 'cuda'
  Call Stack (most recent call first):
    build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:229 (_cutile_kernels_setup)
    build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/CMakeLists.txt:1161 (generate_cutile_kernels)


  -- Configuring incomplete, errors occurred!

(build link

Changing just the all: group in dependencies.yaml wouldn't fix that, as those wheel builds are built without isolation using the dependencies defined in the py_build_libcugraph and py_rapids_build_libcugraph groups:

rapids-dependency-file-generator \
--output requirements \
--file-key "py_build_${package_name}" \
--file-key "py_rapids_build_${package_name}" \
--matrix "${matrix_selectors}" \
| tee /tmp/requirements-build.txt
rapids-logger "Installing build requirements"
rapids-pip-retry install \
-v \
--prefer-binary \
-r /tmp/requirements-build.txt

Short-term, if these wheels are needed at build time they need to be in the wheel-building dependency set.

Longer-term, it'd be great to switch libcugraph to building against libcuvs wheels to not need to keep making changes like this just for cuVS.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will push a fix here.

@jameslamb jameslamb changed the title Add cuda tile dependency, a new cuvs requirement wheels: fix handling of cuda-tile dependency Sep 8, 2026
@msarahan

msarahan commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

This is currently a blocker on the 26.10 release cycle. Please prioritize getting this merged @ChuckHastings. Thanks for your work on this, @jameslamb

@ChuckHastings

Copy link
Copy Markdown
Collaborator Author

This is currently a blocker on the 26.10 release cycle. Please prioritize getting this merged @ChuckHastings. Thanks for your work on this, @jameslamb

I think this is being superseded by #5654. But in either case, it is blocking all PRs planned for 26.10, so it's our highest priority to get this problem resolved.

rapids-bot Bot pushed a commit that referenced this pull request Sep 9, 2026
## Summary

- add libcuvs to the libcugraph wheel build requirements
- discover only the libcuvs CMake package prefix exported by the installed wheel during direct C++ builds, so pip devcontainers use the installed libcuvs wheel instead of cloning cuVS
- remove direct NVRTC and nvJitLink dependencies that were added for the old source-built/static cuVS integration; libcuvs now owns those requirements

Part of rapidsai/build-planning#325.

## Root causes

There were two independent gaps after #5499 switched libcugraph to dynamic libcuvs linkage:

1. The libcugraph wheel job installs only the py_build_libcugraph and py_rapids_build_libcugraph dependency sets before running pip wheel --no-deps. libcuvs was a runtime dependency but not a build requirement, so it was absent and rapids_cpm_find fell back to a source build.
2. Pip devcontainers already installed libcuvs-cu12/libcuvs-cu13, but build.sh invokes CMake directly. Unlike scikit-build-core, direct CMake did not consume the wheel cmake.prefix entry point, so it still cloned cuVS into _deps/cuvs-src.

The second gap is fixed with the RAPIDS CMake wheel-prefix discovery helper, filtered to libcuvs. RMM and RAFT must remain source-backed in this direct C++ build so the generated cugraph build-tree package carries complete transitive dependency targets for downstream pylibcugraph configuration.

## Dependency audit

- #5483 added cuda-nvrtc-dev because source-built cuVS compiled a file including nvrtc.h.
- #5479 and #5457 added explicit nvJitLink build/runtime requirements for source-built/static cuVS.
- Those direct requirements are no longer needed with dynamic linkage to the prebuilt libcuvs package. The release/26.10 libcuvs conda and wheel metadata already owns NVRTC and nvJitLink.
- The cutile additions proposed in #5646 are not needed by cuGraph once it stops building cuVS.

I also verified NVIDIA/cuML has the same pip-devcontainer issue: its current CUDA 13.3 pip job installs libcuvs-cu13, then logs CPM: Adding package cuvs and uses _deps/cuvs-src. cuML needs the analogous CMake prefix-discovery change and can then remove its direct cutile workaround separately.

## Validation

Using rapidsai/ci-wheel:26.10-cuda13.3.0-rockylinux8-py3.14:

- reproduced the original failure at e6cb493: CMake cloned cuVS and failed under _deps/cuvs-src because cuda.tile was unavailable
- after this change, CMake reports CPM: Using local package cuvs@26.10.0 and cuvs_DIR points into site-packages/libcuvs
- no _deps/cuvs-src directory is created and CMake configuration completes
- generated CUDA 13.3 pip-devcontainer requirements contain libcuvs-cu13 and no direct cuda-tile, NVRTC, or nvJitLink requirements
- generated CUDA 12 requirements select libcuvs-cu12
- strict RAPIDS dependency generation passes
- pre-commit passes on all changed files

Using ghcr.io/rapidsai/cugraph/devcontainer:26.10-cuda13.3-pip with libcuvs, libraft, and librmm wheels installed:

- reproduced the CI regression caused by exposing every wheel prefix: downstream pylibcugraph generation could not resolve nvtx3::nvtx3-cpp or rapids_logger::rapids_logger from wheel-backed RMM/RAFT exports
- verified the filtered prefix discovery selects source RMM/RAFT and wheel libcuvs
- verified both libcugraph and downstream pylibcugraph CMake configure/generate successfully with all three wheel prefixes present

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #5654
@jameslamb

Copy link
Copy Markdown
Member

This is superseded by #5654 and #5656

@jameslamb jameslamb closed this Sep 9, 2026
@ChuckHastings
ChuckHastings deleted the fix_cuda_tile_dependency branch September 11, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants