wheels: fix handling of cuda-tile dependency - #5646
ChuckHastings wants to merge 2 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
jameslamb
left a comment
There was a problem hiding this comment.
Started a thread with my comments from our offline conversation, so it's here in the GitHub history.
| - common_build | ||
| - cpp_build | ||
| - cuda | ||
| - cutile_python |
There was a problem hiding this comment.
corrects the broken nightly builds.
I went to the build history to figure out what that means specifically. It looks like the failure in CI is CUDA 13 libcugraph wheel builds
-- Found Python: /pyenv/versions/3.14.7/bin/python (found version "3.14.7") found components: Interpreter
CMake Error at build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61 (message):
cuda.tile (cuTile Python) is required to build cuTile embedded kernels.
Install cutile-python and cuda-tileiras (conda), or cuda-tile[tileiras]
(pip).
Interpreter: /pyenv/versions/3.14.7/bin/python
Import error: Traceback (most recent call last):
File "<string>", line 1, in <module>
import cuda.tile
ModuleNotFoundError: No module named 'cuda'
Call Stack (most recent call first):
build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:229 (_cutile_kernels_setup)
build/py3-none-linux_x86_64/_deps/cuvs-src/cpp/CMakeLists.txt:1161 (generate_cutile_kernels)
-- Configuring incomplete, errors occurred!
Changing just the all: group in dependencies.yaml wouldn't fix that, as those wheel builds are built without isolation using the dependencies defined in the py_build_libcugraph and py_rapids_build_libcugraph groups:
cugraph/ci/build_wheel_libcugraph.sh
Lines 15 to 26 in cb14128
Short-term, if these wheels are needed at build time they need to be in the wheel-building dependency set.
Longer-term, it'd be great to switch libcugraph to building against libcuvs wheels to not need to keep making changes like this just for cuVS.
d11afd5 to
f2bb1e9
Compare
|
This is currently a blocker on the 26.10 release cycle. Please prioritize getting this merged @ChuckHastings. Thanks for your work on this, @jameslamb |
I think this is being superseded by #5654. But in either case, it is blocking all PRs planned for 26.10, so it's our highest priority to get this problem resolved. |
## Summary - add libcuvs to the libcugraph wheel build requirements - discover only the libcuvs CMake package prefix exported by the installed wheel during direct C++ builds, so pip devcontainers use the installed libcuvs wheel instead of cloning cuVS - remove direct NVRTC and nvJitLink dependencies that were added for the old source-built/static cuVS integration; libcuvs now owns those requirements Part of rapidsai/build-planning#325. ## Root causes There were two independent gaps after #5499 switched libcugraph to dynamic libcuvs linkage: 1. The libcugraph wheel job installs only the py_build_libcugraph and py_rapids_build_libcugraph dependency sets before running pip wheel --no-deps. libcuvs was a runtime dependency but not a build requirement, so it was absent and rapids_cpm_find fell back to a source build. 2. Pip devcontainers already installed libcuvs-cu12/libcuvs-cu13, but build.sh invokes CMake directly. Unlike scikit-build-core, direct CMake did not consume the wheel cmake.prefix entry point, so it still cloned cuVS into _deps/cuvs-src. The second gap is fixed with the RAPIDS CMake wheel-prefix discovery helper, filtered to libcuvs. RMM and RAFT must remain source-backed in this direct C++ build so the generated cugraph build-tree package carries complete transitive dependency targets for downstream pylibcugraph configuration. ## Dependency audit - #5483 added cuda-nvrtc-dev because source-built cuVS compiled a file including nvrtc.h. - #5479 and #5457 added explicit nvJitLink build/runtime requirements for source-built/static cuVS. - Those direct requirements are no longer needed with dynamic linkage to the prebuilt libcuvs package. The release/26.10 libcuvs conda and wheel metadata already owns NVRTC and nvJitLink. - The cutile additions proposed in #5646 are not needed by cuGraph once it stops building cuVS. I also verified NVIDIA/cuML has the same pip-devcontainer issue: its current CUDA 13.3 pip job installs libcuvs-cu13, then logs CPM: Adding package cuvs and uses _deps/cuvs-src. cuML needs the analogous CMake prefix-discovery change and can then remove its direct cutile workaround separately. ## Validation Using rapidsai/ci-wheel:26.10-cuda13.3.0-rockylinux8-py3.14: - reproduced the original failure at e6cb493: CMake cloned cuVS and failed under _deps/cuvs-src because cuda.tile was unavailable - after this change, CMake reports CPM: Using local package cuvs@26.10.0 and cuvs_DIR points into site-packages/libcuvs - no _deps/cuvs-src directory is created and CMake configuration completes - generated CUDA 13.3 pip-devcontainer requirements contain libcuvs-cu13 and no direct cuda-tile, NVRTC, or nvJitLink requirements - generated CUDA 12 requirements select libcuvs-cu12 - strict RAPIDS dependency generation passes - pre-commit passes on all changed files Using ghcr.io/rapidsai/cugraph/devcontainer:26.10-cuda13.3-pip with libcuvs, libraft, and librmm wheels installed: - reproduced the CI regression caused by exposing every wheel prefix: downstream pylibcugraph generation could not resolve nvtx3::nvtx3-cpp or rapids_logger::rapids_logger from wheel-backed RMM/RAFT exports - verified the filtered prefix discovery selects source RMM/RAFT and wheel libcuvs - verified both libcugraph and downstream pylibcugraph CMake configure/generate successfully with all three wheel prefixes present Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) - Chuck Hastings (https://github.com/ChuckHastings) URL: #5654
(heavily edited by @jameslamb on Sep 8, 2026)
Contributes to rapidsai/build-planning#324
Wheel builds have been failing like this:
(build link
Because:
cuda-tile(Python library) at build time, since Generate cubins fromcutileat build time and embed them in C++ DSO NVIDIA/cuvs#2548libcugraphwheels build cuVS from sourceThis PR fixes that by adding a build-time dependency on
cuda-tileforlibcugraph.Notes for Reviewers
Why no conda changes?
libcugraphconda builds do not compile cuVS from source, they link against pre-built packages.