Skip to content

refactor: use libnvcomp from libkvikio wheel to unblock Python 3.13 upgrade #18593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 17 commits into from
May 2, 2025
Merged

refactor: use libnvcomp from libkvikio wheel to unblock Python 3.13 upgrade #18593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
569295a
[DO NOT MERGE]: test upstream libkvikio wheel with vendored nvcomp
gforsyth Apr 29, 2025
9b40d9f
bucket path
gforsyth Apr 29, 2025
30073fe
tarball and wheel have different suffixes
gforsyth Apr 29, 2025
0626b60
update sha and version suffix for updated kvikio
gforsyth Apr 29, 2025
c9f7402
make sure we use the updated kvikio wheel everywhere
gforsyth Apr 29, 2025
66b507e
cleaner downloading of artifacts from kvikio PR
gforsyth Apr 29, 2025
c0af5c7
disable using nvcomp runtime wheel
gforsyth Apr 29, 2025
c351029
Revert "cleaner downloading of artifacts from kvikio PR"
gforsyth Apr 29, 2025
8e5378a
Revert "disable using nvcomp runtime wheel"
gforsyth Apr 29, 2025
97faca6
try to fix install paths
gforsyth Apr 30, 2025
6e75249
update libcudf RPATH for libkvikio-bundled nvcomp
gforsyth Apr 30, 2025
3121d4e
don't export nvcomp
gforsyth Apr 30, 2025
a3d9625
make sure upstream libkvikio is installed for pandas and polars tests
gforsyth Apr 30, 2025
d39e41a
and again
gforsyth Apr 30, 2025
8970333
yet another spot to constrain install
gforsyth May 1, 2025
b16cbe6
refactor: use upstream nightly wheel for libkvikio+nvcomp
gforsyth May 2, 2025
8f9990d
Merge branch 'branch-25.06' into libkvikio_vendor_nvcomp
gforsyth May 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion ci/build_wheel_libcudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,11 @@ rapids-pip-retry install \
# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735)
export PIP_NO_BUILD_ISOLATION=0

export SKBUILD_CMAKE_ARGS="-DUSE_NVCOMP_RUNTIME_WHEEL=ON"
# TODO(nvcomp): when `nvcomp` supports Python 3.13 and we de-vendor `nvcomp` from `kvikio`
# this should be switched back to using the nvcomp runtime wheel
# https://github.com/rapidsai/build-planning/issues/171
# export SKBUILD_CMAKE_ARGS="-DUSE_NVCOMP_RUNTIME_WHEEL=ON"
export SKBUILD_CMAKE_ARGS="-DUSE_NVCOMP_FROM_LIBKVIKIO_WHEEL=ON"
./ci/build_wheel.sh "${package_name}" "${package_dir}"

# repair wheels and write to the location that artifact-uploading code expects to find them
Expand Down
5 changes: 4 additions & 1 deletion dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,10 @@ files:
includes:
- depends_on_libkvikio
- depends_on_librmm
- depends_on_nvcomp
# TODO(nvcomp): when `nvcomp` supports Python 3.13 and we de-vendor `nvcomp` from `kvikio`
# this dependency should be restored
# https://github.com/rapidsai/build-planning/issues/171
# - depends_on_nvcomp
- depends_on_rapids_logger
py_build_pylibcudf:
output: pyproject
Expand Down
20 changes: 19 additions & 1 deletion python/libcudf/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ project(

option(USE_NVCOMP_RUNTIME_WHEEL "Use the nvcomp wheel at runtime instead of the system library" OFF)

# TODO(nvcomp): when `nvcomp` supports Python 3.13 and we de-vendor `nvcomp` from `kvikio` this
# option should be removed https://github.com/rapidsai/build-planning/issues/171
option(USE_NVCOMP_FROM_LIBKVIKIO_WHEEL "Use nvcomp bundled with libkvikio" OFF)

# Check if cudf is already available. If so, it is the user's responsibility to ensure that the
# CMake package is also available at build time of the Python cudf package.
find_package(cudf "${RAPIDS_VERSION}")
Expand All @@ -38,7 +42,10 @@ set(BUILD_TESTS OFF)
set(BUILD_BENCHMARKS OFF)
set(CUDF_BUILD_TESTUTIL OFF)
set(CUDF_BUILD_STREAMS_TEST_UTIL OFF)
if(USE_NVCOMP_RUNTIME_WHEEL)

# TODO(nvcomp): when `nvcomp` supports Python 3.13 and we de-vendor `nvcomp` from `kvikio` the
# libkvikio branch should be removed
if(USE_NVCOMP_RUNTIME_WHEEL OR USE_NVCOMP_FROM_LIBKVIKIO_WHEEL)
set(CUDF_EXPORT_NVCOMP OFF)
endif()
set(CUDA_STATIC_RUNTIME ON)
Expand All @@ -55,3 +62,14 @@ if(USE_NVCOMP_RUNTIME_WHEEL)
APPEND
)
endif()

# TODO(nvcomp): when `nvcomp` supports Python 3.13 and we de-vendor `nvcomp` from `kvikio` this
# block should be removed
if(USE_NVCOMP_FROM_LIBKVIKIO_WHEEL)

@jameslamb jameslamb May 2, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just putting this in writing to make it explicit... I think it's ok to allow USE_NVCOMP_RUNTIME_WHEEL and and USE_NVCOMP_FROM_LIBVIKIO_WHEEL to both be ON, as you've done here.

Having both RPATHs set on libcudf.so, even if accidentally, shouldn't be a problem. But just calling out that possibility, to give you an opportunity to say "oh yeah, I hadn't thought of that... ok no that's bad, let's make them mutually exclusive".

@gforsyth gforsyth May 2, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good callout -- I do think that it's ok if they're both set. It should be a temporary addition, in any case.

set(rpaths "$ORIGIN/../../libkvikio/lib64")
set_property(
TARGET cudf
PROPERTY INSTALL_RPATH ${rpaths}
APPEND
)
endif()
1 change: 0 additions & 1 deletion python/libcudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ classifiers = [
dependencies = [
"libkvikio==25.6.*,>=0.0.0a0",
"librmm==25.6.*,>=0.0.0a0",
"nvidia-nvcomp==4.2.0.11",
"rapids-logger==0.1.*,>=0.0.0a0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down