Skip to content

refactor: use libnvcomp from libkvikio wheel to unblock Python 3.13 upgrade #18593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 17 commits into from
May 2, 2025
Merged

refactor: use libnvcomp from libkvikio wheel to unblock Python 3.13 upgrade #18593

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
569295a
[DO NOT MERGE]: test upstream libkvikio wheel with vendored nvcomp
gforsyth Apr 29, 2025
9b40d9f
bucket path
gforsyth Apr 29, 2025
30073fe
tarball and wheel have different suffixes
gforsyth Apr 29, 2025
0626b60
update sha and version suffix for updated kvikio
gforsyth Apr 29, 2025
c9f7402
make sure we use the updated kvikio wheel everywhere
gforsyth Apr 29, 2025
66b507e
cleaner downloading of artifacts from kvikio PR
gforsyth Apr 29, 2025
c0af5c7
disable using nvcomp runtime wheel
gforsyth Apr 29, 2025
c351029
Revert "cleaner downloading of artifacts from kvikio PR"
gforsyth Apr 29, 2025
8e5378a
Revert "disable using nvcomp runtime wheel"
gforsyth Apr 29, 2025
97faca6
try to fix install paths
gforsyth Apr 30, 2025
6e75249
update libcudf RPATH for libkvikio-bundled nvcomp
gforsyth Apr 30, 2025
3121d4e
don't export nvcomp
gforsyth Apr 30, 2025
a3d9625
make sure upstream libkvikio is installed for pandas and polars tests
gforsyth Apr 30, 2025
d39e41a
and again
gforsyth Apr 30, 2025
8970333
yet another spot to constrain install
gforsyth May 1, 2025
b16cbe6
refactor: use upstream nightly wheel for libkvikio+nvcomp
gforsyth May 2, 2025
8f9990d
Merge branch 'branch-25.06' into libkvikio_vendor_nvcomp
gforsyth May 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions ci/build_wheel_libcudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,16 @@ rapids-dependency-file-generator \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \
| tee /tmp/requirements-build.txt


set -x
#kvikio_wheel_cpp_libkvikio_cu11_aarch64.tar.gz
LIBKVIKIO_WHL="libkvikio_${RAPIDS_PY_CUDA_SUFFIX}-25.6.0a32-py3-none-${AUDITWHEEL_PLAT}.whl"

@bdice bdice Apr 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can follow the instructions in rapidsai/docs#593 for a more concise solution.

Maybe something like:

# ci/use_wheels_from_prs.sh
RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")

# download wheels, store the directories holding them in variables
LIBKVIKIO_WHEELHOUSE=$(
  RAPIDS_PY_WHEEL_NAME="libkvikio_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github kvikio 702 cpp
)
KVIKIO_WHEELHOUSE=$(
  RAPIDS_PY_WHEEL_NAME="kvikio_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github kvikio 702 python
)

# write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel"
cat > /tmp/constraints.txt <<EOF
libkvikio-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBKVIKIO_WHEELHOUSE}/libkvikio_*.whl)
kvikio-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${KVIKIO_WHEELHOUSE}/kvikio_*.whl)
EOF
export PIP_CONSTRAINT=/tmp/constraints.txt

@gforsyth gforsyth Apr 30, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to work unless I specify the GITHUB_RUN_ID, at which point I'm doing roughly the same amount of work

LIBKVIKIO_TARBALL="kvikio_wheel_cpp_libkvikio_${RAPIDS_PY_CUDA_SUFFIX}_${AUDITWHEEL_ARCH}.tar.gz"
LIBKVIKIO_DIR=$(rapids-get-artifact "ci/kvikio/pull-request/702/7f957eb/${LIBKVIKIO_TARBALL}")
echo "libkvikio-${RAPIDS_PY_CUDA_SUFFIX} @ file://${LIBKVIKIO_DIR}/${LIBKVIKIO_WHL}" > /tmp/constraints.txt
export PIP_CONSTRAINT="/tmp/constraints.txt"
set +x

rapids-logger "Installing build requirements"
rapids-pip-retry install \
-v \
Expand Down
2 changes: 1 addition & 1 deletion dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ files:
includes:
- depends_on_libkvikio
- depends_on_librmm
- depends_on_nvcomp
# - depends_on_nvcomp
- depends_on_rapids_logger
py_build_pylibcudf:
output: pyproject
Expand Down
1 change: 0 additions & 1 deletion python/libcudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ classifiers = [
dependencies = [
"libkvikio==25.6.*,>=0.0.0a0",
"librmm==25.6.*,>=0.0.0a0",
"nvidia-nvcomp==4.2.0.11",
"rapids-logger==0.1.*,>=0.0.0a0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down