Skip to content

remove RAPIDS dependencies from cu12, cu13 extras #1549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 6 commits into from
Sep 5, 2025
Merged

remove RAPIDS dependencies from cu12, cu13 extras #1549

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
105ed4b
remove RAPIDS dependencies from cu12, cu13 extras
jameslamb Sep 3, 2025
0b0acd6
ugh, a 25.8 snuck in there
jameslamb Sep 3, 2025
ae52b0a
remove ucx-py
jameslamb Sep 4, 2025
bfe3317
add numba-cuda back to runtime deps
jameslamb Sep 4, 2025
fc467d3
add cuda-core runtime dependency
jameslamb Sep 4, 2025
c49210e
wildcard specs are spelled differently for conda
jameslamb Sep 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conda/recipes/dask-cuda/recipe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ requirements:
run:
- python
- click >=8.1
- cuda-core =0.3
- numba >=0.60.0,<0.62.0a0
- numba-cuda >=0.19.1,<0.20.0a0
- numpy >=1.23,<3.0a0
Expand Down
45 changes: 32 additions & 13 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ dependencies:
packages:
- click >=8.1
- cuda-core==0.3.*
- numba-cuda>=0.19.1,<0.20.0a0
- numpy>=1.23,<3.0a0
# 'nvidia-ml-py' provides the 'pynvml' module, since v12.560.30
# ref: https://github.com/conda-forge/nvidia-ml-py-feedstock/pull/24
Expand All @@ -171,19 +172,13 @@ dependencies:
common:
- output_types: [pyproject]
packages:
- cudf-cu12==25.10.*,>=0.0.0a0
- dask-cudf-cu12==25.10.*,>=0.0.0a0
- distributed-ucxx-cu12==0.46.*,>=0.0.0a0
- kvikio-cu12==25.10.*,>=0.0.0a0
- cuda-core[cu12]==0.3.*
- numba-cuda[cu12]>=0.19.1,<0.20.0a0
extra_cu13:
common:
- output_types: [pyproject]
packages:
- cudf-cu13==25.10.*,>=0.0.0a0
- dask-cudf-cu13==25.10.*,>=0.0.0a0
- distributed-ucxx-cu13==0.46.*,>=0.0.0a0
- kvikio-cu13==25.10.*,>=0.0.0a0
- cuda-core[cu13]==0.3.*
- numba-cuda[cu13]>=0.19.1,<0.20.0a0
test_python:
common:
Expand All @@ -194,11 +189,35 @@ dependencies:
- pytest-timeout
- output_types: [conda]
packages:
- cudf==25.10.*,>=0.0.0a0
- dask-cudf==25.10.*,>=0.0.0a0
- distributed-ucxx==0.46.*,>=0.0.0a0
- kvikio==25.10.*,>=0.0.0a0
- numba-cuda>=0.19.1,<0.20.0a0
Copy link
Member

@pentschev pentschev Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still want numba-cuda to be installed for testing only, this ensures

dask-cuda/dask_cuda/tests/test_gds.py

Lines 30 to 32 in 224a407

elif cuda_lib == "numba.cuda":
data_create = lambda: lib.to_device(range(10))
data_compare = lambda x, y: all(x.copy_to_host() == y.copy_to_host())
still runs.

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

numba-cuda to be installed for testing only

#1531 added numba-cuda as a runtime dependency, don't the reasons for that PR still hold, and therefore shouldn't numba-cuda continue to be a runtime dependency?

Copy link
Member

@pentschev pentschev Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, that's right, the conda environment files got me confused, why is numba-cuda removed from there, shouldn't it be in there too to satisfy it?

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhhh yes YOU are right!

When I saw those removals I thought "ok, makes sense, runtime dependencies get installed via the package metadata for dask-cuda, not pre-installed in the environment" but I don't think that's what we usually do in RAPIDS for those conda env files checked into the repo.

All the other runtime dependencies (like pandas) are in there.

I just pushed bfe3317 restoring it as a runtime dependency. Now numba-cuda will be in those conda env files, and it'll also show up in the wheel's strong runtime dependencies.

I think this should be ok, and is actually desirable.

# depends on: 'numba-cuda>=0.19.1,<0.20.0a0'
pip install dask-cuda

# depends on: 'numba-cuda>=0.19.1,<0.20.0a0', 'numba-cuda[cu12]>=0.19.1,<0.20.0a0'
pip install 'dask-cuda[cu12]'

Having one dependency with constraints and one without should be fine. Tested like this:

docker run \
  --rm \
  -it rapidsai/citestwheel:25.10-cuda12.9.1-ubuntu24.04-py3.13 \
  pip install 'numba-cuda>=0.19.1,<0.20.0a0' 'numba-cuda[cu12]>=0.19.1,<0.20.0a0'

And saw it succeed and respect numba-cuda's constraints, like this:

...
Collecting cuda-bindings<13.0.0,>=12.9.1 (from numba-cuda[cu12]<0.20.0a0,>=0.19.1)
  Downloading cuda_bindings-12.9.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (2.6 kB)
Collecting cuda-core==0.3.* (from numba-cuda[cu12]<0.20.0a0,>=0.19.1)
...

Copy link
Member

@pentschev pentschev Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but wait, we did make a decision to NOT depend on numba-cuda anymore in favor of cuda-bindings, no? I think I'm confused to why we need it to be a runtime in dask-cuda, presumably because of cudf-cu[12,13]? This is what I previously proposed, and I thought that's what we ultimately had done/intended to do in #1536 . Am I mixing things up again?

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From that link I shared, it looks to me like cuda-core depends (optionally, via a [cu12,13] extra) on cuda-bindings, not the other way around.

https://github.com/NVIDIA/cuda-python/blob/fa07cf58ef5c70bb6a42e9aa859487e88f5699a2/cuda_core/pyproject.toml#L50-L51

Depending on cuda-bindings alone won't bring in cuda-core.

docker run \
  --rm \
  --gpus all \
  -it rapidsai/citestwheel:25.10-cuda12.9.1-ubuntu24.04-py3.13 \
  bash

# install 'cuda-bindings' (no version)
pip install cuda-bindings

Installing cuda-bindings without a version pulls in CUDA 13 versions of cuda-bindings and cuda-pathfinder:

Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/rapidsai-wheels-nightly/simple, https://pypi.nvidia.com
Collecting cuda-bindings
  Downloading cuda_bindings-13.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Collecting cuda-pathfinder~=1.1 (from cuda-bindings)
  Downloading cuda_pathfinder-1.2.1-py3-none-any.whl.metadata (3.1 kB)
Downloading cuda_bindings-13.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (12.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 9.7 MB/s  0:00:01
Downloading cuda_pathfinder-1.2.1-py3-none-any.whl (22 kB)
Installing collected packages: cuda-pathfinder, cuda-bindings
Successfully installed cuda-bindings-13.0.1 cuda-pathfinder-1.2.1

And the cuda.core namespace is not available.

python -c "import cuda.core"
# ModuleNotFoundError: No module named 'cuda.core'

So I think we want the following for dask-cuda's runtime requirements

  • dask-cuda -> cuda-core>=12.0,<14
  • dask-cuda[cu12] -> cuda-core[cu12]==12.*
  • dask-cuda[cu13] -> cuda-core[cu13]==13.*

Copy link
Member

@pentschev pentschev Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's what I wanted to say, cuda-bindings would allow a proper existing environment to work. It's looking more and more like there's no way around it, let's just pin cuda-core as you proposed, it seems that is the only correct pin, unfortunately. Could you do that?

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yep, I can do that!

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah one clarification... cuda-core's versioning does not follow the CTK, sorry for that mistake.

So it'd be:

  • dask-cuda -> cuda-core==0.3.*
  • dask-cuda[cu12] -> cuda-core[cu12]==0.3.*
  • dask-cuda[cu13] -> cuda-core[cu13]==0.3.*

Matching this pin that's already there in dask-cuda's test environment:

dask-cuda/dependencies.yaml

Line 162 in c749356

- cuda-core==0.3.*

Copy link
Member Author

@jameslamb jameslamb Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added that in fc467d3

- &cudf_unsuffixed cudf==25.10.*,>=0.0.0a0
- &dask_cudf_unsuffixed dask-cudf==25.10.*,>=0.0.0a0
- &distributed_ucxx_unsuffixed distributed-ucxx==0.46.*,>=0.0.0a0
- &kvikio_unsuffixed kvikio==25.10.*,>=0.0.0a0
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix:
cuda: "12.*"
cuda_suffixed: "true"
packages:
- cudf-cu12==25.10.*,>=0.0.0a0
- dask-cudf-cu12==25.10.*,>=0.0.0a0
- distributed-ucxx-cu12==0.46.*,>=0.0.0a0
- kvikio-cu12==25.10.*,>=0.0.0a0
- matrix:
cuda: "13.*"
cuda_suffixed: "true"
packages:
- cudf-cu13==25.10.*,>=0.0.0a0
- dask-cudf-cu13==25.10.*,>=0.0.0a0
- distributed-ucxx-cu13==0.46.*,>=0.0.0a0
- kvikio-cu13==25.10.*,>=0.0.0a0
- matrix:
packages:
- *cudf_unsuffixed
- *dask_cudf_unsuffixed
- *distributed_ucxx_unsuffixed
- *kvikio_unsuffixed
depends_on_dask_cuda:
common:
- output_types: conda
Expand Down
11 changes: 3 additions & 8 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ requires-python = ">=3.10"
dependencies = [
"click >=8.1",
"cuda-core==0.3.*",
"numba-cuda>=0.19.1,<0.20.0a0",
"numpy>=1.23,<3.0a0",
"nvidia-ml-py>=12.560.30",
"pandas>=1.3",
Expand Down Expand Up @@ -52,17 +53,11 @@ docs = [
"sphinx-rtd-theme>=0.5.1",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit dependencies.yaml and run `rapids-dependency-file-generator`.
cu12 = [
"cudf-cu12==25.10.*,>=0.0.0a0",
"dask-cudf-cu12==25.10.*,>=0.0.0a0",
"distributed-ucxx-cu12==0.46.*,>=0.0.0a0",
"kvikio-cu12==25.10.*,>=0.0.0a0",
"cuda-core[cu12]==0.3.*",
"numba-cuda[cu12]>=0.19.1,<0.20.0a0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit dependencies.yaml and run `rapids-dependency-file-generator`.
cu13 = [
"cudf-cu13==25.10.*,>=0.0.0a0",
"dask-cudf-cu13==25.10.*,>=0.0.0a0",
"distributed-ucxx-cu13==0.46.*,>=0.0.0a0",
"kvikio-cu13==25.10.*,>=0.0.0a0",
"cuda-core[cu13]==0.3.*",
"numba-cuda[cu13]>=0.19.1,<0.20.0a0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down