Skip to content

Build and test with CUDA 13.0.0 #1536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 11 commits into from
Aug 26, 2025
Merged

Build and test with CUDA 13.0.0 #1536

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2fbe13a
Build and test with CUDA 13.0.0
jameslamb Aug 19, 2025
25d028c
update dependencies.yaml
jameslamb Aug 19, 2025
1824e35
empty commit to fully re-build and re-test with moved numba-cuda pins
jameslamb Aug 20, 2025
37e3086
merge main
jameslamb Aug 22, 2025
caf5b43
merge main
jameslamb Aug 22, 2025
08d5a84
introduce [cu12] and [cu13] extras
jameslamb Aug 25, 2025
041fab0
try to make 'cudf' and 'dask_cudf' optional for tests
jameslamb Aug 25, 2025
4a7e4db
add cudf back to Python 3.12 environment
jameslamb Aug 25, 2025
c457f1f
add dask[dataframe] to wheel testing dependencies
jameslamb Aug 25, 2025
97f2bfb
restore cudf and dask-cudf on CUDA 12, update comment, fix == for kvikio
jameslamb Aug 25, 2025
087d1fe
remove numactl
jameslamb Aug 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-ucxx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ concurrency:
jobs:
conda-python-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ concurrency:
jobs:
conda-python-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -45,7 +45,7 @@ jobs:
if: github.ref_type == 'branch'
needs: [conda-python-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@cuda13.0
with:
arch: "amd64"
branch: ${{ inputs.branch }}
Expand All @@ -58,15 +58,15 @@ jobs:
upload-conda:
needs: [conda-python-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
wheel-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -83,7 +83,7 @@ jobs:
wheel-publish:
needs: wheel-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- wheel-tests
- telemetry-setup
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@cuda13.0
with:
needs: ${{ toJSON(needs) }}
telemetry-setup:
Expand All @@ -34,27 +34,27 @@ jobs:
checks:
secrets: inherit
needs: telemetry-setup
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@cuda13.0
with:
ignored_pr_jobs: telemetry-summarize
conda-python-build:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@cuda13.0
with:
build_type: pull-request
script: ci/build_python.sh
conda-python-tests:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@cuda13.0
with:
build_type: pull-request
script: ci/test_python.sh
docs-build:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@cuda13.0
with:
build_type: pull-request
node_type: "gpu-l4-latest-1"
Expand All @@ -64,7 +64,7 @@ jobs:
wheel-build:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda13.0
with:
build_type: pull-request
# Package is pure Python and only ever requires one build.
Expand All @@ -78,7 +78,7 @@ jobs:
wheel-tests:
needs: wheel-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda13.0
with:
build_type: pull-request
script: "ci/test_wheel.sh"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-ucxx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ on:
jobs:
conda-python-ucxx-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ on:
jobs:
conda-python-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand All @@ -34,7 +34,7 @@ jobs:
sha: ${{ inputs.sha }}
wheel-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda13.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/trigger-breaking-change-alert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
trigger-notifier:
if: contains(github.event.pull_request.labels.*.name, 'breaking')
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@branch-25.10
uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@cuda13.0
with:
sender_login: ${{ github.event.sender.login }}
sender_avatar: ${{ github.event.sender.avatar_url }}
Expand Down
37 changes: 37 additions & 0 deletions conda/environments/all_cuda-130_arch-aarch64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
- click >=8.1
- cuda-nvcc-impl
- cuda-nvrtc
- cuda-version=13.0
- cudf==25.10.*,>=0.0.0a0
- dask-cudf==25.10.*,>=0.0.0a0
- distributed-ucxx==0.46.*,>=0.0.0a0
- kvikio==25.10.*,>=0.0.0a0
- numactl-devel-cos7-aarch64
- numba-cuda>=0.19.0,<0.20.0a0
- numba>=0.60.0,<0.62.0a0
- numpy>=1.23,<3.0a0
- numpydoc>=1.1.0
- pandas>=1.3
- pre-commit
- pynvml>=12.0.0,<13.0.0a0
- pytest
- pytest-cov
- pytest-timeout
- python>=3.10,<3.14
- rapids-build-backend>=0.4.0,<0.5.0dev0
- rapids-dask-dependency==25.10.*,>=0.0.0a0
- setuptools>=64.0.0
- sphinx
- sphinx-click>=2.7.1
- sphinx-rtd-theme>=0.5.1
- ucx-py==0.46.*,>=0.0.0a0
- ucxx==0.46.*,>=0.0.0a0
- zict>=2.0.0
name: all_cuda-130_arch-aarch64
37 changes: 37 additions & 0 deletions conda/environments/all_cuda-130_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
- click >=8.1
- cuda-nvcc-impl
- cuda-nvrtc
- cuda-version=13.0
- cudf==25.10.*,>=0.0.0a0
- dask-cudf==25.10.*,>=0.0.0a0
- distributed-ucxx==0.46.*,>=0.0.0a0
- kvikio==25.10.*,>=0.0.0a0
- numactl-devel-cos7-x86_64
- numba-cuda>=0.19.0,<0.20.0a0
- numba>=0.60.0,<0.62.0a0
- numpy>=1.23,<3.0a0
- numpydoc>=1.1.0
- pandas>=1.3
- pre-commit
- pynvml>=12.0.0,<13.0.0a0
- pytest
- pytest-cov
- pytest-timeout
- python>=3.10,<3.14
- rapids-build-backend>=0.4.0,<0.5.0dev0
- rapids-dask-dependency==25.10.*,>=0.0.0a0
- setuptools>=64.0.0
- sphinx
- sphinx-click>=2.7.1
- sphinx-rtd-theme>=0.5.1
- ucx-py==0.46.*,>=0.0.0a0
- ucxx==0.46.*,>=0.0.0a0
- zict>=2.0.0
name: all_cuda-130_arch-x86_64
33 changes: 23 additions & 10 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ files:
all:
output: conda
matrix:
cuda: ["12.9"]
cuda: ["12.9", "13.0"]
arch: [x86_64, aarch64]
includes:
- build_python
Expand Down Expand Up @@ -92,15 +92,16 @@ dependencies:
cuda: "12.9"
packages:
- cuda-version=12.9
cuda:
specific:
- output_types: conda
matrices:
- matrix:
cuda: "12.*"
cuda: "13.0"
packages:
- cuda-nvcc-impl
- cuda-nvrtc
- cuda-version=13.0
cuda:
common:
- output_types: conda
packages:
- cuda-nvcc-impl
- cuda-nvrtc
develop:
common:
- output_types: [conda, requirements]
Expand Down Expand Up @@ -156,10 +157,13 @@ dependencies:
matrices:
- matrix: {cuda: "12.*"}
packages:
- &numba_cuda_cu12 numba-cuda[cu12]>=0.19.0,<0.20.0a0
- numba-cuda[cu12]>=0.19.0,<0.20.0a0
- matrix: {cuda: "13.*"}
packages:
- &numba_cuda_cu13 numba-cuda[cu13]>=0.19.0,<0.20.0a0
- matrix: # Fallback for no matrix
packages:
- *numba_cuda_cu12
- *numba_cuda_cu13

@jameslamb jameslamb Aug 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blegh, ok we have to do something here.

CUDA 12.9 wheel tests are failing like this:

ERROR: Cannot install -r /tmp/requirements-test.txt (line 3) and numba-cuda[cu13]==0.19.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    numba-cuda[cu13] 0.19.0 depends on cuda-python==13.*; extra == "cu13"
    cudf-cu12 25.10.0a261 depends on cuda-python<13.0a0 and >=12.9.1
    cudf-cu12 25.10.0a260 depends on cuda-python<13.0a0 and >=12.9.1

(build link)

The fallback matrix is used because of this:

dask-cuda/pyproject.toml

Line 127 in 0c83e51

disable-cuda = true

We distribute unsuffixed packages here (dask-cuda, not dask-cuda-cu{12,13}). Which has been fine, because this project's code isn't directly CUDA-major-version dependent and neither was its list of dependencies.... until #1531

That PR introduced a CUDA-major-version-specific dependency like numba-cuda[cu12]. When I reviewed that, I forgot dask-cuda isn't suffixed 😭

This is a problem because now we want to depend on numba-cuda[cu12] when targeting CUDA 12 and numba-cuda[cu13] when targeting CUDA 13.

The options I see are:

  • depend on numba-cuda without extras
  • start distributing suffixed packages like dask-cuda-cu{12,13}

It looks like those extras are probably necessary for correctness:

https://github.com/NVIDIA/numba-cuda/blob/d2827b7701289932ba390cdbf90642dcc7b2eeab/pyproject.toml#L26-L44

But I'm not sure. What do you think @bdice @brandon-b-miller @pentschev @TomAugspurger ?

@jameslamb jameslamb Aug 21, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked offline and @bdice mentioned it might be possible to factor the numba.cuda usage here out / down in a way that allows us to continue publishing unsuffixed dask-cuda wheels.

@jameslamb jameslamb Aug 22, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying with this change: #1537

@jameslamb jameslamb Aug 25, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After #1537, in more offline discussion we agreed to try adding [cu12] and [cu13] extras for wheels, to ensure that users can get an environment with a compatible set of packages based on CUDA version.

test_python:
common:
- output_types: [conda, requirements, pyproject]
Expand Down Expand Up @@ -197,6 +201,15 @@ dependencies:
- distributed-ucxx-cu12==0.46.*,>=0.0.0a0
- kvikio-cu12==25.10.*,>=0.0.0a0
- ucx-py-cu12==0.46.*,>=0.0.0a0
- matrix:
cuda: "13.*"
cuda_suffixed: "true"
packages:
- cudf-cu13==25.10.*,>=0.0.0a0
- dask-cudf-cu13==25.10.*,>=0.0.0a0
- distributed-ucxx-cu13==0.46.*,>=0.0.0a0
- kvikio-cu13==25.10.*,>=0.0.0a0
- ucx-py-cu13==0.46.*,>=0.0.0a0
- matrix:
packages:
- *cudf_unsuffixed
Expand Down
2 changes: 1 addition & 1 deletion docs/source/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ To install the latest version of Dask-CUDA along with the CUDA Toolkit:

.. code-block:: bash

conda install -c rapidsai -c conda-forge -c nvidia dask-cuda cuda-version=12.9
conda install -c rapidsai -c conda-forge -c nvidia dask-cuda cuda-version=13.0

Pip
---
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ license = { text = "Apache-2.0" }
requires-python = ">=3.10"
dependencies = [
"click >=8.1",
"numba-cuda[cu12]>=0.19.0,<0.20.0a0",
"numba-cuda[cu13]>=0.19.0,<0.20.0a0",
"numba>=0.60.0,<0.62.0a0",
"numpy>=1.23,<3.0a0",
"pandas>=1.3",
Expand Down
Loading