Skip to content

build and test against CUDA 13.1.0 #2912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 14 commits into from
Jan 7, 2026
+32 −32
Merged

build and test against CUDA 13.1.0 #2912

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
583e7c3
build and test against CUDA 13.1.0
jameslamb Jan 5, 2026
f7ec901
test CUDA 13.1 devcontainers in CI
jameslamb Jan 5, 2026
9e15fab
try system-installing NCCL
jameslamb Jan 6, 2026
fedf7b7
and devcontainers
jameslamb Jan 6, 2026
bf0ff5a
Merge branch 'main' into cuda13.1.0-workflows
jameslamb Jan 6, 2026
fabf7f9
try hacking around with onCreateCommand
jameslamb Jan 6, 2026
c52fe9f
Merge branch 'cuda13.1.0-workflows' of github.com:jameslamb/raft into…
jameslamb Jan 6, 2026
e0839dd
permissions
jameslamb Jan 6, 2026
6fc306f
Merge branch 'main' of github.com:rapidsai/raft into cuda13.1.0-workf…
jameslamb Jan 6, 2026
7a1aba9
use new CI containers, make devcontainers workaround stricter
jameslamb Jan 6, 2026
36380aa
revert devcontainers to 13.0
jameslamb Jan 6, 2026
7b5d9dc
actually restore CUDA 13.0 devcontainers
jameslamb Jan 6, 2026
a56e104
restore CUDA 13.1 devcontainers (just do not test them in CI)
jameslamb Jan 7, 2026
946981f
switch back to CUDA 13.1 devcontainers
jameslamb Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .devcontainer/cuda13.1-pip/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
"installcuBLAS": true,
"installcuSOLVER": true,
"installcuRAND": true,
"installcuSPARSE": true
"installcuSPARSE": true,
"installNCCL": true

@jameslamb jameslamb Jan 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not working, and I see why.

The code processing this flag is explicitly looking for packages with +cuda13.1 in their versions.

if [ "${INSTALLNCCL:-false}" = true ] \
&& test -n "$(apt-cache search libnccl2 2>/dev/null)" \
&& apt-cache policy libnccl2 2>/dev/null | grep -q "+${cuda_tag}"; then
    PKGS+=("libnccl2=*+${cuda_tag}");
    if [ "${INSTALLDEVPACKAGES:-false}" = true ]; then
        PKGS+=("libnccl-dev=*+${cuda_tag}");
    fi
fi

(rapidsai/devcontainers - features/src/cuda/install.sh)

And those don't exist today.

So options I see:

  • relax that condition
  • add a manual install in the devcontainer.json here and in other RAPIDS repos (like in postCreateCommand or something)
  • get RAFT wheel builds using the files delivered by nvidia-nccl-cu{12,13} wheels (build and test against CUDA 13.1.0 #2912 (comment))

@jakirkham jakirkham Jan 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That code is 3 years old. Sounds worthwhile to revisit whether it can be changed

@jameslamb jameslamb Jan 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sure. I hacked around it here like this: fabf7f9

which seems to at least allow all RAFT components to compile: https://github.com/rapidsai/raft/actions/runs/20757771141/job/59604424645?pr=2912

But it definitely feels like a hack. Opened rapidsai/devcontainers#644 to discuss

@jameslamb jameslamb Jan 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We decided offline to skip CUDA 13.1 pip devcontainers in CI for now to keep moving, and to restore them hopefully soon when there are CUDA 13.1 apt packages for NCCL: rapidsai/devcontainers#644 (comment)

@jameslamb jameslamb Jan 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed a56e104

Thinking about this some more, we should at least still include the CUDA 13.1 devcontainers in the affected repos, even if we can't test them in CI, so they can be used for other testing. For example, the cuda13.1-conda devcontainers are not affected by this issue and could be used to debug CUDA 13.1 builds.

@jameslamb jameslamb Jan 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are now apt packages for NCCL + CUDA 13.1, so no need to skip anything any more 😁

#2912 (comment)

},
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:26.2": {}
},
Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ concurrency:
jobs:
cpp-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -46,7 +46,7 @@ jobs:
python-build:
needs: [cpp-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -57,7 +57,7 @@ jobs:
upload-conda:
needs: [cpp-build, python-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -67,7 +67,7 @@ jobs:
if: github.ref_type == 'branch'
needs: python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@cuda-13.1.0
with:
arch: "amd64"
branch: ${{ inputs.branch }}
Expand All @@ -79,7 +79,7 @@ jobs:
sha: ${{ inputs.sha }}
wheel-build-libraft:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -95,7 +95,7 @@ jobs:
wheel-publish-libraft:
needs: wheel-build-libraft
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -106,7 +106,7 @@ jobs:
wheel-build-pylibraft:
needs: wheel-build-libraft
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -120,7 +120,7 @@ jobs:
wheel-publish-pylibraft:
needs: wheel-build-pylibraft
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -131,7 +131,7 @@ jobs:
wheel-build-raft-dask:
needs: wheel-build-libraft
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -145,7 +145,7 @@ jobs:
wheel-publish-raft-dask:
needs: wheel-build-raft-dask
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand Down
32 changes: 16 additions & 16 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
- devcontainer
- telemetry-setup
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@cuda-13.1.0
if: always()
with:
needs: ${{ toJSON(needs) }}
Expand All @@ -53,7 +53,7 @@ jobs:
changed-files:
secrets: inherit
needs: telemetry-setup
uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@cuda-13.1.0
with:
files_yaml: |
test_cpp:
Expand Down Expand Up @@ -83,14 +83,14 @@ jobs:
checks:
secrets: inherit
needs: telemetry-setup
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@cuda-13.1.0
with:
enable_check_generated_files: false
ignored_pr_jobs: telemetry-summarize
conda-cpp-build:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@cuda-13.1.0
with:
build_type: pull-request
script: ci/build_cpp.sh
Expand All @@ -99,7 +99,7 @@ jobs:
conda-cpp-tests:
needs: [conda-cpp-build, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@cuda-13.1.0
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp
with:
build_type: pull-request
Expand All @@ -108,20 +108,20 @@ jobs:
conda-cpp-checks:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@cuda-13.1.0
with:
build_type: pull-request
conda-python-build:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@cuda-13.1.0
with:
build_type: pull-request
script: ci/build_python.sh
conda-python-tests:
needs: [conda-python-build, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@cuda-13.1.0
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
Expand All @@ -130,7 +130,7 @@ jobs:
docs-build:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@cuda-13.1.0
with:
build_type: pull-request
node_type: "gpu-l4-latest-1"
Expand All @@ -140,7 +140,7 @@ jobs:
wheel-build-libraft:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: pull-request
branch: ${{ inputs.branch }}
Expand All @@ -156,7 +156,7 @@ jobs:
wheel-build-pylibraft:
needs: [checks, wheel-build-libraft]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: pull-request
node_type: cpu8
Expand All @@ -167,7 +167,7 @@ jobs:
wheel-tests-pylibraft:
needs: [wheel-build-pylibraft, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda-13.1.0
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
Expand All @@ -176,7 +176,7 @@ jobs:
wheel-build-raft-dask:
needs: [checks, wheel-build-libraft]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@cuda-13.1.0
with:
build_type: pull-request
node_type: cpu8
Expand All @@ -187,7 +187,7 @@ jobs:
wheel-tests-raft-dask:
needs: [wheel-build-raft-dask, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda-13.1.0
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
Expand All @@ -196,10 +196,10 @@ jobs:
devcontainer:
needs: telemetry-setup
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@cuda-13.1.0
with:
arch: '["amd64", "arm64"]'
cuda: '["13.0"]'
cuda: '["13.1"]'
node_type: "cpu8"
rapids-aux-secret-1: GIST_REPO_READ_ORG_GITHUB_TOKEN
env: |
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ on:
jobs:
conda-cpp-checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
conda-cpp-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand All @@ -43,7 +43,7 @@ jobs:
sccache-dist-token-secret-name: GIST_REPO_READ_ORG_GITHUB_TOKEN
conda-python-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand All @@ -53,7 +53,7 @@ jobs:
sccache-dist-token-secret-name: GIST_REPO_READ_ORG_GITHUB_TOKEN
wheel-tests-pylibraft:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand All @@ -63,7 +63,7 @@ jobs:
sccache-dist-token-secret-name: GIST_REPO_READ_ORG_GITHUB_TOKEN
wheel-tests-raft-dask:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@cuda-13.1.0
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/trigger-breaking-change-alert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
trigger-notifier:
if: contains(github.event.pull_request.labels.*.name, 'breaking')
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@cuda-13.1.0
with:
sender_login: ${{ github.event.sender.login }}
sender_avatar: ${{ github.event.sender.avatar_url }}
Expand Down
5 changes: 4 additions & 1 deletion ci/build_wheel_libraft.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION.
# SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

source rapids-init-pip

# TODO(jameslamb): revert this when it's pre-installed in CI images
source ci/install_libnccl.sh

package_name="libraft"
package_dir="python/libraft"

Expand Down
5 changes: 4 additions & 1 deletion ci/build_wheel_pylibraft.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2023-2024, NVIDIA CORPORATION.
# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

source rapids-init-pip

# TODO(jameslamb): revert this when it's pre-installed in CI images
source ci/install_libnccl.sh

package_dir="python/pylibraft"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
Expand Down
5 changes: 4 additions & 1 deletion ci/build_wheel_raft_dask.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2023-2025, NVIDIA CORPORATION.
# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

source rapids-init-pip

# TODO(jameslamb): revert this when it's pre-installed in CI images
source ci/install_libnccl.sh

package_dir="python/raft-dask"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
Expand Down
15 changes: 15 additions & 0 deletions ci/install_libnccl.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION.
# SPDX-License-Identifier: Apache-2.0

# TODO(jameslamb): revert this when it's pre-installed in CI images
rapids-logger "installing libnccl"
if type -f apt; then
apt-get update
apt-get install -y --no-install-recommends \
libnccl-dev
else
yum update -y
yum install -y \
libnccl-devel
fi
Loading