Add CI jobs for PyTorch and CuPy integration tests

[bdice]

Description

Closes #2239.

Adds separate CI jobs for testing RMM integrations with PyTorch and CuPy.
This splits tests into minimal (no PyTorch/CuPy) and integration test jobs for both conda and wheel builds.

• Add test_pytorch and test_cupy file keys to dependencies.yaml
• Add depends_on_pytorch and depends_on_cupy dependency groups with CUDA-version-specific package specifications for conda and pip
• Add ci/test_python_integrations.sh for conda-based integration tests
• Add ci/test_wheel_integrations.sh for wheel-based integration tests
• Add conda-python-tests-integration-optional and wheel-tests-integration-optional jobs to pr.yaml
• These jobs are marked as optional (via ignored_pr_jobs) so they don't block PR merges
• PyTorch tests require CUDA 12.9+ for conda and CUDA 12.8+ for wheels (based on upstream package availability)

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[jameslamb]

Left a few suggestions for your consideration, enough that I'm leaving a "comment" review and not an "approve one". @ me when you're ready for another review.

[jameslamb]

Approving so you don't need to chase me for a review, do what you want with the remaining comments.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Tests

  • Added optional integration pipelines for conda- and wheel-based Python environments.
  • Introduced PyTorch and CuPy integration test runs, conditionally executed by CUDA version, with aggregated exit reporting.
  • New optional pipelines are included in PR flow but ignored by default telemetry checks.

• Chores

  • Expanded test groups and dependency matrices to cover PyTorch and CuPy variants, including wheel-aware configurations.

Walkthrough

Adds CI integration testing for PyTorch and CuPy: two optional PR workflow jobs (conda and wheel variants), two integration test scripts (conda- and wheel-based), new dependency groups for PyTorch/CuPy, and updates to PR checks to ignore the optional jobs.

Changes

Cohort / File(s)	Summary
PR workflow \.github/workflows/pr.yaml	Adds conda-python-tests-integration-optional and wheel-tests-integration-optional jobs, wires them into pr-builder needs and PR flow, and conditions them on changed-files outputs.
Checks configuration checks.yaml	Adds the two new optional job names to ignored_pr_jobs so they are excluded from PR status checks.
Integration test scripts ci/test_python_integrations.sh, ci/test_wheel_integrations.sh	Adds two bash scripts: test_python_integrations.sh (conda envs, runs PyTorch [CUDA ≥12.9] and CuPy tests with coverage) and test_wheel_integrations.sh (downloads wheels, installs constraints, runs PyTorch [CUDA ≥12.8] and CuPy tests). Both aggregate exit codes.
Dependency metadata dependencies.yaml	Adds test groups test_pytorch, test_cupy, test_wheels_pytorch, test_wheels_cupy and new public dependency entries depends_on_pytorch and depends_on_cupy with conda/common and requirements/pyproject matrices for CUDA-aware variants.
Changed-files exclusions \.github/workflows/pr.yaml (changed-files group)	Adds exclusions for integration scripts: ci/test_wheel_integrations.sh and ci/test_python_integrations.sh in relevant changed-files conditions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding CI jobs for PyTorch and CuPy integration tests, which aligns with the primary focus of the changeset.
Description check	✅ Passed	The description clearly relates to the changeset, explaining the purpose of adding integration tests for PyTorch and CuPy, and lists the specific files and changes included.
Linked Issues check	✅ Passed	The PR comprehensively addresses all requirements from issue #2239: adds test_pytorch and test_cupy dependency specifications with CUDA-aware package matrices, includes integration test scripts (ci/test_python_integrations.sh and ci/test_wheel_integrations.sh), and adds optional CI jobs that run integration tests while keeping base tests lightweight.
Out of Scope Changes check	✅ Passed	All changes are directly related to adding CI coverage for PyTorch and CuPy integration tests as specified in issue #2239; no unrelated modifications or scope creep detected.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

No actionable comments were generated in the recent review. 🎉

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ci/test_wheel_integrations.sh`:
- Around line 31-35: Add a GPU availability check before running GPU-dependent
tests: run a quick nvidia-smi call right after obtaining RAPIDS_CUDA_VERSION
(before pytest invocation) and fail fast if it errors; update the script to
print a clear error message and set/exit with a non-zero EXITCODE (use the
existing EXITCODE variable) when nvidia-smi is not available or returns a
non-zero status so tests don’t proceed on machines without a usable GPU.

🧹 Nitpick comments (1)

.github/workflows/pr.yaml (1)

205-212: Missing sccache-dist-token-secret-name — inconsistent with the other test jobs.

The existing conda-python-tests job (line 204) and the new wheel-tests-integration-optional job (line 264) both pass sccache-dist-token-secret-name, but this new conda integration job does not. If this was intentionally removed per the "remove unused sccache" commit, consider also removing it from wheel-tests-integration-optional for consistency — otherwise, add it here to match the existing pattern.

Option A: Add sccache token to match other jobs

   conda-python-tests-integration-optional:
     needs: [conda-python-build, changed-files]
     secrets: inherit
     uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
     if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_conda
     with:
       build_type: pull-request
       script: ci/test_python_integrations.sh
+      sccache-dist-token-secret-name: GIST_REPO_READ_ORG_GITHUB_TOKEN

[bdice]

/merge