refactor(cudf_polars): use cudf_streaming for cudf-specific streaming imports

[vyasr]

This PR updates the cudf_polars execution engine to import cudf-specific streaming primitives from the new cudf_streaming package instead of from rapidsmpf directly.

This is part of the ongoing work to make the dependency between cudf and rapidsmpf unidirectional by moving cudf-specific streaming code into the cudf repository.

Dependencies:

• Add Python/Cython bindings for libcudf_streaming #22760 (cudf_streaming Python package)

Changes:

• Update cudf_polars source to import from cudf_streaming instead of rapidsmpf for cudf-specific streaming types
• Update cudf_polars tests accordingly

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[vyasr]

/ok to test

[vyasr]

/ok to test

[vyasr]

/ok to test

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Updates move streaming types and partition helpers from rapidsmpf to the new cudf_streaming package, add cudf-streaming to packaging and dependency matrices, and change CI/workflow scripts to build, download, and install libcudf_streaming and cudf_streaming wheels for tests.

Changes

Streaming dependency migration

Layer / File(s)	Summary
Dependency declarations and package configuration python/cudf_polars/pyproject.toml, conda/recipes/cudf-polars/recipe.yaml, dependencies.yaml	Adds cudf-streaming to project and recipe requirements, updates isort/ruff rapids section, and expands dependencies.yaml with CUDA-specific cudf-streaming selectors and fallback.
CI workflow and wheel build integration .github/workflows/pr.yaml, ci/test_cudf_polars_polars_tests.sh, ci/test_wheel_cudf_polars.sh	Adds wheel-build-cudf-streaming to test job needs; CI scripts download libcudf_streaming and cudf_streaming wheels and install them during per-version test venv setup.
Core engine imports and SPMD integration python/cudf_polars/cudf_polars/engine/core.py, dask.py, ray.py, spmd.py, utils/config.py	Redirects runtime and TYPE_CHECKING imports (ChannelMetadata, TableChunk, unpack_and_concat) to cudf_streaming.* and reorders some imports; SPMD AllGather now uses cudf_streaming.integrations.partition.
Streaming actor graph collectives python/cudf_polars/cudf_polars/streaming/actor_graph/collectives/allgather.py, shuffle.py, sort.py	Collective operators now import partitioning helpers and streaming metadata/table-chunk types from cudf_streaming and update a shuffle docstring reference.
Actor graph core, utils, and nodes python/cudf_polars/cudf_polars/streaming/actor_graph/core.py, utils.py, io.py, join.py, groupby.py, nodes.py, over.py, repartition.py, union.py	Core actor graph modules and node implementations import ChannelMetadata, HashScheme, Partitioning, TableChunk, and related helpers from cudf_streaming.streaming and adjust in-function parquet helper imports.
Streaming tests python/cudf_polars/tests/streaming/*	Tests updated to import streaming types from cudf_streaming; several tests only have import-order or whitespace adjustments.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• rapidsai/cudf#22760: Introduces libcudf_streaming / cudf_streaming bindings and related build/publish workflows used by this PR.
• rapidsai/cudf#22655: Dependency migration away from rapidsmpf that aligns with the import and packaging changes here.

Suggested reviewers

• jameslamb
• mroeschke
• Matt711
• bdice
• TomAugspurger

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly summarizes the main change: migrating from rapidsmpf to cudf_streaming for cudf-specific streaming imports throughout the codebase.
Description check	✅ Passed	The PR description accurately explains that the change updates cudf_polars to import from cudf_streaming instead of rapidsmpf, with context about unidirectional dependency management and lists specific changes made.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Matt711]

From copilot re: devcontainer failures FWIW:

Solution for Failing Job

The failure is a CMake configuration error related to imported target promotion to global scope. The error occurs when building libcudf_kafka after the main cudf build completes.

Root Cause

CMake Error at /home/coder/cudf/cpp/build/pip/cuda-13.2/latest/cudf-config.cmake:129 (set_target_properties): Attempt to promote imported target "cudf::cudftestutil" to global scope (by setting IMPORTED_GLOBAL) which is not built in this directory.

When libcudf_kafka tries to import cudf's exported config file, it attempts to set IMPORTED_GLOBAL on imported targets that were created in a different directory context. CMake prohibits this for targets not built in the current directory.

Solution

The fix is in the cudf-config.cmake export file. You need to remove or conditionally skip the IMPORTED_GLOBAL property when exporting test utility targets, since these are typically not needed by external consumers.

File to modify: cpp/src/cudf-config.cmake.in (or the generated cudf-config.cmake)

Change needed:

# Around line 129, modify the set_target_properties call to exclude IMPORTED_GLOBAL
# for targets that aren't strictly necessary for external consumers

# BEFORE (problematic):
set_target_properties(
  cudf::cudftestutil
  cudf::cudftestutil_impl
  PROPERTIES IMPORTED_GLOBAL TRUE
)

# AFTER (fixed):
# Only mark as IMPORTED_GLOBAL if it's being used in the same build context
if(NOT CMAKE_FIND_PACKAGE_NAME STREQUAL "cudf" OR CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_BINARY_DIR)
  set_target_properties(
    cudf::cudftestutil
    cudf::cudftestutil_impl
    PROPERTIES IMPORTED_GLOBAL TRUE
  )
endif()

[vyasr]

@Matt711 your agent is wrong. The issue isn't the kafka build, it's the new streaming build. The issue is from cuCascade, it's why we need NVIDIA/cuCascade#141. That PR is now merged, but since rapidsai/devcontainers#715 is also merged the dependency relationship has already been flipped in devcontainers, so we need the PR removing all cudf bits from rapidsmpf merged first now. I'm working through that now in rapidsai/rapidsmpf#1083, there's been some back and forth on the exact scope of that work today so we're trying to sort through it now.

[vyasr]

/merge