Add Python/Cython bindings for libcudf_streaming - #22760
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test |
github-actions
Bot
added
libcudf
Contributor
Author
|
/ok to test |
1 similar comment
Contributor
Author
|
/ok to test |
vyasr
added
feature request
Contributor
Author
|
/ok to test |
vyasr
force-pushed
the
branch-cudf-streaming-python
branch
from
June 3, 2026 16:15
4cdd4c1 to
0c20b1a
Compare
Contributor
Author
|
/ok to test |
2 similar comments
Contributor
Author
|
/ok to test |
Contributor
Author
|
/ok to test |
3 tasks
vyasr
force-pushed
the
branch-cudf-streaming-python
branch
from
June 3, 2026 22:14
072fc83 to
e2f1058
Compare
Contributor
Author
|
/ok to test |
vyasr
changed the title
vyasr
force-pushed
the
branch-cudf-streaming-python
branch
from
June 4, 2026 16:09
e2f1058 to
33a20c5
Compare
vyasr
marked this pull request as ready for review
June 4, 2026 16:09
…ution - Add 'cudf_streaming' to build.sh VALIDARGS and build section so conda recipe's './build.sh cudf_streaming' invocation succeeds. - Switch ci/test_wheel_cudf_streaming.sh to use rapids-download-from-github with rapids-package-name --stable for abi3 artifact resolution, matching how cudf handles Python-version-independent wheel downloads.
Copy comm, stream, and supporting fixtures directly into our conftest instead of importing rapidsmpf.tests.conftest. The rapidsmpf test module is not installed in wheel environments and the try/import/fallback logic added unnecessary complexity. The fixtures are simple and self-contained.
…reaming Use 'libcudf-streaming' instead of 'libcudf_streaming' in conda package specs in dependencies.yaml for precision. The dependency file generator propagates the fix to pyproject.toml as well.
- build.yaml: use min_by for stable-ABI wheel builds (match pr.yaml) - parquet.pyi: fix Filter.__init__ parameter name (expression -> filter) - partition.pyx: fix docstring reference to partition_and_pack - channel_metadata.pxd: use except +ex_handler consistently
- Remove unnecessary pytest.importorskip for cudf_streaming.streaming (always available when running these tests) - Add numpy to test dependencies since tests use it directly
vyasr
force-pushed
the
branch-cudf-streaming-python
branch
from
June 4, 2026 19:12
e0deb07 to
80d31c8
Compare
bdice
reviewed
Jun 4, 2026
bdice
approved these changes
Jun 4, 2026
bdice
left a comment
bdice
left a comment
Contributor
There was a problem hiding this comment.
Approving with one comment.
Matt711
approved these changes
Jun 4, 2026
Matt711
left a comment
Matt711
left a comment
Member
There was a problem hiding this comment.
Minor comments / question. Migration looks good.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci/build_wheel_cudf_streaming.sh`:
- Line 18: The echo line uses an unquoted ${LIBCUDF_STREAMING_WHEELHOUSE} which
can be word-split; update the echo to quote the variable expansion so
spaces/special chars are preserved, e.g. use "${LIBCUDF_STREAMING_WHEELHOUSE}"
inside the double-quoted string in the existing echo (reference the echo command
that writes "libcudf-streaming-${RAPIDS_PY_CUDA_SUFFIX} @ file://...") so the
path is not split while leaving the wildcard portion intact.
In `@python/cudf_streaming/cudf_streaming/tests/conftest.py`:
- Around line 55-71: The comm fixture currently skips tests if either transport
is missing; change the availability check to only skip when the requested
transport (request.param) is not present in COMMUNICATORS. In the comm function,
replace the global checks with a conditional that inspects
comm_name/request.param and calls pytest.skip only if that specific transport
(e.g., "mpi" or "ucxx") is not in COMMUNICATORS; keep the MPI barrier/yield
pattern and the yield of request.getfixturevalue(f"_{comm_name}_comm")
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 128be7c7-22cb-4c51-85f7-cd479905195f
📒 Files selected for processing (46)
.github/CODEOWNERS.github/workflows/build.yaml.github/workflows/pr.yamlbuild.shci/build_python.shci/build_wheel_cudf_streaming.shci/run_cudf_streaming_pytests.shci/test_python_other.shci/test_wheel_cudf_streaming.shci/validate_wheel.shconda/recipes/cudf_streaming/conda_build_config.yamlconda/recipes/cudf_streaming/recipe.yamldependencies.yamlpython/cudf_streaming/CMakeLists.txtpython/cudf_streaming/README.mdpython/cudf_streaming/cudf_streaming/VERSIONpython/cudf_streaming/cudf_streaming/__init__.pypython/cudf_streaming/cudf_streaming/integrations/CMakeLists.txtpython/cudf_streaming/cudf_streaming/integrations/__init__.pxdpython/cudf_streaming/cudf_streaming/integrations/__init__.pypython/cudf_streaming/cudf_streaming/integrations/partition.pxdpython/cudf_streaming/cudf_streaming/integrations/partition.pyipython/cudf_streaming/cudf_streaming/integrations/partition.pyxpython/cudf_streaming/cudf_streaming/streaming/CMakeLists.txtpython/cudf_streaming/cudf_streaming/streaming/__init__.pxdpython/cudf_streaming/cudf_streaming/streaming/__init__.pypython/cudf_streaming/cudf_streaming/streaming/bloom_filter.pxdpython/cudf_streaming/cudf_streaming/streaming/bloom_filter.pyipython/cudf_streaming/cudf_streaming/streaming/bloom_filter.pyxpython/cudf_streaming/cudf_streaming/streaming/channel_metadata.pxdpython/cudf_streaming/cudf_streaming/streaming/channel_metadata.pyipython/cudf_streaming/cudf_streaming/streaming/channel_metadata.pyxpython/cudf_streaming/cudf_streaming/streaming/parquet.pyipython/cudf_streaming/cudf_streaming/streaming/parquet.pyxpython/cudf_streaming/cudf_streaming/streaming/partition.pyipython/cudf_streaming/cudf_streaming/streaming/partition.pyxpython/cudf_streaming/cudf_streaming/streaming/table_chunk.pxdpython/cudf_streaming/cudf_streaming/streaming/table_chunk.pyipython/cudf_streaming/cudf_streaming/streaming/table_chunk.pyxpython/cudf_streaming/cudf_streaming/tests/conftest.pypython/cudf_streaming/cudf_streaming/tests/test_bloom_filter.pypython/cudf_streaming/cudf_streaming/tests/test_channel_metadata.pypython/cudf_streaming/cudf_streaming/tests/test_partition.pypython/cudf_streaming/cudf_streaming/tests/test_read_parquet.pypython/cudf_streaming/cudf_streaming/tests/test_table_chunk.pypython/cudf_streaming/pyproject.toml
✅ Files skipped from review due to trivial changes (6)
- .github/CODEOWNERS
- python/cudf_streaming/cudf_streaming/integrations/init.pxd
- python/cudf_streaming/cudf_streaming/VERSION
- python/cudf_streaming/README.md
- python/cudf_streaming/cudf_streaming/streaming/init.pxd
- python/cudf_streaming/cudf_streaming/integrations/partition.pyi
🚧 Files skipped from review as they are similar to previous changes (32)
- conda/recipes/cudf_streaming/conda_build_config.yaml
- python/cudf_streaming/cudf_streaming/init.py
- ci/validate_wheel.sh
- .github/workflows/build.yaml
- python/cudf_streaming/cudf_streaming/streaming/init.py
- python/cudf_streaming/cudf_streaming/streaming/bloom_filter.pxd
- .github/workflows/pr.yaml
- python/cudf_streaming/cudf_streaming/streaming/partition.pyx
- build.sh
- python/cudf_streaming/cudf_streaming/integrations/partition.pxd
- python/cudf_streaming/cudf_streaming/integrations/init.py
- conda/recipes/cudf_streaming/recipe.yaml
- ci/run_cudf_streaming_pytests.sh
- python/cudf_streaming/cudf_streaming/streaming/bloom_filter.pyi
- dependencies.yaml
- python/cudf_streaming/cudf_streaming/streaming/partition.pyi
- python/cudf_streaming/cudf_streaming/streaming/table_chunk.pxd
- python/cudf_streaming/cudf_streaming/streaming/channel_metadata.pxd
- ci/test_python_other.sh
- python/cudf_streaming/cudf_streaming/streaming/channel_metadata.pyi
- python/cudf_streaming/cudf_streaming/tests/test_partition.py
- python/cudf_streaming/cudf_streaming/streaming/parquet.pyi
- python/cudf_streaming/cudf_streaming/streaming/parquet.pyx
- python/cudf_streaming/pyproject.toml
- python/cudf_streaming/cudf_streaming/streaming/table_chunk.pyi
- python/cudf_streaming/cudf_streaming/streaming/bloom_filter.pyx
- python/cudf_streaming/cudf_streaming/tests/test_bloom_filter.py
- python/cudf_streaming/cudf_streaming/tests/test_channel_metadata.py
- python/cudf_streaming/cudf_streaming/streaming/channel_metadata.pyx
- python/cudf_streaming/cudf_streaming/tests/test_read_parquet.py
- python/cudf_streaming/cudf_streaming/streaming/table_chunk.pyx
- python/cudf_streaming/cudf_streaming/tests/test_table_chunk.py
mroeschke
approved these changes
Jun 4, 2026
Contributor
Author
|
/merge |
This was referenced Jun 8, 2026
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jun 9, 2026
… imports (#22786) This PR updates the cudf_polars execution engine to import cudf-specific streaming primitives from the new `cudf_streaming` package instead of from `rapidsmpf` directly. This is part of the ongoing work to make the dependency between cudf and rapidsmpf unidirectional by moving cudf-specific streaming code into the cudf repository. **Dependencies:** - #22760 (cudf_streaming Python package) **Changes:** - Update cudf_polars source to import from `cudf_streaming` instead of `rapidsmpf` for cudf-specific streaming types - Update cudf_polars tests accordingly Authors: - Vyas Ramasubramani (https://github.com/vyasr) - Peter Andreas Entschev (https://github.com/pentschev) - Bradley Dice (https://github.com/bdice) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Mads R. B. Kristensen (https://github.com/madsbk) - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Peter Andreas Entschev (https://github.com/pentschev) - Niranda Perera (https://github.com/nirandaperera) - Tom Augspurger (https://github.com/TomAugspurger) URL: #22786
This was referenced Jun 9, 2026
This was referenced Jun 16, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR ports over the Python bindings for the parts of librapidsmpf as the new
libcudf_streaminglibrary in #22747Note that, just like in the preexisting rapidsmpf testing setup, only the conda environment will run the MPI/UCXX tests.
Contributes to #22528
Checklist