Create new libcudf_streaming C++ library by vyasr · Pull Request #22747 · NVIDIA/cudf

vyasr · 2026-06-02T13:30:45Z

Description

This PR ports over the libcudf-specific components of librapidsmpf into a new C++ library inside this repository that depends on librapidsmpf.

Contributes to #22528

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2026-06-02T13:30:48Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

vyasr · 2026-06-02T13:30:56Z

/ok to test

vyasr · 2026-06-02T13:36:39Z

/ok to test

vyasr · 2026-06-02T15:09:50Z

/ok to test

copy-pr-bot · 2026-06-02T15:17:44Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

vyasr · 2026-06-02T15:18:20Z

/ok to test

copy-pr-bot · 2026-06-02T15:18:23Z

/ok to test

@vyasr, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

vyasr · 2026-06-02T15:22:16Z

/ok to test

madsbk

Looks good

madsbk · 2026-06-03T18:25:56Z

+  [[nodiscard]] void* data() noexcept;
+
+  /**
+   * @brief @return Cnst Pointer to the underlying storage.


Suggested change

* @brief @return Cnst Pointer to the underlying storage.

* @brief @return Const Pointer to the underlying storage.

pentschev · 2026-06-03T18:32:04Z

 if (( SUITEERROR == 0 )); then
    rapids-logger "Run libcudf examples"
-    timeout 30m ./ci/run_cudf_examples.sh
+    timeout 5m ./ci/run_cudf_examples.sh


Are the timeout reductions really intentional here? From my experience with CI, there's high variability and I'd suggest being wary.

Fair point. I think it's safe based on what I've seen, but happy to revisit after this PR. Reverted to 30m for now.

pentschev · 2026-06-03T18:39:15Z

+  NAME cudf_streaming_single_tests
+  COMMAND cudf_streaming_single_tests
+  GPUS 1
+  PERCENT 25


Why is this needed?

It's to allow for test parallelism (rapids_test_add uses PERCENT to determine how many tests can run concurrently based on GPU resource allocation). But for now I've set it to 100 to disable that, since we don't have enough tests to benefit from parallelism yet.

nirandaperera

I think we can (and should) remove single communicator from the tests.

nirandaperera · 2026-06-03T18:45:19Z

+#include <cstdint>
+#include <memory>
+
+namespace cudf_streaming::integrations {


I'm wondering if this integrations ns is required or not. Previously, cudf was one integration to rapidsmpf. But in cudf_streaming, I feel like this is not really necessary. 😇

I agree, but I want the initial set of PRs to be as close to a copy of the rapidsmpf code as possible because 1) we should move quickly since rapidsmpf development is fully paused until these PRs merged and 2) these PRs are huge and hard to review and I'd rather have meaningful changes done in follow-up PRs that are smaller in scope and reasonable to review.

Tracking in #22771

nirandaperera · 2026-06-03T18:48:04Z

+#include <rapidsmpf/communicator/single.hpp>
+#include <rapidsmpf/progress_thread.hpp>


So, cudf_streaming tests depend on rapidsmpf?

nirandaperera · 2026-06-03T18:49:14Z

@@ -0,0 +1,45 @@
+/**


I doubt we would need a custom test Environment for these tests. We might not even need rapidsmpf or the Single Communicator. Because these tests might not be passing messages around. (Do they?)

I'm not sure, you would know better than me. But as above, I'd prefer for this PR to be a faithful upstreaming of rapidsmpf code and to defer this type of improvement to a follow-up PR.

Tracking in #22771

- Fix typo in bloom_filter.hpp doxygen (Cnst -> Const) - Revert timeout reductions in test_cpp.sh back to 30m - Set PERCENT 100 for cudf_streaming tests (disable parallelism for now) - Add note to libcudf_streaming README that most users need not install directly

vyasr · 2026-06-03T20:25:57Z

/ok to test

pentschev

LGTM, thanks Vyas!

bdice

Reviewed packaging / CMake changes again. Minor comments/questions, please address as you see fit. Thanks!

bdice · 2026-06-03T21:01:16Z


+if (( SUITEERROR == 0 )); then
+    rapids-logger "Run libcudf_streaming gtests"
+    timeout 30m ./ci/run_cudf_streaming_ctests.sh -j20


Should we reduce the timeout, then?

Rename conda package from libcudf_streaming to libcudf-streaming to follow the convention used by cudf-polars and dask-cudf. Also remove the now-unnecessary entry from the version update script.

I addressed these reviews and Niranda gave the OK offline.

nirandaperera

LGTM

vyasr · 2026-06-04T16:01:41Z

/merge

This PR ports over the Python bindings for the parts of librapidsmpf as the new `libcudf_streaming` library in #22747 Note that, just like in the preexisting rapidsmpf testing setup, only the conda environment will run the MPI/UCXX tests. Contributes to #22528 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Matthew Roeschke (https://github.com/mroeschke) URL: #22760

The cudf_streaming tests are lightweight and complete quickly. Reduce the timeout from 30m to 5m which still provides ample headroom while avoiding masking long hangs. References: - NVIDIA#22747 (comment) - NVIDIA#22747 (comment) Signed-off-by: Vyas Ramasubramani <vyasr@nvidia.com>

Reduce the `timeout` for `cudf_streaming` C++ tests from 30m to 5m in `ci/test_cpp.sh`. The cudf_streaming tests are lightweight and complete quickly. The 30m timeout was kept during the initial PR (#22747) out of caution after reviewer feedback (#22747 (comment), #22747 (comment)), but 5m provides ample headroom while catching hangs much sooner. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #22880

Add `cudf_streaming` to the 'other libcudf includes' regex in `.clang-format` so that `cudf_streaming/` headers are sorted into the correct include category (priority 5) instead of falling through to the generic system includes bucket (priority 9). Follow-up from #22747 (comment)

vyasr self-assigned this Jun 2, 2026

vyasr added feature request New feature or request non-breaking Non-breaking change labels Jun 2, 2026

github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue labels Jun 2, 2026

github-project-automation Bot added this to cuDF Python Jun 2, 2026

GPUtester moved this to In Progress in cuDF Python Jun 2, 2026

github-actions Bot added the cudf-polars Issues specific to cudf-polars label Jun 2, 2026

vyasr force-pushed the branch-cudf-streaming-cpp branch from f47f5fa to f11245f Compare June 2, 2026 15:17

vyasr added 10 commits June 2, 2026 15:19

feat(libcudf_streaming): add CMake build system

b408816

feat(libcudf_streaming): add C++ headers

9f39615

feat(libcudf_streaming): add C++ source files

de9333d

feat(libcudf_streaming): add C++ tests

7a5dc7a

feat(libcudf_streaming): add packaging

a84dada

feat(libcudf_kafka): use shared get_cudf.cmake

71b73fc

feat(libcudf_streaming): add CODEOWNERS for C++ code

d4474e7

Fix style

ef1253d

chore(deps): remove unused libcudf_streaming dependency entry

1281ea1

Add librapidsmpf to libcudf cache host deps

5b4e206

vyasr force-pushed the branch-cudf-streaming-cpp branch from f11245f to 5b4e206 Compare June 2, 2026 15:20

Fetch cuco

0c357f9

PointKernel mentioned this pull request Jun 3, 2026

[FEA] Rename PascalCase types introduced by the rapidsmpf migration to snake_case #22769

Open

madsbk approved these changes Jun 3, 2026

View reviewed changes

pentschev approved these changes Jun 3, 2026

View reviewed changes

nirandaperera previously requested changes Jun 3, 2026

View reviewed changes

vyasr mentioned this pull request Jun 3, 2026

Follow-ups for rapidsmpf to cudf migration #22771

Open

10 tasks

pentschev approved these changes Jun 3, 2026

View reviewed changes

bdice approved these changes Jun 3, 2026

View reviewed changes

vyasr added 2 commits June 3, 2026 21:28

Use hyphenated conda package name for libcudf-streaming

d34574f

Rename conda package from libcudf_streaming to libcudf-streaming to follow the convention used by cudf-polars and dask-cudf. Also remove the now-unnecessary entry from the version update script.

Merge branch 'main' into branch-cudf-streaming-cpp

15d6c1d

nirandaperera approved these changes Jun 3, 2026

View reviewed changes

wence- mentioned this pull request Jun 4, 2026

Use std::span for cudf::split/cudf::concatenate calls rapidsai/rapidsmpf#1077

Open

rapids-bot Bot merged commit fbe9799 into NVIDIA:main Jun 4, 2026
122 checks passed

github-project-automation Bot moved this from In Progress to Done in cuDF Python Jun 4, 2026

vyasr deleted the branch-cudf-streaming-cpp branch June 4, 2026 16:03

vyasr mentioned this pull request Jun 4, 2026

Add Python/Cython bindings for libcudf_streaming #22760

Merged

3 tasks

This was referenced Jun 9, 2026

Add remaining cudf_streaming tests and benchmarks #22814

Merged

Enforce east const via clang-format QualifierAlignment #22834

Merged

Build libcudf_streaming comm benchmarks with partial MPI/UCXX availability #22860

Merged

vyasr mentioned this pull request Jun 13, 2026

ci: reduce timeout for cudf_streaming C++ tests to 5m #22880

Merged

3 tasks

vyasr mentioned this pull request Jun 13, 2026

Add cudf_streaming to clang-format IncludeCategories #22881

Merged

coderabbitai Bot mentioned this pull request Jun 16, 2026

refactor(streaming): flatten namespaces and rename to snake_case #22909

Merged

coderabbitai Bot mentioned this pull request Jul 6, 2026

Rapidsmpf backref API changes for Host and Pinned MR #23128

Merged

3 tasks

	* @brief @return Cnst Pointer to the underlying storage.
	* @brief @return Const Pointer to the underlying storage.

		#include <rapidsmpf/communicator/single.hpp>
		#include <rapidsmpf/progress_thread.hpp>

Uh oh!

Conversation

vyasr commented Jun 2, 2026

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 2, 2026

Uh oh!

vyasr commented Jun 2, 2026

Uh oh!

vyasr commented Jun 2, 2026

Uh oh!

vyasr commented Jun 2, 2026

Uh oh!

copy-pr-bot Bot commented Jun 2, 2026

Uh oh!

vyasr commented Jun 2, 2026

Uh oh!

copy-pr-bot Bot commented Jun 2, 2026

Uh oh!

vyasr commented Jun 2, 2026

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirandaperera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vyasr commented Jun 3, 2026

Uh oh!

pentschev left a comment

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nirandaperera left a comment

Choose a reason for hiding this comment

Uh oh!

vyasr commented Jun 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone