Cudf-free shuffler tests by nirandaperera · Pull Request #1084 · rapidsai/rapidsmpf

nirandaperera · 2026-06-07T18:54:05Z

`[C++]` Make shuffler distributed tests cudf-free

Summary

The shuffler round-trip tests depended on cudf for input generation and result checking (random_table_with_index, partition_and_pack/unpack_and_concat, CUDF_TEST_EXPECT_TABLES_EQUIVALENT, cudf::test::BaseFixture), which made them heavy and coupled to the cudf integration. This replaces that with a self-contained, conservation-preserving data model built directly from PackedData, so the shuffler tests no longer depend on cudf and run as part of the default (non-BUILD_CUDF_TESTS) build.

Changes

C++ utilities

Add ceil_div and a chunk_indices view to utils/misc.hpp. chunk_indices tiles [0, count) into exactly num_chunks contiguous (possibly empty) ranges whose sizes sum to count.
ceil_div is computed as x / y + (x % y != 0) rather than the naive (x + y - 1) / y, which overflows for large unsigned values and is UB on signed overflow near the type maximum.
Add compile-time edge-case coverage for ceil_div in test_misc.cpp (static_asserts for exact/remainder division, zero numerator, denominator of one, and values at uint64/int64 maxima).

C++ shuffler tests (now cudf-free)

Rework the shuffler tests (test_shuffler.cpp, streaming/test_shuffler.cpp) around make_partition_data (produces routed chunks) and validate_partition_data (verifies received chunks), dropping all cudf/cudf_test usage and switching to rmm:: stream/MR helpers and ::testing::Test.
Generalize the test helpers generate_packed_data/validate_packed_data in tests/utils.hpp over the element type and build device buffers via the buffer resource + cuda_memcpy_async instead of cudf.
De-couple streaming/test_allgather.cpp from cudf_streaming/integrations/partition.hpp.
cpp/tests/CMakeLists.txt: move test_shuffler.cpp, streaming/test_shuffler.cpp, and streaming/test_allgather.cpp out of the BUILD_CUDF_TESTS block into the default build; the cudf block now only carries test_partition.cpp, test_shuffler_many_streams.cpp, and streaming/test_leaf_actor.cpp.

Python (now cudf-free)

Add a PackedData.from_device_buffer(gpu_data, metadata, stream, br) factory (plus .pyi stub) so packed data can be built from an rmm DeviceBuffer without cudf.
Add shared, cudf-free shuffle test helpers to rapidsmpf/testing.py: chunk_indices, generate_packed_data, validate_packed_data, make_partition_data, and validate_partition_data.
Rewrite the Python shuffler tests (tests/test_shuffler.py, tests/streaming/test_shuffler.py) around those helpers and the Shuffler/ShufflerAsync object interfaces, removing the cudf / cudf_streaming / pylibcudf dependencies.

Depends on #1087

copy-pr-bot · 2026-06-07T18:54:08Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

nirandaperera · 2026-06-07T18:58:45Z

/ok to test

Signed-off-by: niranda perera <niranda.perera@gmail.com>

pentschev · 2026-06-10T07:21:24Z

+    A ``PackedData`` containing the integer sequence.
+    """
+    data = np.arange(offset, offset + n_elements, dtype=_DTYPE).tobytes()
+    gpu_data = rmm.DeviceBuffer.to_device(data, stream=stream)


Please allocate this test payload through br.device_mr (or a small BR-backed helper) before wrapping it in PackedData. to_device() uses the current/default RMM resource and from_device_buffer() only adopts the allocation, so the replacement tests bypass BufferResource tracking that the old partition_and_pack(..., br=br) path covered.

Signed-off-by: niranda perera <niranda.perera@gmail.com>

…ist-tests

Signed-off-by: niranda perera <niranda.perera@gmail.com>

madsbk

Thanks @nirandaperera

pentschev · 2026-06-11T07:36:09Z

+/// Conservation-preserving data model shared by the shuffler round-trip tests.
+///
+/// The index range `[0, total_num_rows)` is split into `total_num_partitions^2`
+/// contiguous sub-regions via `chunk_indices` (front-loaded, so trailing sub-regions
+/// are empty when `N < P*P`). Sub-region `(local_pidx, split_idx)` is piece
+/// `k = local_pidx * P + split_idx` and is routed to destination partition
+/// `split_idx`. The pieces exactly tile `[0, N)`, so the total shuffled row
+/// count equals `N` regardless of rank or partition counts (conservation). A
+/// per-shuffle `base` offset is added to every value so distinct shuffles carry
+/// distinct data.


Is this supposed to be part of make_partition_data docs? Currently looks like a strange placing not really connected to anything.

I was trying to explain test approach here, especially the P^2 splits because I felt that a new reader might find it a little confusing.

Since this relates to make_partition_data, validate_partition_data, test_shuffler methods I thought this is the best place to put it, more as a detail rather than a doc string.

It's a strange placement, but I don't have a better idea where to put it either. Let's leave it there for now.

pentschev · 2026-06-11T09:41:19Z

 async def do_shuffle(
    context: Context,
    comm: Communicator,
-    ch_in: Channel[TableChunk],
-    ch_out: Channel[TableChunk],
+    ch_in: Channel[PartitionMapChunk],
+    ch_out: Channel[PartitionVectorChunk],


I can't add the comment on the exact lines, but assumes this extends to line 146:

This replacement still bypasses the exported streaming actor wrapper. The removed cudf-based test drove rapidsmpf.streaming.coll.shuffler.shuffler() in the channel pipeline (see that in 26.06 for example), which exercised streaming/coll/shuffler.pyx including the cpp_shuffler call and PartitionAssignment mapping. This new actor instantiates ShufflerAsync directly, so a regression in the public shuffler() wrapper would no longer be caught. Now that PartitionMapChunk.from_packed_data_map() exists, can this test use the actual shuffler() actor between ch_in and ch_shuffled?

Yes you are right. I was more focused on porting the previous tests as is, and missed this detail. I removed the do_shuffle actor alltogether, because we can strictly replace it from shuffle actor as you said.

pentschev · 2026-06-11T14:56:00Z

I have opened #1092 and #1093 adding cuDF-free coverage for the remaining missing tests. Both require changes from this PR (specifically, rapidsmpf.testing.generate_packed_data and rapidsmpf.testing.validate_packed_data). I think after this PR #1084 plus the aforementioned PRs are merged we would have covered everything missing and could then remove all remaining cuDF pieces without functionality coverage loss.

Signed-off-by: niranda perera <niranda.perera@gmail.com>

pentschev

Thanks @nirandaperera , one final minor request, otherwise LGTM.

Co-authored-by: Peter Andreas Entschev <peter@entschev.com>

nirandaperera · 2026-06-11T16:07:32Z

Thanks @pentschev

nirandaperera · 2026-06-11T16:18:18Z

/merge

Removes the remaining cuDF-specific dependency surface from RapidsMPF after the cuDF-dependent coverage was moved out and replaced with RapidsMPF-native tests. This removes: - `BUILD_CUDF_TESTS` and the `cudf_streaming` CMake fetch path - cuDF/cudf_streaming links from C++ tests, examples, and benchmarks - cuDF-only C++ tests, examples, benchmarks, and NDSH benchmark tooling - Python examples/tests/benchmark code that imports or requires cuDF/pylibcudf - cuDF-related package dependencies from `dependencies.yaml`, generated conda envs, pyproject metadata, and conda recipes - CI smoke-test references to deleted cuDF-specific binaries - documentation and comments that described RapidsMPF APIs in cuDF-specific terms The remaining shuffler, streaming, memory, communicator, and Ray coverage stays library-neutral and is covered by the tests added in the preceding PRs. Depends on #1084 and #1092 . Authors: - Peter Andreas Entschev (https://github.com/pentschev) - Niranda Perera (https://github.com/nirandaperera) Approvers: - Niranda Perera (https://github.com/nirandaperera) - Vyas Ramasubramani (https://github.com/vyasr) URL: #1094

nirandaperera added breaking Introduces a breaking change improvement Improves an existing functionality labels Jun 7, 2026

pentschev reviewed Jun 8, 2026

View reviewed changes

Comment thread cpp/include/rapidsmpf/memory/spill.hpp

Comment thread cpp/tests/utils.hpp

nirandaperera marked this pull request as ready for review June 8, 2026 17:37

nirandaperera requested review from a team as code owners June 8, 2026 17:37

nirandaperera mentioned this pull request Jun 8, 2026

Make distributed tests independent from cudf #1086

Closed

8 tasks

nirandaperera requested a review from pentschev June 8, 2026 18:33

nirandaperera force-pushed the cuda-free-dist-tests branch from 0ce351e to fb06ec7 Compare June 8, 2026 21:06

nirandaperera marked this pull request as draft June 8, 2026 21:06

rebased

8820818

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera force-pushed the cuda-free-dist-tests branch from fb06ec7 to 8820818 Compare June 9, 2026 17:58

nirandaperera marked this pull request as ready for review June 9, 2026 17:58

cudf-free streaming shuffler

85e3443

Signed-off-by: niranda perera <niranda.perera@gmail.com>

pentschev requested changes Jun 9, 2026

View reviewed changes

Comment thread cpp/include/rapidsmpf/utils/misc.hpp Outdated

Comment thread cpp/tests/test_shuffler.cpp

Comment thread cpp/tests/streaming/test_shuffler.cpp Outdated

Comment thread cpp/tests/streaming/test_shuffler.cpp Outdated

nirandaperera added 2 commits June 9, 2026 16:23

python tests

5f066f7

Signed-off-by: niranda perera <niranda.perera@gmail.com>

addressign comments

1aad864

Signed-off-by: niranda perera <niranda.perera@gmail.com>

madsbk reviewed Jun 10, 2026

View reviewed changes

Comment thread cpp/tests/test_shuffler.cpp

Comment thread cpp/tests/test_shuffler.cpp

Comment thread python/rapidsmpf/rapidsmpf/testing.py

pentschev requested changes Jun 10, 2026

View reviewed changes

nirandaperera added 3 commits June 10, 2026 14:33

add docs

bcee4f2

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Merge branch 'main' of github.com:rapidsai/rapidsmpf into cuda-free-d…

7f2e603

…ist-tests

addressing comments

3095342

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera requested review from madsbk and pentschev June 10, 2026 23:59

madsbk approved these changes Jun 11, 2026

View reviewed changes

pentschev requested changes Jun 11, 2026

View reviewed changes

pentschev mentioned this pull request Jun 11, 2026

Add generic core tests for cuDF-migrated coverage gaps #1092

Merged

pentschev mentioned this pull request Jun 11, 2026

Add generic streaming tests for cuDF-migrated coverage gaps #1093

Closed

nirandaperera added 2 commits June 11, 2026 08:17

using shuffler actor

394d9a4

Signed-off-by: niranda perera <niranda.perera@gmail.com>

relax cython type

328cf4c

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera requested a review from pentschev June 11, 2026 15:31

Merge branch 'main' into cuda-free-dist-tests

ea5c08b

pentschev approved these changes Jun 11, 2026

View reviewed changes

Comment thread python/rapidsmpf/rapidsmpf/tests/streaming/test_shuffler.py

Update python/rapidsmpf/rapidsmpf/tests/streaming/test_shuffler.py

15e8562

Co-authored-by: Peter Andreas Entschev <peter@entschev.com>

pentschev mentioned this pull request Jun 11, 2026

Remove cuDF-specific dependencies and examples from rapidsmpf #1094

Merged

rapids-bot Bot merged commit 3cad33f into rapidsai:main Jun 11, 2026
66 checks passed

Uh oh!

Conversation

nirandaperera commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

[C++] Make shuffler distributed tests cudf-free

Summary

Changes

C++ utilities

C++ shuffler tests (now cudf-free)

Python (now cudf-free)

Uh oh!

copy-pr-bot Bot commented Jun 7, 2026

Uh oh!

nirandaperera commented Jun 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pentschev commented Jun 11, 2026

Uh oh!

pentschev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nirandaperera commented Jun 11, 2026

Uh oh!

nirandaperera commented Jun 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nirandaperera commented Jun 7, 2026 •

edited

Loading

`[C++]` Make shuffler distributed tests cudf-free