Make RapidsMPF the default runtime for cudf_polars streaming executor by mroeschke · Pull Request #22281 · NVIDIA/cudf

mroeschke · 2026-04-23T22:24:17Z

Description

closes #21466
closes #21767

Waiting for #22212

Makes rapidsmpf a required dependency of cudf_polars
Removes the following StreamingExecutor options as they were "experimental" with associated code paths
- StreamingExecutor.runtime
- StreamingExecutor.shuffle_method
- StreamingExecutor.unique_fraction
- StreamingExecutor.groupby_n_ary
- StreamingExecutor.rapidsmpf_spill
Removes the task runtime and associated tests
Some tests we modified to only test 1 specific test configuration because of [FEA] Change cudf_polars testing assertion function to require an engine argument. #22346 to pass these tests for now. Planning on revisiting this once rapidsmpf becomes the default

Ops-Bot-Merge-Barrier: true

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

…df_polars

…idsmpf_default

copy-pr-bot · 2026-04-23T22:24:20Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

…idsmpf_default

mroeschke · 2026-04-28T00:40:09Z

/ok to test 4a37ee2

…idsmpf_default

mroeschke · 2026-04-28T19:09:25Z

/ok to test 3e4dfe0

…idsmpf_default

mroeschke · 2026-04-28T21:34:42Z

/ok to test 49849d0

…idsmpf_default

mroeschke · 2026-04-28T22:18:36Z

/ok to test bc92082

… error

Matt711

Exciting!

bdice

Devcontainers look better now. I approve as long as CI passes.

wence- · 2026-05-06T09:41:12Z

-        )
-        for i, key in enumerate(partition_info[ir].keys(ir))
-    }
+    return evaluate_rapidsmpf(ir, config_options)


In a followup we should merge evaluate_rapidsmpf and evaluate_streaming...

wence- · 2026-05-06T09:47:02Z

+    # Target about 12.5% of the device memory when
+    # default_target_partition_size is used to set the
+    # target partition size (i.e. 5x the 2.5% default).
+    return min(5, int(max(1, (device_size * 0.125) // 1e9)))


TODO: I think we want to move towards making these limits all byte-based, rather than partition count.

Yes, I agree. I proposed this verbally on Monday, and had a feeling you'd agree. It seems unnecessarily complicated tie target_partition_size and broadcast_join_limit together in the way we currently do. After this PR goes in, we may want to consider replacing broadcast_join_limit with a byte-based broadcast_limit before 26.02.

wence- · 2026-05-06T09:47:47Z

    client_device_threshold
-        Threshold for spilling data from device memory in rapidsmpf.
+        Threshold for spilling data from device memory.


nit: Why is client in this name?

I think it's only used for the "single" cluster case and can be removed when we remove that code path (in favor of "spmd").

…idsmpf_default

…aude)

…idsmpf_default

bdice

Approving to unblock, some final questions.

bdice · 2026-05-06T22:47:18Z

        sccache --zero-stats;
-        build-all -j0 -DBUILD_BENCHMARKS=ON --verbose 2>&1 | tee telemetry-artifacts/build.log;
+        clone-rapidsmpf -j$(nproc) -v -q --branch "$(cat ~/cudf/RAPIDS_BRANCH)" --clone-upstream --depth 1 --single-branch --shallow-submodules;
+        if [ "$PYTHON_PACKAGE_MANAGER" = "pip" ]; then rapids-make-pip-env --force; fi;


I feel like you would need the same thing in conda, since rapidsmpf is being cloned late. Let’s check on the dependency list and make sure it doesn’t have rapidsmpf or cudf before merging.

Yeah, fair point. In fe5502b, the final, recreated conda environment creation doesn't include rapidsmpf

bdice · 2026-05-06T22:48:00Z

-        build-all -j0 -DBUILD_BENCHMARKS=ON --verbose 2>&1 | tee telemetry-artifacts/build.log;
+        clone-rapidsmpf -j$(nproc) -v -q --branch "$(cat ~/cudf/RAPIDS_BRANCH)" --clone-upstream --depth 1 --single-branch --shallow-submodules;
+        if [ "$PYTHON_PACKAGE_MANAGER" = "pip" ]; then rapids-make-pip-env --force; fi;
+        build-all -j0 -DBUILD_BENCHMARKS=OFF -DBUILD_NUMA_SUPPORT=OFF -DBUILD_TESTS=OFF -DCUDF_BUILD_TESTUTIL=OFF --verbose 2>&1 | tee telemetry-artifacts/build.log;


We might need to call rapids-generate-scripts. I am not sure if build-all will do everything unless the scripts were regenerated.

Sure I added it, but in fe5502b I can't tell if it materially made a difference

…idsmpf_default

mroeschke · 2026-05-07T01:55:23Z

/merge

…NVIDIA#22281) closes NVIDIA#21466 closes NVIDIA#21767 Waiting for NVIDIA#22212 * Makes rapidsmpf a required dependency of cudf_polars * Removes the following `StreamingExecutor` options as they were "experimental" with associated code paths * `StreamingExecutor.runtime` * `StreamingExecutor.shuffle_method` * `StreamingExecutor.unique_fraction` * `StreamingExecutor.groupby_n_ary` * `StreamingExecutor.rapidsmpf_spill` * Removes the task runtime and associated tests * Some tests we modified to only test 1 specific test configuration because of NVIDIA#22346 to pass these tests for now. Planning on revisiting this once rapidsmpf becomes the default Ops-Bot-Merge-Barrier: true Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Lawrence Mitchell (https://github.com/wence-) URL: NVIDIA#22281

…lars (#22417) * Address the follow up review #22281 (comment) > In a followup we should merge evaluate_rapidsmpf and evaluate_streaming... * Removes a temporary iterable in `select` that I had from a leftover branch * Removes the `devcontainers` key in `dependencies.yml` that was leftover from troubleshooting the devcontainer CI job Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - James Lamb (https://github.com/jameslamb) - Peter Andreas Entschev (https://github.com/pentschev) URL: #22417

…NVIDIA#22281) closes NVIDIA#21466 closes NVIDIA#21767 Waiting for NVIDIA#22212 * Makes rapidsmpf a required dependency of cudf_polars * Removes the following `StreamingExecutor` options as they were "experimental" with associated code paths * `StreamingExecutor.runtime` * `StreamingExecutor.shuffle_method` * `StreamingExecutor.unique_fraction` * `StreamingExecutor.groupby_n_ary` * `StreamingExecutor.rapidsmpf_spill` * Removes the task runtime and associated tests * Some tests we modified to only test 1 specific test configuration because of NVIDIA#22346 to pass these tests for now. Planning on revisiting this once rapidsmpf becomes the default Ops-Bot-Merge-Barrier: true Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Lawrence Mitchell (https://github.com/wence-) URL: NVIDIA#22281

…lars (NVIDIA#22417) * Address the follow up review NVIDIA#22281 (comment) > In a followup we should merge evaluate_rapidsmpf and evaluate_streaming... * Removes a temporary iterable in `select` that I had from a leftover branch * Removes the `devcontainers` key in `dependencies.yml` that was leftover from troubleshooting the devcontainer CI job Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - James Lamb (https://github.com/jameslamb) - Peter Andreas Entschev (https://github.com/pentschev) URL: NVIDIA#22417

…y; add [dask] pip extra (#22480) * Follow up to making rapidsmpf a required dependency of cudf-polars, #22281, we no longer need a `cudf-polars[rapidsmpf]` pip extra * Removes `numpy` as a testing dependency as it was only used for `np.random` and `np.full` * Removes `"nvidia-ml-py>=12"` from the `experimental` extra as it's already a required dependency of cudf_polars * Adds `cudf-polars[dask]` as an alias for `cudf-polars[experimental]` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Gil Forsyth (https://github.com/gforsyth) URL: #22480

mroeschke added 6 commits April 23, 2026 18:34

Modify dependencies.yml to make rapidsmpf a required dependency of cu…

90653a9

…df_polars

Change Runtime default to None to indicate rapdsmpf default

23c5ed5

Deprecate the shuffle_method argument

4c0a86b

Deprecate unique_fraction, groupby_n_ary, rapidsmpf_spill

e6aeb4d

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

fa737fe

…idsmpf_default

Remove some asserts

4f90aad

mroeschke self-assigned this Apr 23, 2026

mroeschke added improvement Improvement / enhancement to an existing function breaking Breaking change labels Apr 23, 2026

github-actions Bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels Apr 23, 2026

github-project-automation Bot added this to cuDF Python Apr 23, 2026

GPUtester moved this to In Progress in cuDF Python Apr 23, 2026

mroeschke added 2 commits April 23, 2026 22:28

Make plugin.py diff nicer

71e1ca9

Remove rapidsmpf extra from ci/test_cudf_polars_polars_tests.sh

a5cba7a

madsbk mentioned this pull request Apr 24, 2026

Overhaul cudf-polars docs for new streaming multi-GPU engines #22252

Merged

mroeschke added 2 commits April 28, 2026 00:37

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

577314c

…idsmpf_default

Modify comment

4a37ee2

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

3e4dfe0

…idsmpf_default

mroeschke added 2 commits April 28, 2026 20:49

Try defining dependencies another way

7195f5f

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

49849d0

…idsmpf_default

mroeschke added 3 commits April 28, 2026 22:05

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

4831c3f

…idsmpf_default

Remove depends_on_rapidsmpf from test dependencies

a84a1d0

Use minor_version for rapidsmpf in conda recipe

bc92082

Try formatting python/cudf_polars/cudf_polars/utils/config.py for doc…

abe246f

… error

Try matching rapidsmpf base pip docker image

e062e80

Matt711 approved these changes May 6, 2026

View reviewed changes

bdice approved these changes May 6, 2026

View reviewed changes

wence- approved these changes May 6, 2026

View reviewed changes

mroeschke added 12 commits May 6, 2026 16:40

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

02d33f4

…idsmpf_default

Add -DBUILD_NUMA_SUPPORT=OFF in build_command

85af11b

Add DcuCascade_DIR to force CPM to build from source (according to Cl…

4d07468

…aude)

Add devcontainers section in dependencies excluding rapidsmpf

46b607e

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

5afc788

…idsmpf_default

Remove more runtime inclusions

02ba87d

Only include cudf, rapidsmpf in include_repos

12fa676

try reforcing the pip env generation after clone-rapidsmpf

833e5c7

don't build tests because of rapidsmpf

4d60df4

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

38e890e

…idsmpf_default

Add -DCUDF_BUILD_TESTUTIL=OFF

0646bb4

Don't build benchmarks either

7a90931

bdice approved these changes May 6, 2026

View reviewed changes

mroeschke added 2 commits May 7, 2026 00:10

Call rapids-generate-scripts, remake rapids-make-conda-env as well

2c5dc92

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rap…

fe5502b

…idsmpf_default

rapids-bot Bot merged commit 563021f into NVIDIA:main May 7, 2026
133 checks passed

github-project-automation Bot moved this from In Progress to Done in cuDF Python May 7, 2026

mroeschke deleted the enh/cudf_polars/rapidsmpf_default branch May 7, 2026 01:55

mroeschke mentioned this pull request May 7, 2026

Consolidate evaluate_rapidsmpf into evaluate_streaming in cudf_polars #22417

Merged

3 tasks

mroeschke mentioned this pull request May 12, 2026

Remove cudf-polars[rapidsmpf] pip extra & numpy as a [test] dependency; add [dask] pip extra #22480

Merged

3 tasks

Uh oh!

Conversation

mroeschke commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Apr 23, 2026

Uh oh!

mroeschke commented Apr 28, 2026

Uh oh!

mroeschke commented Apr 28, 2026

Uh oh!

mroeschke commented Apr 28, 2026

Uh oh!

mroeschke commented Apr 28, 2026

Uh oh!

Matt711 left a comment

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mroeschke commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mroeschke commented Apr 23, 2026 •

edited

Loading