Make RapidsMPF the default runtime for cudf_polars streaming executor - #22281

Merged

rapids-bot[bot] merged 86 commits into

NVIDIA:mainfrom

mroeschke:enh/cudf_polars/rapidsmpf_default

May 7, 2026

Commits on Apr 23, 2026

Modify dependencies.yml to make rapidsmpf a required dependency of cudf_polars
mroeschke
committed
90653a9
Change Runtime default to None to indicate rapdsmpf default
mroeschke
committed
23c5ed5
Deprecate the shuffle_method argument
mroeschke
committed
4c0a86b
Deprecate unique_fraction, groupby_n_ary, rapidsmpf_spill
mroeschke
committed
e6aeb4d
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
fa737fe
Remove some asserts
mroeschke
committed
4f90aad
Make plugin.py diff nicer
mroeschke
committed
71e1ca9
Remove rapidsmpf extra from ci/test_cudf_polars_polars_tests.sh
mroeschke
committed
a5cba7a

Commits on Apr 28, 2026

Commits on May 1, 2026

Fix some engine configurations for now
mroeschke
committed
264f641
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
feb2474
Add tests for FutureWarnings
mroeschke
committed
d3a2e79
Add no covers to no longer tested
mroeschke
committed
dcbe494
Fix(?) test_trace_basic
mroeschke
committed
8acb2c9
Correct scoping of FutureWarning
mroeschke
committed
2f5436f
Ignore FutureWarning when reconstructing Dask engine opts internally, fix test_dask
mroeschke
committed
1e19d2c
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
125de67
Fix test_scan_properties with rapidsmpf testing
mroeschke
committed
03ed9a5
Avoid implicit pyarrow dependency of polars in allgather_polars_dataframe
mroeschke
committed
29dba7e
Adjust tests/experimental/test_unique.py
mroeschke
committed
d390d51
Fix test_executor_options_unique_fraction
mroeschke
committed
c550b06
Fix test_groupby_agg_config_options
mroeschke
committed
0bb226f
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
e0e635a
Add pragmas for uncovered lines
mroeschke
committed
dfd7b6a

Commits on May 2, 2026

tests/experimental/test_unique.py passing in CI
mroeschke
committed
29b2527
Avoid pl.from_arrow to avoid a pyarrow import
mroeschke
committed
6746ab5
Use DataFrame container to avoid pyarrow
mroeschke
committed
487e28c

Commits on May 4, 2026

Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
2fc0975
Remove groupby_n_ary parameter
mroeschke
committed
11895e1
Remove unique_fraction
mroeschke
committed
6455625
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
e4c27ef
Remove remaining rapidsmpf_spill usage
mroeschke
committed
405d012
Change the shuffle_method to be internally set
mroeschke
committed
a84cd04
Remove StreamingOptions.runtime
mroeschke
committed
14e0545
Remove importskipping checks for rapidsmpf
mroeschke
committed
c0adafe

Commits on May 5, 2026

Fix error match for test_cuda_stream_policy_pool_in_memory_unsupported
mroeschke
committed
d9d3375
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
5ea9e8c
Retrigger CI
mroeschke
committed
5093c79
Remove more task runtime code suggested by mads
mroeschke
committed
b8cd7c4
Remove additional PartitionInfo code identified by claude
mroeschke
committed
d481460
Remove ShuffleMethod per Rick's review
mroeschke
committed
965884e
More removals from lawrence
mroeschke
committed
2f03ba6
Final dead code removals identified by claude
mroeschke
committed
fe43ff3
Remove ShuffleMethod from api.md
mroeschke
committed
3679280
make stream keyword argument to plc.Table.from_arrow
mroeschke
committed
b3a12db
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
0a5be84
add rapidmpf as a mounted sibbling directory
mroeschke
committed
c69e2d8
Condintionally clone rapidsmpf in initializeCommand
mroeschke
committed
6b43470
Add libnumba-dev for cucascade for pip devcontainers
mroeschke
committed
c94c90b
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
22a6a6f
Apply suggestions from code review

mroeschke
and
trxcllnt
authored
fd529e2
Merge branch 'enh/cudf_polars/rapidsmpf_default' of https://github.com/mroeschke/cudf into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
7a7b271
clone-rapidsmpf before build-all in build_command
mroeschke
committed
57b92fb

Commits on May 6, 2026

Revert "Add libnumba-dev for cucascade for pip devcontainers"
mroeschke
committed
de4ade6
Reapply "Add libnumba-dev for cucascade for pip devcontainers"
mroeschke
committed
126bdc9
Try matching rapidsmpf base pip docker image
mroeschke
committed
e062e80
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
02d33f4
Add -DBUILD_NUMA_SUPPORT=OFF in build_command
mroeschke
committed
85af11b
Add DcuCascade_DIR to force CPM to build from source (according to Claude)
mroeschke
committed
4d07468
Add devcontainers section in dependencies excluding rapidsmpf
mroeschke
committed
46b607e
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
5afc788
Remove more runtime inclusions
mroeschke
committed
02ba87d
Only include cudf, rapidsmpf in include_repos
mroeschke
committed
12fa676
try reforcing the pip env generation after clone-rapidsmpf
mroeschke
committed
833e5c7
don't build tests because of rapidsmpf
mroeschke
committed
4d60df4
Merge remote-tracking branch 'upstream/main' into enh/cudf_polars/rapidsmpf_default
mroeschke
committed
38e890e
Add -DCUDF_BUILD_TESTUTIL=OFF
mroeschke
committed
0646bb4
Don't build benchmarks either
mroeschke
committed
7a90931

Commits on May 7, 2026