Use (un)spill_partitions from RapidsMPF

[pentschev]

#22810 was merged with changes to remove the (un)spill_partition functions from cudf-streaming in favor of using them from RapidsMPF. However, simultaneously #22814 got merged introducing some new use cases for those functions, that allowed
#22810 to pass CI and be merged. This change fixes coverage.

[madsbk]

Thanks @pentschev

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bf5f5938-402a-4b54-9bb8-38de6fd119f6

📥 Commits

Reviewing files that changed from the base of the PR and between 502bdad and 33992c0.

📒 Files selected for processing (10)

• ci/cpp_linters.sh
• cpp/libcudf_streaming/benchmarks/bench_shuffle.cpp
• cpp/libcudf_streaming/benchmarks/streaming/ndsh/join.cpp
• cpp/libcudf_streaming/examples/example_shuffle.cpp
• cpp/libcudf_streaming/tests/test_shuffler.cpp
• python/cudf_streaming/cudf_streaming/examples/bulk_mpi_shuffle.py
• python/cudf_streaming/cudf_streaming/examples/bulk_ray_shuffle.py
• python/cudf_streaming/cudf_streaming/examples/ray_shuffle_example.py
• python/cudf_streaming/cudf_streaming/tests/test_integration_partition.py
• python/cudf_streaming/cudf_streaming/tests/test_shuffler.py

---

📝 Walkthrough

Summary by CodeRabbit

• Refactor
  • Switched spill/unspill handling to use an external memory spill API across benchmarks, examples, and tests to standardize memory operations.
• Chores
  • CI linting script now normalizes compiler flags before running checks, removing incompatible GCC-only flags when present.

Walkthrough

This PR moves unspill_partitions calls and imports from cudf_streaming.integrations/partition to rapidsmpf (C++ and Python), adds rapidsmpf spill includes where needed, and normalizes some CI linter flags.

Changes

Unspill Function Namespace Migration

Layer / File(s)	Summary
C++ header dependencies for rapidsmpf spill cpp/libcudf_streaming/benchmarks/bench_shuffle.cpp, cpp/libcudf_streaming/benchmarks/streaming/ndsh/join.cpp, cpp/libcudf_streaming/examples/example_shuffle.cpp, cpp/libcudf_streaming/tests/test_shuffler.cpp	Adds rapidsmpf/memory/spill.hpp include to C++ translation units that call unspill_partitions.
C++ join benchmark namespace migration cpp/libcudf_streaming/benchmarks/streaming/ndsh/join.cpp	Removes the using cudf_streaming::integrations::unspill_partitions alias and updates both broadcast and shuffle unpack_and_concat call sites to use rapidsmpf::unspill_partitions.
C++ bench & example unspill updates cpp/libcudf_streaming/benchmarks/bench_shuffle.cpp, cpp/libcudf_streaming/examples/example_shuffle.cpp	Replaces per-partition/packed-data unspill calls with rapidsmpf::unspill_partitions at the unpack/unpack-and-concat sites.
C++ shuffler test unspill calls cpp/libcudf_streaming/tests/test_shuffler.cpp	Updates multiple test helper paths and TEST cases to call rapidsmpf::unspill_partitions instead of the prior cudf_streaming::integrations namespaced function.
Python import rewiring for rapidsmpf python/cudf_streaming/cudf_streaming/examples/bulk_mpi_shuffle.py, python/cudf_streaming/cudf_streaming/examples/bulk_ray_shuffle.py, python/cudf_streaming/cudf_streaming/examples/ray_shuffle_example.py, python/cudf_streaming/cudf_streaming/tests/test_integration_partition.py, python/cudf_streaming/cudf_streaming/tests/test_shuffler.py	Moves spill_partitions/unspill_partitions imports to rapidsmpf.memory.spill and narrows cudf_streaming.integrations.partition imports to partition helpers only.
CI clang-tidy flag normalization ci/cpp_linters.sh	Strips -fno-merge-constants from CFLAGS/CXXFLAGS when set and collapses/trims whitespace before running clang-tidy.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• Follow-ups for rapidsmpf to cudf migration #22771: Implements the migration of spill/unspill_partitions back to rapidsmpf referenced by this issue.

Possibly related PRs

• rapidsai/cudf#22810: Related migration PR that updates shuffle/partition call sites to use rapidsmpf::unspill_partitions.

Suggested reviewers

• nirandaperera
• vyasr
• madsbk

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.44% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Use (un)spill_partitions from RapidsMPF' clearly and concisely summarizes the main change across all modified files—switching from cudf_streaming integrations to RapidsMPF implementations.
Description check	✅ Passed	The description relates directly to the changeset by explaining the context (PRs 22810 and 22814) and purpose (fixing coverage gaps) for switching to RapidsMPF's (un)spill_partitions implementations.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nirandaperera]

LGTM. Thanks @pentschev

[jameslamb]

/merge