Implement streaming window functions in cudf-polars - #22191

Merged

rapids-bot[bot] merged 96 commits into

NVIDIA:release/26.06from

Matt711:fea/polars/streaming-over

May 18, 2026

Commits on Apr 17, 2026

Support streaming over expression
Matt711
committed
5861da9
Add fast path for scalar aggs case, support preserving the full oreder
Matt711
committed
66be8e9
add more tests, protect against OOMs when preserving the full order
Matt711
committed
3b15601
check style
Matt711
committed
6b91f1b
add pre-shuffle test, and clarifying comments
Matt711
committed
ac73ede
add multi-rank test
Matt711
committed
0c8e3de
style
Matt711
committed
696bbc9
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
fdc40c9
reject non decomposable aggregations
Matt711
committed
e477439
fix xfail condition for mg test
Matt711
committed
4a6eb6d
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
0c48e1d
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
0d6d6b1
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
01d8777

Commits on Apr 21, 2026

Add Over IR node
Matt711
committed
f1cf13c
decompose select with over expressions with mixed partition by keys
Matt711
committed
4ac5e65
move IR to type checking block
Matt711
committed
c02eea9
add Over IR node
Matt711
committed
edc704e

Commits on Apr 22, 2026

merge conflict
Matt711
committed
5d28fb9

Commits on May 1, 2026

merge conflict
Matt711
committed
4fc4d25

Commits on May 2, 2026

Merge branch 'main' of https://github.com/rapidsai/cudf into fea/polars/streaming-over
Matt711
committed
e1442f2

Commits on May 4, 2026

add _decompose_grouped_window_node
rjzamora
committed
d632dd4
Merge remote-tracking branch 'upstream/main' into streaming-over-rick
rjzamora
committed
a7a914a
cull unnecessary code
rjzamora
committed
6bee1b8
address review
Matt711
committed
4ca3e3c
heavy revisions
rjzamora
committed
d690b1e
Merge remote-tracking branch 'upstream/main' into fea/polars/streaming-over
rjzamora
committed
9dab468
Merge branch 'streaming-over-rick' into fea/polars/streaming-over
rjzamora
committed
f8db130
Merge branch 'main' into fea/polars/streaming-over
rjzamora
authored
3893253
fix non-scalar over() by absorbing passthrough cols into Over node
Matt711
committed
ca1e808

Commits on May 5, 2026

Commits on May 6, 2026

restoring _fuse_over_nodes
Matt711
committed
563bac7
merge conflict
Matt711
committed
dad1a79
update fixture name
Matt711
committed
30e8826
add a second (reverse) shuffle for mg correctness
Matt711
committed
f9fdd46
estimate the modulus
Matt711
committed
c6a7b10
remove xfail marker from test
Matt711
committed
3051291
simplify skip condition in spmd over test
Matt711
committed
bd076c3
set allow_subset=True
Matt711
committed
12620c8
remove input_ir arg
Matt711
committed
5f4dd3e
simplify conditional statement
Matt711
committed
09a4586
TODO tracer API
Matt711
committed
ccaee76
rename split* instead of boundaries
Matt711
committed
bc584d7
use int32
Matt711
committed
a723dc8
clean up docstring OriginStamps docstring
Matt711
committed
902a4cd
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
338757a

Commits on May 7, 2026

fix upstream polars tests
Matt711
committed
0a94101
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
370fc21
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
5b25eea
address driveby nits
Matt711
committed
7c81adf
drop id() as dict key
Matt711
committed
68a7983
simplify _evaluate_window_with_stamps
Matt711
committed
7975ff4
address more nits
Matt711
committed
a6f397c
docstrings & use names_to_indices
Matt711
committed
ea39bd7
simplify scalar-Over IR + cleanup
Matt711
committed
658e4bf
colleect ir rewrite for scalar path ahead of time, comment improvements, clean up Over.do_evaluate signature
Matt711
committed
d3b6562
add module doc string overviewing the algorithm, few smaller clean ups
Matt711
committed
7b828be
more clean ups
Matt711
committed
397d50f
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
946582a
oh yeah, dont use to_arrow
Matt711
committed
f7687f1

Commits on May 11, 2026

merge conflict
Matt711
committed
beee7d9
Merge branch 'main' of https://github.com/rapidsai/cudf into fea/polars/streaming-over
Matt711
committed
2b8b357
no allgather unecessarily
Matt711
committed
fc9f7bb

Commits on May 12, 2026

update NormalizedPartitioning.from_keys call site
Matt711
committed
3a23609
merge conflict
Matt711
committed
6f29e67
move names_to_indices to avoid circular import
Matt711
committed
564aaf0
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
4d10e42
shared name generator
Matt711
committed
8cdf98b
address small nits
Matt711
committed
0582e29
more fuse over nodes simplifications
Matt711
committed
191e537
Stream-sync the broadcast against global_agg_df in rapidsmpf over actor
Matt711
committed
5dbb1dc
clean ups
Matt711
committed
232b8ae
more review nits
Matt711
committed
dfb3ffe
remove rename comment and clarify docstrings for scalar agg path
Matt711
committed
caadc2f

Commits on May 13, 2026

promote ChunkStore to a shared utility and use it in rapidsmpf over actor
Matt711
committed
0db0c7f
abstract allgather-post-reduce pattern
Matt711
committed
f63c7db
Use LocalRepartitioner to avoid materialising the full return shuffle
Matt711
committed
80cb585
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
5480a29
add TODO for slicing the input when only rank 0 inserts
Matt711
committed
b834838
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
4cc0003
zip local partition ids with sequence numbers in the over emit loop
Matt711
committed
9491a9e
add a TODO for per stream work
Matt711
committed
92501aa
Merge branch 'main' into fea/polars/streaming-over
Matt711
authored
cf5b099
distinct stream per chunk
Matt711
committed
96f24e0

Commits on May 14, 2026

check is strictly partitioned
Matt711
committed
520badf
exercise both single- and multi-row chunks in test_over_many_partitions
Matt711
committed
0483abd
merge conflict
Matt711
committed
d2a0a60

Commits on May 15, 2026

address small nits
Matt711
committed
5b75ae3
prune gw_nodes; tighten bound for reserve_extra; use streaming fixture in tests
Matt711
committed
590317f
Merge branch 'release/26.06' into fea/polars/streaming-over
Matt711
authored
8162194

Commits on May 18, 2026

Merge branch 'release/26.06' into fea/polars/streaming-over
Matt711
authored
bb0a75f