Speed up hash partitioning

[Dandandan]

Is your feature request related to a problem or challenge?

Also see request in arrow apache/arrow-rs#4476

In DataFusion, a common operation is to repartition a RecordBatch by hashing one or more columns and dividing them into partition record batches using the "formula" hash % num_partitions.

The current approach is to create the indices that match and use them to take the individual arrays (see BatchPartitioner in datafusion).

This is relatively expensive however, as we visit the arrays num_partitions times in different places of the array, leading to cache inefficient operators (especially when the number of partitions is high).

Describe the solution you'd like

Faster hash-partitioning implementation

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I recommend we look into implementing Selection Vectors / bitmaks -- then repartitioning could become a calculation of such filters/ bitmasks