Support set operations

[ttnghia]

This PR adds the following APIs for set operations:

• lists::have_overlap
• lists::intersect_distinct
• lists::union_distinct
• lists::difference_distinct

Name Convention

Except for the first API (lists::have_overlap) that returns a boolean column, the suffix _distinct of the rest APIs denotes that their results will be lists columns in which all list rows have been post-processed to remove duplicates. As such, their results are actually "set" columns in which each row is a "set" of distinct elements.

---

Depends on:

• Generate group labels from offsets #10945
• Generate group offsets from element labels #11017
• Add static_multimap::pair_contains NVIDIA/cuCollections#175
• Support duplicate_keep_option in cudf::distinct #11052
• Support nan_equality in cudf::distinct #11118
• Refactor semi_anti_join #11100
• Implement lists::distinct and cudf::detail::stable_distinct #11149

Closes #10409.

[jrhemstad]

This looks like it has the potential to be a very large PR. I would strongly advocate for breaking this up into smaller, easier to review ones. Likely be doing one of the set_ algorithms at a time.

[ttnghia]

This looks like it has the potential to be a very large PR. I would strongly advocate for breaking this up into smaller, easier to review ones. Likely be doing one of the set_ algorithms at a time.

I'm not sure. If you read my algorithm descriptions (from here) then the implementation for each set-op should be very short. I will try to have smaller dependent PRs if I figured out what to extract from here.

Currently, I see some overlap in implementation for this PR and the cudf::contains with nested type supports (#10656). In both places, will are doing similar steps: calling create/insert/contains with map. Maybe I will extract the common code and put it somewhere in stream_compaction module.

[PointKernel]

Another round of review focuses on implementations. Unit tests not reviewed yet.

[bdice]

@ttnghia I have some comments attached. I think this is some of your best work I've reviewed. Really great job! The way the set algorithms all fall back to contains is very nice -- I love when we can write concise higher-order algorithms like these.

[bdice]

A few minor suggestions, otherwise LGTM!

[PointKernel]

Looks great!

I've learned a lot by going through your PRs in the chain and here we are finally! Thanks @ttnghia for your persistence and effort put into this work.

[ttnghia]

@gpucibot merge