Use native implementation of Iceberg delete filters

[electrum]

Description

Improve performance of Iceberg queries for tables with updated or deleted rows.

Related issues, pull requests, and links

• Fixes Iceberg scanning with Delete Files is extremely/unusably slow #13092

Documentation

(x) No documentation is needed.

Release notes

(x) Release notes entries required with the following suggested text:

# Iceberg connector
* Improve performance of Iceberg queries for tables with updated or deleted rows. ({issue}`13092`)

[lhofhansl]

Works well on my scenario. And faster than #13112.
(Only tested postional deletes.)

[findepi]

@electrum can you please extract some of the prep easy commits to a separate PR to have them merged quickly?

[electrum]

@findepi extracted to #13236

[electrum]

@lhofhansl thanks for the feedback! That's great to hear.

[djsagain]

Lots of very nice cleanups in this PR!

I didn't take the time to figure out Iceberg's StructLikeSet and StructProjection, but the rest looked right to me.

[findepi]

@findepi extracted to #13236

thanks, approved that one

[findepi]

This creates intermediate nodes. It would be more efficient to have a composite flat "and" predicate with a list of sub-predicates

[electrum]

I considered that originally, but I'm not sure it would be more efficient. We expect to only have one or a few filters, so with inlining, the JVM might be able to de-virtualize and flatten the calls into a single expression. The flat list seems to have more overhead for the common cases.

[electrum]

We can optimize this later if it becomes an issue. An obvious one is to combine all the deletes into a single bitmap. For equality, we could use method handle combinators to combine the predicates, and possibly also to "compile" the equality comparison expression.

[electrum]

I combined position deletes into a single bitmap.

[findepi]

We probably need to track memory contained in this field to prevent worker node crashes.
See #9914
cc @losipiuk

[electrum]

Agreed, though we aren't doing that today, so this isn't a regression. We can do that as a follow up. I didn't see an easy way to get memory usage from RoaringBitmap. We could serialize and use that as an estimate, or do a simple estimate based on the cardinality (multiply by some constant factor).

[alexjo2144]

Nice