Spark, Arrow, Parquet: Add vectorized read support for parquet v2 encodings

[jbewing]

What

This PR adds vectorized read support to Iceberg for the Apache Parquet v2 specification (see #7162). This builds on top of the existing support for reading DELTA_BINARY_PACKED implemented by @eric-maynard in #13391 with:

• Building upon @eric-maynard 's support for DELTA_LENGTH_BYTE_ARRAY encoding in Add support for DELTA_LENGTH_BYTE_ARRAY Parquet encoding #13709
  • I made a couple of changes to that implementation around using temporary on-heap buffers rather than off-heap buffers as while it's possible to use temporary off-heap buffers, the plumbing required is significant and I wasn't able to get things totally resolved with that path working as there was a remaining subtle bug around leaked off-heap buffers & correctness
• Implementing vectorized read support for DELTA_BYTE_ARRAY encoding
• Implementing vectorized read support for BYTE_STREAM_SPLIT encoding
• Implementing vectorized read support for RLE encoded data pages
• Bolstering golden files test coverage to cover each of the paths above. In addition, I added golden file tests that include rows with null values for each data type as well to ensure our handling of those is correct

Background

This solves a longstanding issue of: the reference Apache Iceberg Spark implementation with the default settings enabled (e.g. spark.sql.iceberg.vectorization.enabled = true) isn't able to read iceberg tables that may have been written by other compute engines that utilize the not so new anymore Apache Parquet v2 writer specification. It's a widely known workaround to need to disable the vectorized reader in Spark if you need to interop with other compute engines or adjust all compute engines to use the Apache Parquet v1 writer specification when writing parquet files. With disabling the vectorization flag, clients take a performance hit that we've anecdotally measured is quite large in some cases/workloads. If forcing all writers of an iceberg table to write in Apache Parquet v1 format, clients are incurring additional performance and storage penalties (files written with parquet v2 tend to be smaller than those written with the v1 spec as the new encodings tend to save space and are often faster to read/write). So really, it's a lose-lose for performance & data size in the current setup with the additional papercut of Apache Iceberg not being super portable across engines in it's default setup. This PR seeks to solve that by finishing the swing on implementing vectorized parquet read support for the v2 format. In the future, we may also consider allowing clients to write Apache Parquet v2 files natively gated via a setting from Apache Iceberg. Even longer down that road, we may even consider changing that to be the "default" setting.

Previous Work / Thanks

This PR is a revival + extension to the work that @eric-maynard was doing in #13709. That PR had been active for a little while, so I literally started from where Eric left off. Thank you for the great work here @eric-maynard, you made implementing the rest of the changes required for vectorized read support way easier!

Note to Reviewers

I debated splitting up the implementation by encoding type but decided that given that the diff isn't crazy with everything all together that it may be better to bunch these all together. Especially since sometimes internal APIs got added on successive encoding versions to support implementations of subsequent encodings. I can split this up into multiple PRs if we think that would be substantially easier to review or if we're having problems reviewing this as a single atomic unit.

Testing

I've tested this on a fork of Spark 3.5 & Iceberg 1.10.0 and verified that a Spark job is able to read a table written with Parquet V2 writer without issues.

Successor to: #13290, #13391, #13709
Issue: #7162

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.