Hive: Vectorized ORC reads for Hive

[szlta]

Currently Hive relies on org.apache.iceberg.mr.mapreduce.IcebergInputFormat for reading data from Iceberg. This in turn relies on the internal reader implementations for the various file formats, such as GenericOrcReader(s). Unfortunately for Hive, the whole read pipeline is record based, whereas GenericRecord instances hold the information on which Hive has to use the Iceberg object inspectors to retrieve the encoded data from.

From a design point of view I see 3 layers where during reading data flows upwards:

• Execution engine: Hive
• Table format: Iceberg
• File format: ORC/Parquet/Avro

The current design however combines the two bottom layers, and although Hive already has support for vectorized execution, and already has implementations for vectorized reading from various file formats; with Iceberg in the picture it can't have the advantages of vectorization.

In this change I propose to swap out the bottom layer with the implementations Hive already has, so in the case of ORC: with VectorizedOrcInputFormat. This produces VectorizedRowBatch instances which Hive expects as its vectorized format. This could be used as the 'Hive' in memory data model, something IcebergInputFormat already distinguishes upon.

One big obstacle is that Iceberg depends on a type of ORC that has the nohive classifier. This means that classes that are actually coming from Hive/storage-api are shaded inside ORC with an ORC classname. In order to revert this while also causing the minimal disruption to other parts of Iceberg I propose to create an intermediary module for iceberg-hive3 to depend on (rather then depending on iceberg-orc and iceberg-data)

Tracking issue: #2612

[szlta]

cc: @rdblue @rymurr

[rymurr]

I am not a hive expert so I don't have a strong opinion on the Hive specific stuff. the only thing I notice is we have added some less than stellar code from Hive that I would rather see cleanly implemented. I understand this is temporary, what are the timelines for Hive4 and iceberg being able to drop the copied Hive code?

[rymurr]

how does this compare to the original hive bundle? Do we need to support both? Apologies if thats a dumb question, I am just not 100% sure I understand what will be in this resulting bundle and who will use it.

[szlta]

Thanks for taking a look on this @rymurr!
Originally I wanted to stick to not modifying any code snippets that needs to be copied from Hive to Iceberg as it's meant to be temporary. But.. since I don't see Hive4 getting released in the foreseeable future yet I have now rather amended these as per checkstyle.

About this hive3-orc-bundle: it's something that only hive3 depends on, and the soul purpose of it is to re-shade hive-storage-api classes that are currently being referred to as org.apache.orc.storage... within Iceberg.
This bundle will produce iceberg-data, and iceberg-orc classes with the original package name references to storage-api classes which is required runtime, where Hive3 code (from Hive codebase) is already linked with the original class names.
The reason this is not a dependency of iceberg-mr module, is that we won't support vectorization for the Hive2 integration of Iceberg

[pvary]

Thanks for the review @rymurr!

@rdblue, @RussellSpitzer: Would you like to take a look before we merge this change? If you have time, I would really appreciate it!

Thanks,
Peter

[pvary]

Thanks @szlta for the PR and @rymurr for the review!

[szlta]

Thanks for the reviews and the commit!