[HUDI-431] Adding support for Parquet in MOR `LogBlock`s#4333

Merged

nsivabalan merged 63 commits intoapache:masterfrom

onehouseinc:ak/parq-log-blk

Feb 2, 2022

Commits on Feb 1, 2022

Adding parquet data block and inline read support

nsivabalan
authored and
Alexey Kudinkin
committed
Assert accepted `Path` instances actually represent InlineFS block paths;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up `HoodieDataBlock`:
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing layout
Alexey Kudinkin
committed
Fixing `HoodieHFileDataBlock` to properly handle cases when point lookups are enabled for the records
Alexey Kudinkin
committed
Tidying up:
Alexey Kudinkin
committed
Control ser/de flow fully from w/in the `HodieDataBlock`
Alexey Kudinkin
committed
Added assertions for `HoodieDataBlock`;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added `ByteBufferBackedInputStream`
Alexey Kudinkin
committed
Rebased `HoodiHFileReader` onto `ByteBufferBackedInputStream` (to avoid `ByteArrayInputStream` locking on every operation)
Alexey Kudinkin
committed
Completed `HoodieParquetDataBlock`:
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up `HoodieLogFileReader`;
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Tidying up;
Alexey Kudinkin
committed
Reverted `hasContentLength` change
Alexey Kudinkin
committed
Fixed NPEs
Alexey Kudinkin
committed
Tidied up config
Alexey Kudinkin
committed
Added tests for Parquet data blocks
Alexey Kudinkin
committed
Streamlining
Alexey Kudinkin
committed
Extracted `ByteBufferBackedInputFile` as a standalone class
Alexey Kudinkin
committed
Extracted `OutputStreamBackedOutputFile` as a standalone class;
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Make sure `LogBlock`s schema is always set to either Reader's schema or Writer's one;
Alexey Kudinkin
committed
Added `Option.or`
Alexey Kudinkin
committed
Abstracted common point-wise record lookups w/in `HoodieDataBlock` to share across formats;
Alexey Kudinkin
committed
Disable point-lookups for Parquet data blocks
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Made `readRecordsFromContent` overridable to make sure that Parquet Block isn't reading columns that are not included into provided projected Reader's schema
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Repacked `HoodieLogFile` w/in `HoodieLogFileReader` to make sure that provided log-file path could be handled by Block readers;
Alexey Kudinkin
committed
Down-scoping some utils
Alexey Kudinkin
committed
Added test to assert that Data Block produces records w/ correct projected schema
Alexey Kudinkin
committed
Tydying up
Alexey Kudinkin
committed
Make sure `HoodieAvroDataBlock` doesn't alter original records' list
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Fixing styling
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up ctors used only in tests
Alexey Kudinkin
committed
Thread through appropritately configured Parquet compression codec into `HoodieParquetDataBlock`
Alexey Kudinkin
committed
Thread through appropriately configured Parquet compression codec into `HoodieParquetDataBlock`
Alexey Kudinkin
committed
Thread t/h Hadoop `Configuration` to the Log Blocks to make sure proper configuration is used when reading block's contents
Alexey Kudinkin
committed
Properly respect Hadoop Configuration
Alexey Kudinkin
committed
Fixed compilation
Alexey Kudinkin
committed
Leverage appropriately configured HFile compression algo
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Avoid superfluous `fs.getFileStatus` by amending the Path w/ Scheme directly
Alexey Kudinkin
committed
Added UT for `ByteBufferBackedInputStream`
Alexey Kudinkin
committed
Fixed how Path is qualified w/ FS default URI, scheme
Alexey Kudinkin
committed
Properly pass in file-size when re-packaging `HoodieLogFile`;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Moved `LOGFILE_DATA_BLOCK_FORMAT` into `HoodieWriteConfig`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Unused imports
Alexey Kudinkin
committed
After rebase fixes
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up more
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed

Commits on Feb 2, 2022

Fixed writers' ctor of the `HoodieHFileDataBlock`
Alexey Kudinkin
committed