[HUDI-3760] Adding capability to fetch Metadata Records by prefix #5208

Merged

nsivabalan merged 73 commits intoapache:masterfrom

onehouseinc:ak/mtd-kpfx

Apr 6, 2022

Commits on Apr 6, 2022

Added `getRecordsByKeyPrefix` to `HoodieHFileReader`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added `getRecordIteratorByKeyPrefix` to `HoodieHFileReader`;
Alexey Kudinkin
committed
Made `HoodieDataBlock` accept keys specified as prefixes;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Modified `AbstractHoodieLogRecordReader` to be able to scan records based on key-prefix rather than the full-key;
Alexey Kudinkin
committed
Adjusted `HoodieFileReader` interface
Alexey Kudinkin
committed
Add `HoodieBackedTableMetadata::getRecordsByKeyPrefixes`
Alexey Kudinkin
committed
Added test for `HFileReader` seq fetching records by key-prefixes
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing read from Metadata table for col stats partition

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing prefix lookup in HFile reader and enhancing tests for the same

nsivabalan
authored and
Alexey Kudinkin
committed
Adding test to TestHoodieBackedMetadata to test prefix look up in col stats partition in Metadata table

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing determining col stats partition availability in dataskipping code path

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing compilation error in test with rebase

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing HFileScanner usage and HoodieData for prefix based look up with metadata table

nsivabalan
authored and
Alexey Kudinkin
committed
fixing arg name for useCachedReaders

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing test failures related to write schema not added to Hfile on the write path

nsivabalan
authored and
Alexey Kudinkin
committed
Extracted common utils
Alexey Kudinkin
committed
Rebased `ColumnStatsIndexSupport` to fetch CSI records by key-prefix instead of reading the whole table (if available)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Rebased `HoodieFileIndex` onto updated API
Alexey Kudinkin
committed
Fixed `HoodieDataBlock` to properly handle when records are fetched by key-prefixes
Alexey Kudinkin
committed
Fixed `HoodieBackedTableMetadata` to avoid doing full-scans
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed iterator to properly loop
Alexey Kudinkin
committed
Fixed incorrect key-prefix access gen
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Removed references to HFile from `HoodieFileReader`;
Alexey Kudinkin
committed
Cleaned up `HoodieFileReader` APIs;
Alexey Kudinkin
committed
Cleaned up `HoodieHFileReader` APIs;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Removed APIs which dereference iterators from `HoodieFileReader`, instead inlining them where used
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed records iterators
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing some more
Alexey Kudinkin
committed
Fixing a little more
Alexey Kudinkin
committed
Fixing task being non-serializable
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Consolidated configuration around `forceFullScan` property for `LogReader`s
Alexey Kudinkin
committed
Adding assertions
Alexey Kudinkin
committed
Test with all log-files scan modes
Alexey Kudinkin
committed
Killed dead-code
Alexey Kudinkin
committed
Fixing incorrect sorting of the keys
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up `ColumnStatsIndexSupport` removing duplication
Alexey Kudinkin
committed
Fixed schemas used to read from MT, to be fetched from HFile
Alexey Kudinkin
committed
Fixed `TestHoodieFileIndex`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Disallow full-scans for "column_stats", "bloom_filters" partitions
Alexey Kudinkin
committed
Killed `fsDataInputStream`
Alexey Kudinkin
committed
Rebased `schema` field to become final, thread-safe
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added `sharedScanner` instance to be re-used by point-lookup queries
Alexey Kudinkin
committed
Fixed `RecordIterator`, `RecordByKeysIterator`
Alexey Kudinkin
committed
Fixed `RecordByKeyPrefixIterator`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed `RecordByKeyPrefixIterator` to properly handle EOF
Alexey Kudinkin
committed
Rebased `HoodieMergeOnReadRDD` to use Metadata Config from the File Index
Alexey Kudinkin
committed
Fixed opts for MT reads
Alexey Kudinkin
committed
Force MT full-scan when reading it from Spark DS;
Alexey Kudinkin
committed
Fixed query-type missing from config props in Spark Streaming
Alexey Kudinkin
committed
Added fallback to default val for `QUERY_TYPE` config in Spark File Index
Alexey Kudinkin
committed
Fixed `TestLayoutOptimization` missing required MT configs
Alexey Kudinkin
committed
Cleaned up duplicated methods
Alexey Kudinkin
committed
Close reader and remove public API from metadata payload
codope
committed