[HUDI-4250][HUDI-4202] Optimizing performance of Column Stats Index reading in Data Skipping #5746

Merged

yihua merged 43 commits intoapache:masterfrom

onehouseinc:ak/col-st-opt

Jul 25, 2022

Commits on Jul 25, 2022

Converted all methods into static ones;
Alexey Kudinkin
committed
Pass all filters including partition ones to Column Stats evaluation seq
Alexey Kudinkin
committed
Using `CachingPath`;
Alexey Kudinkin
committed
Abstracted `ColumnStatsIndexSupport` as standalone component
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Persist CS data-frame to avoid recomputing
Alexey Kudinkin
committed
Revert "Persist CS data-frame to avoid recomputing"
Alexey Kudinkin
committed
Avoid `InternalRow` > `Row` conversion
Alexey Kudinkin
committed
XXX
Alexey Kudinkin
committed
Avoid serializing full MT payload, and instead just cherry-pick CSI payload for serialziation
Alexey Kudinkin
committed
Use native `Dataset` API to make sure WSCG could collapse most of the computation into a single stage
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Rollback `ColumnStatsIndexSupport` onto proper RDD API
Alexey Kudinkin
committed
Add a flag to whether MT should load records as RDD/In-memory
Alexey Kudinkin
committed
Made `JFunction` convertes implicit
Alexey Kudinkin
committed
XXX
Alexey Kudinkin
committed
Rebased `ColumnStatsIndexSupport` onto leveraging `HoodieData` to be able to discretionary load CSI either a) via RDD on Cluster, b) in-memory
Alexey Kudinkin
committed
Added more utils to create `Dataset`s based on `LocalRelation`;
Alexey Kudinkin
committed
Fixed `TestColumnStatsIndex`
Alexey Kudinkin
committed
Fixed handling of non-indexed columns (to avoid reporting any stats for such columns whatsoever)
Alexey Kudinkin
committed
Fixing non-serializable task;
Alexey Kudinkin
committed
Rebased `HoodieFileIndex` onto the new API;
Alexey Kudinkin
committed
Added config to overide whether CSI should be read in-memory or by Spark
Alexey Kudinkin
committed
Extrated `shouldReadInMemory` utility to CSI
Alexey Kudinkin
committed
Moved `isIndexAvailable` check into `ColumnStatsIndexSupport`
Alexey Kudinkin
committed
Adding missing scala-docs
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Missing changes to `BaseHoodieTableFileIndex`
Alexey Kudinkin
committed
Added caching to `ColumnStatsIndexSupport`
Alexey Kudinkin
committed
Test `COLUMN_STATS_INDEX_PROCESSING_MODE_OVERRIDE` working appropriately
Alexey Kudinkin
committed
Typo
Alexey Kudinkin
committed
Make caching w/in `ColumnStatsIndexSupport` configurable
Alexey Kudinkin
committed
Fixed `shouldReadInMemory` check
Alexey Kudinkin
committed
Fixed compilation
Alexey Kudinkin
committed
Convert Scala lambdas to Java's ones explicitly (required for Scala 2.11)
Alexey Kudinkin
committed
Extracted `convertStorageLevelToString` to `SparkAdapter`
Alexey Kudinkin
committed
Fixed compilation
Alexey Kudinkin
committed
Delay sanity assertions;
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Neutralize the config to be engine-agnostic (and not locked-in to Spark)
Alexey Kudinkin
committed
Made in-memory projection threshold configurable
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed