[HUDI-3396] Make sure `BaseFileOnlyViewRelation` only reads projected columns#4818

Merged

nsivabalan merged 40 commits intoapache:masterfrom

onehouseinc:ak/spkds-ref-1

Mar 10, 2022

Commits on Mar 4, 2022

Refactored `HoodieTestDataGenerator` to make data generation reproducible
Alexey Kudinkin
committed
Extracted common fields to `HoodieBaseRelation`;
Alexey Kudinkin
committed
Cleaned up `BaseFileOnlyViewRelation`
Alexey Kudinkin
committed
Streamlined `BaseFileOnlyViewRelation`
Alexey Kudinkin
committed
Introduced `HoodieBaseRDD` to make sure all Hudi custom impls are implemented appropriately
Alexey Kudinkin
committed
Fixed all of the Relations to avoid superfluous Row <> InternalRow conversions
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing compilation after rebase
Alexey Kudinkin
committed
After rebase fixes
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
`HoodieBaseRDD` > `HoodieUnsafeRDD`;
Alexey Kudinkin
committed
`HoodieBaseRDD` > `HoodieUnsafeRDD`;
Alexey Kudinkin
committed
Bootstrapped `HoodieUnsafeRDDUtils`
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Bootstrapped test for MOR table validating proper column projections upon querying
Alexey Kudinkin
committed
Amended test to run against non-partitioned table (to query just a single file)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added tests with no Delta Logs
Alexey Kudinkin
committed
Extracted Parquet projection related tests into standalone file
Alexey Kudinkin
committed
Added payload-combine merging cases
Alexey Kudinkin
committed
Tiyding up
Alexey Kudinkin
committed
Broken down into multiple tests
Alexey Kudinkin
committed
Added test for COW
Alexey Kudinkin
committed
Added test for MOR Incremental Relation
Alexey Kudinkin
committed
Expose `mandatoryColumns` to be accessible in tests
Alexey Kudinkin
committed
Typo
Alexey Kudinkin
committed
`HoodieAvroSerializerTrait` > `HoodieAvroSerializer`;
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Added `SparkHadoopUtil` delegating to Spark's private impl
Alexey Kudinkin
committed
Fixed tests for Spark 2.x
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Removing logs from MOR / Incremental test
Alexey Kudinkin
committed
Removed `SparkHadoopUtil`
Alexey Kudinkin
committed
Fixed `HoodieWriteConfig` to be able to disable compression in Parquet
Alexey Kudinkin
committed
Disable compression in MOR / Incremental test to make test stable
Alexey Kudinkin
committed
Disable MOR / Incremental test
Alexey Kudinkin
committed

Commits on Mar 5, 2022

Disable full-table-read tests as unstable in Spark 2.x
Alexey Kudinkin
committed

Commits on Mar 7, 2022

Incorrect schema ref
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed