[HUDI-3396] Refactoring `MergeOnReadRDD` to avoid duplication, fetch only projected columns#4888

Merged

xushiyan merged 37 commits intoapache:masterfrom

onehouseinc:ak/spkds-ref-3

Mar 25, 2022

Commits on Mar 25, 2022

Rebased `HoodieUnsafeRDD` to become a trait;
Alexey Kudinkin
committed
Refactored `HoodieMergeOnReadRDD` iterators to re-use iteration logic
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up (cont)
Alexey Kudinkin
committed
Cleaned up MOR Relation data-flows
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Adding missing docs
Alexey Kudinkin
committed
Decoupled Record Merging iterator from assuming particular schema of the file reader, and instead rely on explicitly provided one
Alexey Kudinkin
committed
Removed superfluous `Option`, causing incorrect split processing
Alexey Kudinkin
committed
Added optimization to avoid full-schema base-file read in cases when
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Replacing self-type restriction w/ sub-classing
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
After rebase fixes
Alexey Kudinkin
committed
Tidying up (after rebase)
Alexey Kudinkin
committed
Extracted `DeltaLogSupport` trait encapsulating reading from Delta Logs
Alexey Kudinkin
committed
Fixed Merging Record iterator incorrectly projecting merged record
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Inlined `DeltaLogSupport` trait
Alexey Kudinkin
committed
Fixing NPE
Alexey Kudinkin
committed
Adjusted column projection tests to reflect the changes;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Allow non-partitioned table to rely on virtual-keys
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Pushed down `mandatoryColumns` defs into individual Relations
Alexey Kudinkin
committed
Clarified `requiredKeyField` semantic
Alexey Kudinkin
committed
Relaxed requirements to read only projected schema to only cases when non-whitelisted Record Payload classes are used
Alexey Kudinkin
committed
Fixed tests for Spark 2.x
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed tests for Spark 3.x
Alexey Kudinkin
committed
Revert some inadvertent changes
Alexey Kudinkin
committed
Clean up referecnes to MT
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Inlined `maxCompactionMemoryInBytes` w/in `HoodieMergeOnReadRDD`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed