[HUDI-3896] Porting Nested Schema Pruning optimization for Hudi's custom Relations#5428

Merged

codope merged 46 commits intoapache:masterfrom

onehouseinc:ak/spk-schm-prn

Jul 21, 2022

Commits on Jul 19, 2022

Added `customOptimizerRules` to `HoodieAnalysis`;
Alexey Kudinkin
committed
Cleaning up utils
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Added `NestedSchemaPrunning` Spark's Optimizer rule;
Alexey Kudinkin
committed
Handle Spark's Optimizer pruned data schema (to effectively prune nested schemas)
Alexey Kudinkin
committed
Injecting Spark Session extensions for `TestMORDataSource`
Alexey Kudinkin
committed
Disabled fallback to `HadoopFsRelation`
Alexey Kudinkin
committed
Make sure extensions are loaded in COW/MOR tests
Alexey Kudinkin
committed
Fixed compilation for Scala 2.11
Alexey Kudinkin
committed
Added `JFunction` utility to convert b/w Scala/Java lambdas in Scala 2.11
Alexey Kudinkin
committed
Fixed compilation for Spark 2
Alexey Kudinkin
committed
Adding `HoodieSparkSessionExtensions` to quick-start tests
Alexey Kudinkin
committed
Fixing compilation in tests
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Internalized `canPruneRelationSchema` method w/in `HoodieBaseRelation` to make sure we're not apply `NestedSchemaPruning` rule when schema evolution is involved (since `InternalSchema` isn't being …
Alexey Kudinkin
committed
Adding missing scala-docs;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added test for Avro ser-/de
Alexey Kudinkin
committed
Made `SchemaConverters` to appropriatley transform Avro unions to Catalyst schema and back;
Alexey Kudinkin
committed
Disallow schema pruning for MT
Alexey Kudinkin
committed
Make sure we avoid unnecessary conversions for table's schema
Alexey Kudinkin
committed
Fixed union detection heuristic
Alexey Kudinkin
committed
Fixing test
Alexey Kudinkin
committed
Cleaning up handling of `InternalSchema`
Alexey Kudinkin
committed
Restore fallback to `HadoopFsRelation`
Alexey Kudinkin
committed
Fixed handling of Schema Evolution case when actual table's schema has to be derived from `InternalSchema` representation
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing Spark version checkers
Alexey Kudinkin
committed
Extracted Spark version checkers into a standalone trait;
Alexey Kudinkin
committed
Extracted all Catalyst `LogicalPlan` related utilities from `SparkAdapter` into standalone `HoodieCatalystPlanUtils`
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Bifurcated Spark 3.1 vs 3.2 `HoodieCatalystPlanUtils` implementations
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Added `createExplainCommand` to `HoodieCatalystPlanUtils`
Alexey Kudinkin
committed
Added test for `NestedSchemaPruning` optimization;
Alexey Kudinkin
committed
Fixing tests for Spark 2.4;
Alexey Kudinkin
committed
Fixed test for Spark 2.4
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Clean up superfluous flag (MT table will be ruled out by file-format predicate)
Alexey Kudinkin
committed
Fixing partition-path extraction for globbed paths
Alexey Kudinkin
committed

Commits on Jul 20, 2022

Fixed reading t/h globbed paths to properly handle case of partitioned tables not using Hive-style partitioning
Alexey Kudinkin
committed