[HUDI-3594] Supporting Composite Expressions over Data Table Columns in Data Skipping flow#4996

Merged

yihua merged 37 commits intoapache:masterfrom

onehouseinc:ak/mtmod-idx-2

Mar 25, 2022

Commits on Mar 25, 2022

Revisited Data Skipping utils to accept broader scope of "foldable" expressions (in lieu of just "literals");
Alexey Kudinkin
committed
Expanded scope even further to include any expression not referencing any other attributes or holding sub-query exprs
Alexey Kudinkin
committed
Refactor Column Stats Index filter expression translation seq to support arbitrary expressions involving no more than single column;
Alexey Kudinkin
committed
Added test applying Spark standard functions to source Data Table columns in filters
Alexey Kudinkin
committed
Grouped together logically equivalent expressions
Alexey Kudinkin
committed
Generalize all DS patterns to accept "Single Attribute Expressions" (ie expression containing exactly one attribute reference, and no sub-queries)
Alexey Kudinkin
committed
Added composite expression tests
Alexey Kudinkin
committed
Added test for non-literal value expression;
Alexey Kudinkin
committed
Added tests for `like`, `not like` operators
Alexey Kudinkin
committed
Tightened up permitted transformation expression to only accept ones that preserve ordering of the source column;
Alexey Kudinkin
committed
Rebased allowed transformation matching seq to match permitted transformations recursively
Alexey Kudinkin
committed
Extracted Expression utils to `HoodieCatalystExpressionUtils`
Alexey Kudinkin
committed
Worked around bug in Spark not allowing to resolve expressions in a single pass
Alexey Kudinkin
committed
Added tests for composite expression (w/ nested function calls)
Alexey Kudinkin
committed
Added `HoodieSparkTypeUtils`;
Alexey Kudinkin
committed
Simplify expression resolution considerably
Alexey Kudinkin
committed
Fixing incorrect casting
Alexey Kudinkin
committed
Tidying up java-docs
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Adding explicit type (Scala 2.11 not able to deduce it)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Scaffolded `HoodieCatalystExpressionUtils` as Spark-specific object;
Alexey Kudinkin
committed
Bootstrapped Spark2 & Spark3 specific `HoodieCatalystExpressionUtils`
Alexey Kudinkin
committed
Fixing refs
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Rebasing refs in `DataSkippingUtils`
Alexey Kudinkin
committed
Missing imports
Alexey Kudinkin
committed
Fixing refs
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Inlined `swapAttributeRefInExpr` util
Alexey Kudinkin
committed
Branched out `HoodieSpark3_2CatalystExpressionUtils` to support Spark 3.2.x;
Alexey Kudinkin
committed
`HoodieSpark3CatalystExpressionUtils` > `HoodieSpark3_1CatalystExpressionUtils`
Alexey Kudinkin
committed
Rebased `Spark3Adapter` to become `BaseSpark3Adapter`;
Alexey Kudinkin
committed
Dangling ref
Alexey Kudinkin
committed
Fixed `ColumnStatsIndexHelper` handling min/max values from `HoodieColumnRangeMetadata`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed