[HUDI-2814] Addressing issues w/ Z-order Layout Optimization#4060

Merged

vinothchandar merged 59 commits intoapache:masterfrom

onehouseinc:ak/zord-fix-2

Nov 26, 2021

Commits on Nov 26, 2021

`ZCurveOptimizeHelper` > `ZOrderingIndexHelper`;

Alexey Kudinkin
authored and
vinothchandar
committed
Tidying up `ZOrderingIndexHelper`

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing compilation

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed index new/original table merging sequence to always prefer values from new index;

Alexey Kudinkin
authored and
vinothchandar
committed
Added test for `mergeIndexSql`

Alexey Kudinkin
authored and
vinothchandar
committed
Abstracted Z-index name composition w/in `ZOrderingIndexHelper`;

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed `DataSkippingUtils` to interrupt prunning in case data filter contains non-indexed column reference

Alexey Kudinkin
authored and
vinothchandar
committed
Properly handle exceptions origination during pruning in `HoodieFileIndex`

Alexey Kudinkin
authored and
vinothchandar
committed
Make sure no errors are logged upon encountering `AnalysisException`

Alexey Kudinkin
authored and
vinothchandar
committed
Cleaned up Z-index updating sequence;

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed Z-index to properly handle changes of the list of clustered columns

Alexey Kudinkin
authored and
vinothchandar
committed
Tidying up

Alexey Kudinkin
authored and
vinothchandar
committed
`lint`

Alexey Kudinkin
authored and
vinothchandar
committed
Suppressing `JavaDocStyle` first sentence check

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed compilation

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing incorrect `DecimalType` conversion

Alexey Kudinkin
authored and
vinothchandar
committed
Refactored test `TestTableLayoutOptimization`

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed tests re-shuffling column order for Z-Index table `DataFrame` to align w/ the one by one loaded from JSON

Alexey Kudinkin
authored and
vinothchandar
committed
Scaffolded `DataTypeUtils` to do basic checks of Spark types;

Alexey Kudinkin
authored and
vinothchandar
committed
Added test for Z-index tables merging

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed import being shaded by creating internal `hudi.util` package

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed packaging for `TestOptimizeTable`

Alexey Kudinkin
authored and
vinothchandar
committed
Revised `updateMetadataIndex` seq to provide Z-index updating process w/ source table schema

Alexey Kudinkin
authored and
vinothchandar
committed
Make sure existing Z-index table schema is sync'd to source table's one

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed shaded refs

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed tests

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed type conversion of Parquet provided metadata values into Spark expected schemas

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed `composeIndexSchema` utility to propose proper schema

Alexey Kudinkin
authored and
vinothchandar
committed
Added more tests for Z-index:

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing source table

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing tests to read from Parquet w/ proper schema

Alexey Kudinkin
authored and
vinothchandar
committed
Refactored `ParquetUtils` utility reading stats from Parquet footers

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed incorrect handling of Decimals extracted from Parquet footers

Alexey Kudinkin
authored and
vinothchandar
committed
Worked around issues in javac failign to compile stream's collection

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed handling of `Date` type

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed handling of `DateType` to be parsed as `LocalDate`

Alexey Kudinkin
authored and
vinothchandar
committed
Updated fixture;

Alexey Kudinkin
authored and
vinothchandar
committed
Removed superfluous scheme adjusting when reading from Parquet, since Spark is actually able to perfectly restore schema (given Parquet was previously written by Spark as well)

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing race-condition in Parquet's `DateStringifier` trying to share `SimpleDataFormat` object which is inherently not thread-safe

Alexey Kudinkin
authored and
vinothchandar
committed
Tidying up

Alexey Kudinkin
authored and
vinothchandar
committed
Make sure schema is used upon reading to validate input files are in the appropriate format;

Alexey Kudinkin
authored and
vinothchandar
committed
Worked around javac (1.8) inability to infer expression type properly

Alexey Kudinkin
authored and
vinothchandar
committed
Updated fixtures;

Alexey Kudinkin
authored and
vinothchandar
committed
Fixing compilation after rebase

Alexey Kudinkin
authored and
vinothchandar
committed
Assert clustering have in Z-order layout optimization testing

Alexey Kudinkin
authored and
vinothchandar
committed
Tidying up exception messages

Alexey Kudinkin
authored and
vinothchandar
committed
XXX

Alexey Kudinkin
authored and
vinothchandar
committed
Added test validating Z-index lookup filter correctness

Alexey Kudinkin
authored and
vinothchandar
committed
Added more test-cases;

Alexey Kudinkin
authored and
vinothchandar
committed
Added tests for string expressions

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed incorrect Z-index filter lookup translations

Alexey Kudinkin
authored and
vinothchandar
committed
Added more test-cases

Alexey Kudinkin
authored and
vinothchandar
committed
Added proper handling on complex negations of AND/OR expressions by pushing NOT operator down into inner expressions for appropriate handling

Alexey Kudinkin
authored and
vinothchandar
committed
Added `-target:jvm-1.8` for `hudi-spark` module

Alexey Kudinkin
authored and
vinothchandar
committed
Adding more tests

Alexey Kudinkin
authored and
vinothchandar
committed
Added tests for non-indexed columns

Alexey Kudinkin
authored and
vinothchandar
committed
Properly handle non-indexed columns by falling back to a re-write of containing expression as `TrueLiteral` instead

Alexey Kudinkin
authored and
vinothchandar
committed
Fixed tests

Alexey Kudinkin
authored and
vinothchandar
committed
Removing the parquet test files and disabling corresponding tests
vinothchandar
committed