[HUDI-4039] Make sure all builtin `KeyGenerator`s properly implement Spark specific APIs#5523

Merged

yihua merged 58 commits intoapache:masterfrom

onehouseinc:ak/spk-kgen-opt

Jul 22, 2022

Commits on Jul 21, 2022

XXX
Alexey Kudinkin
committed
Inlined `KeyGenerator.getRecordKeyFieldNames` usages, rebased onto `getRecordKeyFields`
Alexey Kudinkin
committed
Cleaned up `SparkKeyGeneratorInterface`: moved in API defs from `BuiltinKeyGenerator`, added java-docs;
Alexey Kudinkin
committed
Rebased `BuiltinKeyGenerator` to return `UTF8String` for API ingesting `InternalRow`
Alexey Kudinkin
committed
Abstracted away nested-field path access, and Spark's `Row` conversion w/in corresponding `SparkRowAccessor`, `SparkRowConverter` w/in `BuiltinKeyGenerator`
Alexey Kudinkin
committed
Rebased `SimpleKeyGenerator`, `NonpartitionedKeyGenerator` onto new framework
Alexey Kudinkin
committed
Propery handle null partition-path parts
Alexey Kudinkin
committed
Rebased remaining builtin key generators
Alexey Kudinkin
committed
Modified `SparkKeyGeneratorInterface` to return `UTF8String` instead of byte array
Alexey Kudinkin
committed
Added `NestedFieldPath` to improve readability when used in Java
Alexey Kudinkin
committed
Cleaned up `RowKeyGeneratorHelper`
Alexey Kudinkin
committed
Fixed `HoodieDatasetBulkInsertHelper`
Alexey Kudinkin
committed
Fixing compilation (after rebase)
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Fixing partition-path extraction in `BulkInsertDataInternalWriterHelper`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Cloned `UTF8StringBuilder` to accommodate for Spark 2
Alexey Kudinkin
committed
Fixing invalid cast
Alexey Kudinkin
committed
Added missing partition path encoding;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing raw values deserialization when extracted from `InternalRow`s;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Extracting composite-key proper composition sequence into `BuiltinKeyGenerator`;
Alexey Kudinkin
committed
Fixed handling of null/empty key-parts for composite-keys to be in-sync b/w Avro/Row seqs in `KeyGenerator`
Alexey Kudinkin
committed
Handle failing to resolve nested field-paths;
Alexey Kudinkin
committed
Fixed handling all-null/empty composite keys
Alexey Kudinkin
committed
Fixing encoding for `Row` seq;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed empty/null partition-path part handling
Alexey Kudinkin
committed
Fixing typo;
Alexey Kudinkin
committed
Fixing more tests
Alexey Kudinkin
committed
Fixed expected exception wrapping
Alexey Kudinkin
committed
Fixed NPE
Alexey Kudinkin
committed
Unified key-gen sequences to be generic across String/UTF8String
Alexey Kudinkin
committed
Fixed null/empty predicate testing
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Rewrote key-gen tests to avoid duplication, add more cases;
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing NPE
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed invalid predicate
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Reverting `KeyGenerator`s API breaking name change
Alexey Kudinkin
committed
Fixed more tests
Alexey Kudinkin
committed
Tidying up, elaborating docs
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed `HoodieRowCreateHandle` to properly instantiate `HoodieParquetWriteSupport` w/ and w/o meta-fields (switching on/off Bloom Filter)
Alexey Kudinkin
committed
Fixed `HoodieRowCreateHandle` to properly write w/ and w/o meta-fields
Alexey Kudinkin
committed
Fixing `CustomKeyGenerator`
Alexey Kudinkin
committed
Fixed `HoodieTimer` API
Alexey Kudinkin
committed
Fixing `CustomKeyGenerator` some more
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed `UTF8String` handling in `TimestampBasedKeyGenerator`;
Alexey Kudinkin
committed
Fixing mutable-buffer-aliasing bug `HoodieInsertDataInternalWriterHelper`
Alexey Kudinkin
committed

Commits on Jul 22, 2022

Fallback to default for hive-style partitioning, url-encoding configs
Alexey Kudinkin
committed
Always dump stdout (since sometimes spark-shell wouldn't exit, and docker will report it as exit-code 0)
Alexey Kudinkin
committed
Reverting `KafkaConnectUtils` change
Alexey Kudinkin
committed