[HUDI-3993] Replacing UDF in Bulk Insert w/ RDD transformation#5470

Merged

yihua merged 46 commits intoapache:masterfrom

onehouseinc:ak/spk-blk-ins-opt

Jul 21, 2022

Commits on Jul 19, 2022

Minor optimizations
Alexey Kudinkin
committed
Cleaned up `BulkInsertDataInternalWriterHelper` init seq
Alexey Kudinkin
committed
Cleaning up partition path extraction in `BulkInsertDataInternalWriterHelper`
Alexey Kudinkin
committed
Added `getRecordKey` to generate record key ingesting `InternalRow` directly
Alexey Kudinkin
committed
Revisited `HoodieDatasetBulkInsertHelper` to avoid using UDFs, streamline the code and avoid unnecessary conversions
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing typo
Alexey Kudinkin
committed
Missing conversion to `UTF8String`
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed meta-fields ordering
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Re-implemented deduplication seq to avoid de/serialization
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Added tests for `HoodieUnsafeRowUtils`
Alexey Kudinkin
committed
Reduced branching in `HoodieUnsafeRowUtils`
Alexey Kudinkin
committed
Fixing test
Alexey Kudinkin
committed
Fixed seed for input dataset generation;
Alexey Kudinkin
committed
Fixed incorrect keeping the reference to a mutable instance of the row (w/o making a copy)
Alexey Kudinkin
committed
Fixing incorrect cast;
Alexey Kudinkin
committed
Moved `HoodieUnsafeRowUtils` into "hudi-spark-client" module
Alexey Kudinkin
committed
Inlined Scala DSL into plain loops
Alexey Kudinkin
committed
Avoid data-types collection in the hot-path;
Alexey Kudinkin
committed
Refactored `HoodieInternalRow` to make it compatible with rows not containing meta-fields
Alexey Kudinkin
committed
Refactored `HoodieRowCreateHandle` to avoid conversion into `String` from Spark's `UTF8String`
Alexey Kudinkin
committed
Make low-level Spark's `Row`-based `FileWriter`, `ParquetWriteSupport` accept `UTF8String`, instead of `String` to avoid conversion
Alexey Kudinkin
committed
Avoid converting `UTF8String` to Java's `String` on the hot-path;
Alexey Kudinkin
committed
Rebase `HoodieDatasetBulkInsertHelper` to use `HoodieInternalRow` as an overlay layer to avoid copying existing row
Alexey Kudinkin
committed
Cleaned up `HoodieRowCreateHandle` to avoid executing constant exprs in the hot-path
Alexey Kudinkin
committed
Fixed record counter
Alexey Kudinkin
committed
Make `HoodieRowCreateHandle` handle the cases when meta-fields are not populated;
Alexey Kudinkin
committed
Added tests
Alexey Kudinkin
committed
Cleaned up `HoodieInternalRow`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up;
Alexey Kudinkin
committed
Avoding unnecessary hash-map lookups;
Alexey Kudinkin
committed
Fixing compilation (restoring changes)
Alexey Kudinkin
committed
Fixing NPE
Alexey Kudinkin
committed
Fixing "hudi-spark3" POM:
Alexey Kudinkin
committed
Fixing copy-paste
Alexey Kudinkin
committed
Fixed pseudo-random gen to generate non-repeating sequences
Alexey Kudinkin
committed
Fixed `ComplexKeyGenerator`, `GlobalDeleteKeyGenerator` to properly override `InternalRow` methods;
Alexey Kudinkin
committed
Fixed deserialization of some logical types when handling `InternalRow` to make sure Avro/Spark key-gens are in-sync;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Replace `copy` w/ `clone` to avoid unnecessary underlying buffer copying
Alexey Kudinkin
committed

Commits on Jul 20, 2022

Refactored `HoodieDatasetBulkInsertHelper` to dedupe `RDD`s to avoid unnecessary `DataFrame` > `RDD` conversions
Alexey Kudinkin
committed