[HUDI-5685] Fixing deduplication in Bulk Insert row-writing path#7825

Closed

alexeykudinkin wants to merge 11 commits intoapache:masterfrom

onehouseinc:ak/blk-ins-ddup-fix

Commits on Feb 2, 2023

Tidying up
Alexey Kudinkin
committed
Fixed `HoodieDatasetBulkInsertHelper` to avoid assumption that meta-fields are not provided
Alexey Kudinkin
committed
Revisited row-deduping sequence to avoid `String` allocation (instead relying on native `UTF8String`);
Alexey Kudinkin
committed
Avoid populating meta-fields in case already present (will be overridden)
Alexey Kudinkin
committed
Extracted schema handling utils into `HoodieSqlCommonUtils`
Alexey Kudinkin
committed
Revisited `HoodieDatasetBulkInsertHelper` to avoid copying of the rows
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make sure that during de-duping in Bulk Insert (row-writer) path we're partitioning records in a way that
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Skip propagating unnecessary boolean flag
Alexey Kudinkin
committed