[HUDI-5633] Fixing performance regression in `HoodieSparkRecord`#7769

Merged

alexeykudinkin merged 50 commits intoapache:masterfrom

onehouseinc:ak/rfc46-perf-fix

Jan 31, 2023

Commits on Jan 31, 2023

Tidying up
Alexey Kudinkin
committed
Revisited `rewriteRecord` utility to decouple transformer generation (involving schema traversals) from actual transformation
Alexey Kudinkin
committed
Rebased `HoodieSparkSqlWriter` and `HoodieSparkRecord` onto using generated row-writer
Alexey Kudinkin
committed
Revisited common caches to avoid multiple lookups
Alexey Kudinkin
committed
Cleaned up unnecessary utils
Alexey Kudinkin
committed
Rebased rewriting Spark's Rows with renaming to decouple transformer composition from the actual execution
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed position map not being updated properly
Alexey Kudinkin
committed
Fixed array/map handling
Alexey Kudinkin
committed
Fixing support of Hudi's custom conversion to String
Alexey Kudinkin
committed
Fixing support of Hudi's custom conversion to Decimals
Alexey Kudinkin
committed
Fixing support of Hudi's custom conversion to Date
Alexey Kudinkin
committed
Fixing unsafe row-writer application
Alexey Kudinkin
committed
Adjusted core writing flows merging/inserting to avoid record's copying when is not required by the Executor used
Alexey Kudinkin
committed
Fixed usage of the caches
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing conversion into String
Alexey Kudinkin
committed
XXX
Alexey Kudinkin
committed
Revisited `MetadataValues` to avoid using `HashMap`
Alexey Kudinkin
committed
Rebased `HoodieRecord` impls to thrown unchecked exception
Alexey Kudinkin
committed
Avoid rewriting records twice when schema evolution is enabled;
Alexey Kudinkin
committed
Deduplicated record transformation and meta-fields prepending into 2 standalone operations:
Alexey Kudinkin
committed
Rebased `WriteHandle`s onto new `HoodieRecord` APIs
Alexey Kudinkin
committed
Fixed `BootstrapMetadataHandler`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing FP into Decimal conversion (might involve rounding)
Alexey Kudinkin
committed
Fixed `AppendHandle` to properly copy record's instance (to avoid being affected by deflation of the incoming one in some cases);
Alexey Kudinkin
committed
Fixed `HoodieSparkRecord` to properly update meta-fields
Alexey Kudinkin
committed
Fixed `composeNestedFieldPath` util to properly handle missing fields
Alexey Kudinkin
committed
Cleaning up
Alexey Kudinkin
committed
Tidying up `HoodieInternalRowUtils`;
Alexey Kudinkin
committed
Combined tests
Alexey Kudinkin
committed
De-duplicated `RowWriter` utils
Alexey Kudinkin
committed
Rebased `HoodieSparkSqlWriter` onto a new utils
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Reverting accidental changes
Alexey Kudinkin
committed
Fixing typo
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed `rewriteRecordWithNewSchema` mising to pass target columns to be renamed
Alexey Kudinkin
committed
Fixed map row-writer to not assume particular impl of `MapData`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Disable DS multi-writer test
Alexey Kudinkin
committed
Fixed `ParquetBootstrapMetadataHandler` to properly instantiate bootstrapping records
Alexey Kudinkin
committed
Cleaned up `updateMetadataFields`
Alexey Kudinkin
committed
Fixing `String` to `UTF8String` conversion
Alexey Kudinkin
committed
Fixed `HoodieSparkRecord` to avoid truncating list of meta-fields when bootstrapping
Alexey Kudinkin
committed
Cleaned up `HoodiesSparkParquetWriter` to avoid unnecessary conversions from `String` to `UTF8String`
Alexey Kudinkin
committed
Fixed `ParquetBootstrapMetadataHandler` to specify complete list of meta-fields to make sure `HoodieSparkRecord` invariants
Alexey Kudinkin
committed
Fixed `HoodieBootstrapHandle` to rely on `METADATA_BOOTSTRAP_RECORD_SCHEMA` instead of `RECORD_KEY_SCHEMA` to avoid confusion
Alexey Kudinkin
committed
Strip all meta-fields in `HoodieIncrSource` to make sure these are not propagated as data columns;
Alexey Kudinkin
committed