[HUDI-4081][HUDI-4472] Addressing Spark SQL vs Spark DS performance gap#6213

Merged

xushiyan merged 26 commits intoapache:masterfrom

onehouseinc:ak/spk-sql-fix

Jul 28, 2022

Commits on Jul 28, 2022

Tidying up
Alexey Kudinkin
committed
Avoid unnecessary RDD dereferencing of the `Dataset`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up query ouput alignment seq to do proper validations and avoid unnecessary conversions;
Alexey Kudinkin
committed
Relaxed ordering requirement
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed validating sequence to properly assert whether query output conforms to the expected table's schema
Alexey Kudinkin
committed
Fixed partition-spec assertion;
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed invalid ref
Alexey Kudinkin
committed
Duct-tape the issue of incorrect schema handling in `HoodieSparkSqlWriter`
Alexey Kudinkin
committed
Revert back to relative-order based mathing (no name lookup)
Alexey Kudinkin
committed
Simplify scehma reconciliation, schema evolution handling
Alexey Kudinkin
committed
Properly reconcile nullability attributes
Alexey Kudinkin
committed
Rebased `InsertIntoHoodieTableCommand` to rely on Spark's `TableSchemaResolver` instead of bespoke implementation
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Extracted query output resolving/reshaping into `HoodieCatalystPlanUtils`
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Added new method `TableSchemaResolver#getTableLatestAvroSchema` to return most recent table's schema;
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Refactored schema handling in `HoodieSparkSqlWriter` to make sure all cases are handled correctly:
Alexey Kudinkin
committed
Reverting changes in `TableSchemaResolver`
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixed more tests (for Spark 2)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-4081][HUDI-4472] Addressing Spark SQL vs Spark DS performance gap#6213

[HUDI-4081][HUDI-4472] Addressing Spark SQL vs Spark DS performance gap#6213
xushiyan merged 26 commits intoapache:masterfrom
onehouseinc:ak/spk-sql-fix

Commits on Jul 28, 2022

Tidying up

Avoid unnecessary RDD dereferencing of the `Dataset`

Tidying up

Cleaned up query ouput alignment seq to do proper validations and avoid unnecessary conversions;

Relaxed ordering requirement

Fixing compilation

Fixed validating sequence to properly assert whether query output conforms to the expected table's schema

Fixed partition-spec assertion;

Fixing tests

Fixed invalid ref

Duct-tape the issue of incorrect schema handling in `HoodieSparkSqlWriter`

Revert back to relative-order based mathing (no name lookup)

Simplify scehma reconciliation, schema evolution handling

Properly reconcile nullability attributes

Rebased `InsertIntoHoodieTableCommand` to rely on Spark's `TableSchemaResolver` instead of bespoke implementation

Fixed tests

Extracted query output resolving/reshaping into `HoodieCatalystPlanUtils`

Fixed tests

Added new method `TableSchemaResolver#getTableLatestAvroSchema` to return most recent table's schema;

Fixing tests

Tidying up

Refactored schema handling in `HoodieSparkSqlWriter` to make sure all cases are handled correctly:

Reverting changes in `TableSchemaResolver`

Fixed tests

Fixed more tests (for Spark 2)

Tidying up

[HUDI-4081][HUDI-4472] Addressing Spark SQL vs Spark DS performance gap#6213xushiyan merged 26 commits intoapache:masterapache/hudi:masterfrom onehouseinc:ak/spk-sql-fixonehouseinc/hudi:ak/spk-sql-fixCopy head branch name to clipboard

Commits

Commits on Jul 28, 2022

[HUDI-4081][HUDI-4472] Addressing Spark SQL vs Spark DS performance gap#6213
xushiyan merged 26 commits intoapache:masterfrom
onehouseinc:ak/spk-sql-fix