[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737

Merged

yihua merged 27 commits intoapache:masterfrom

onehouseinc:ak/spk-ctlg-fix

Jun 7, 2022

Commits on Jun 6, 2022

Make sure `TestCOWDataSource` properly invokes `HoodieSparkSessionExtensions`
Alexey Kudinkin
committed
Make sure `HoodieBaseRelation` always prefers `schemaSpec` over fetching schema from the storage;
Alexey Kudinkin
committed
Fixing `HoodieSpark3Analysis` rule to properly pass in schema from the catalog
Alexey Kudinkin
committed
Fixed missing conversion
Alexey Kudinkin
committed
Added `LogicalPlan` caching for Hudi's relations
Alexey Kudinkin
committed
Properly override outputs of the cached `LogicalRelation`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make `HoodieCatalog` and `Spark3DefaultSource` return `V1Table` instead of `HoodieInternalV2Table` to make sure Spark treats Hudi tables as V1 and not V2
Alexey Kudinkin
committed
Added `HoodieV1Table` extractor to match Hudi tables;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fix missing fixture of the `DateTimeZone.default`
Alexey Kudinkin
committed
Reverting DSv2 APIs
Alexey Kudinkin
committed
Make Hudi resolve into either V1 or V2 relations based on whether Schema Evolution is enabled;
Alexey Kudinkin
committed
Restored resolution-rule to fallback from V2 Relation to V1
Alexey Kudinkin
committed
Rebased Schema Evolution commands resolution rule to use V2 table
Alexey Kudinkin
committed
Fixed schema evoluation rule injection seq
Alexey Kudinkin
committed
Make sure V2 to V1 fallback is properly wired
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed V2 to V1 fallback to also apply to Spark 3.1;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Reverting back invalid bifurcation b/w Spark 3.1 and 3.2
Alexey Kudinkin
committed
Make sure we're looking up configs appropriately
Alexey Kudinkin
committed
Moved `SCHEMA_EVOLUTION_ENABLED` config to `HoodieCommonConfig`
Alexey Kudinkin
committed

Commits on Jun 7, 2022

Fixed rules evaluation order
Alexey Kudinkin
committed
Inline Alter Table commands resolving rules to control ordering of their application
Alexey Kudinkin
committed
Avoid providing catalog schema in cases when Hive catalog is used
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737

[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737
yihua merged 27 commits intoapache:masterfrom
onehouseinc:ak/spk-ctlg-fix

Commits on Jun 6, 2022

Make sure `TestCOWDataSource` properly invokes `HoodieSparkSessionExtensions`

Make sure `HoodieBaseRelation` always prefers `schemaSpec` over fetching schema from the storage;

Fixing `HoodieSpark3Analysis` rule to properly pass in schema from the catalog

Fixed missing conversion

Added `LogicalPlan` caching for Hudi's relations

Properly override outputs of the cached `LogicalRelation`

Tidying up

Make `HoodieCatalog` and `Spark3DefaultSource` return `V1Table` instead of `HoodieInternalV2Table` to make sure Spark treats Hudi tables as V1 and not V2

Added `HoodieV1Table` extractor to match Hudi tables;

Fixing compilation

Fix missing fixture of the `DateTimeZone.default`

Reverting DSv2 APIs

Make Hudi resolve into either V1 or V2 relations based on whether Schema Evolution is enabled;

Restored resolution-rule to fallback from V2 Relation to V1

Rebased Schema Evolution commands resolution rule to use V2 table

Fixed schema evoluation rule injection seq

Make sure V2 to V1 fallback is properly wired

Tidying up

Tidying up

Fixed V2 to V1 fallback to also apply to Spark 3.1;

Fixing compilation

Reverting back invalid bifurcation b/w Spark 3.1 and 3.2

Make sure we're looking up configs appropriately

Moved `SCHEMA_EVOLUTION_ENABLED` config to `HoodieCommonConfig`

Commits on Jun 7, 2022

Fixed rules evaluation order

Inline Alter Table commands resolving rules to control ordering of their application

Avoid providing catalog schema in cases when Hive catalog is used

[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737yihua merged 27 commits intoapache:masterapache/hudi:masterfrom onehouseinc:ak/spk-ctlg-fixonehouseinc/hudi:ak/spk-ctlg-fixCopy head branch name to clipboard

Commits

Commits on Jun 6, 2022

Commits on Jun 7, 2022

[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737
yihua merged 27 commits intoapache:masterfrom
onehouseinc:ak/spk-ctlg-fix