[HUDI-5443] Fixing exception trying to read MOR table after `NestedSchemaPruning` rule has been applied#7528

Merged

yihua merged 26 commits intoapache:masterfrom

onehouseinc:ak/mor-sch-prj-fix

Jan 24, 2023

Commits on Jan 21, 2023

Removing unnecessary wrappers
Alexey Kudinkin
committed
Tidyng up
Alexey Kudinkin
committed
Tidying up `HoodieMergeOnReadRDD`
Alexey Kudinkin
committed
Deduplicate file-slice listing in Spark's relations
Alexey Kudinkin
committed
Rebased MOR iterators onto lazy projection that avoids projecting when it's a no-op
Alexey Kudinkin
committed
Converted all Hudi's Spark relations into case classes;
Alexey Kudinkin
committed
Abstracted `BaseMergeOnReadSnapshotRelation` to be able to inherit from it for both Snapshot and Incremental relations;
Alexey Kudinkin
committed
Fixed typo
Alexey Kudinkin
committed
Make sure Hudi relation's schema pruned only once
Alexey Kudinkin
committed
Replaced `SafeAvroProjection` w/ simplified `AvroProjection`
Alexey Kudinkin
committed
XXX
Alexey Kudinkin
committed
Fixing compilation for Spark 2.x
Alexey Kudinkin
committed
Re-enable `NestedSchemaPruning` rule
Alexey Kudinkin
committed
Abstracted `isProjectionCompatible` utility to control whether `NestedSchemaPruning` rule could be applied to MOR table;
Alexey Kudinkin
committed
Reverting unnecessary changes;
Alexey Kudinkin
committed
Fixing NPE
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Added negative tests for `NestedSchemaPruning`
Alexey Kudinkin
committed
Fixing test
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up tests
Alexey Kudinkin
committed
Combined `generateUnsafeProjection` and `generateLazyProjection` to implement lazy semantic for every projection
Alexey Kudinkin
committed
Combined `create` and `createLazy` for `AvroProjection`s as well
Alexey Kudinkin
committed

Commits on Jan 23, 2023

Fixed tests
Alexey Kudinkin
committed

Commits on Jan 24, 2023

Fixed nullability annotations
Alexey Kudinkin
committed
Restoring `CONFIG_INSTANTIATION_LOCK`
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-5443] Fixing exception trying to read MOR table after `NestedSchemaPruning` rule has been applied#7528

[HUDI-5443] Fixing exception trying to read MOR table after `NestedSchemaPruning` rule has been applied#7528
yihua merged 26 commits intoapache:masterfrom
onehouseinc:ak/mor-sch-prj-fix

Commits on Jan 21, 2023

Removing unnecessary wrappers

Tidyng up

Tidying up `HoodieMergeOnReadRDD`

Deduplicate file-slice listing in Spark's relations

Rebased MOR iterators onto lazy projection that avoids projecting when it's a no-op

Converted all Hudi's Spark relations into case classes;

Abstracted `BaseMergeOnReadSnapshotRelation` to be able to inherit from it for both Snapshot and Incremental relations;

Fixed typo

Make sure Hudi relation's schema pruned only once

Replaced `SafeAvroProjection` w/ simplified `AvroProjection`

XXX

Fixing compilation for Spark 2.x

Re-enable `NestedSchemaPruning` rule

Abstracted `isProjectionCompatible` utility to control whether `NestedSchemaPruning` rule could be applied to MOR table;

Reverting unnecessary changes;

Fixing NPE

Fixing compilation

Added negative tests for `NestedSchemaPruning`

Fixing test

Tidying up

Tidying up tests

Combined `generateUnsafeProjection` and `generateLazyProjection` to implement lazy semantic for every projection

Combined `create` and `createLazy` for `AvroProjection`s as well

Commits on Jan 23, 2023

Fixed tests

Commits on Jan 24, 2023

Fixed nullability annotations

Restoring `CONFIG_INSTANTIATION_LOCK`