[HUDI-4420] Fixing table schema delineation on partition/data schema for Spark relations #5708

Merged

xushiyan merged 24 commits intoapache:masterfrom

onehouseinc:ak/spk-rel-fix

Jul 23, 2022

Commits on Jul 22, 2022

Removed unnessary Catalyst projection
Alexey Kudinkin
committed
Cleaning up naming for Catalyst expression utils
Alexey Kudinkin
committed
Employ Catalyst's `UnsafeProjection` when projecting records w/o updates
Alexey Kudinkin
committed
Consolidated Avro records projections w/in `SafeAvroProjection`
Alexey Kudinkin
committed
Do unsafe Avro projection after merging to avoid `AvroSafeProjection` init costs
Alexey Kudinkin
committed
Fixed handling of partition schema extraction for both COW/MOR relations by:
Alexey Kudinkin
committed
Fixed `BaseFileOnlyRelation`
Alexey Kudinkin
committed
Fixed `MergeOnReadSnapshotRelation`
Alexey Kudinkin
committed
Abstracted base-file readers creation for MOR relation behind a single method
Alexey Kudinkin
committed
Rebased `MergeOnReadIncrementalRelation` onto new `createBaseFileReaders` method
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Typo
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Converted `BaseFileReader` hierarchy to be case-classes (for serializability)
Alexey Kudinkin
committed
Fixed base-file reader creation seq to properly project into the required schema
Alexey Kudinkin
committed
Delineate clearly data-file schema
Alexey Kudinkin
committed
Fixed `SkipMergeIterator` to properly project into required schema
Alexey Kudinkin
committed
Fixed MOR relations to delineate schemas w/in `createBaseFileReaders`
Alexey Kudinkin
committed
Revisited projecting into required-schema to only occur w/in the RDD themselves
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Reverting accidental change
Alexey Kudinkin
committed
Fixing HFile reader creation to match other file-formats;
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-4420] Fixing table schema delineation on partition/data schema for Spark relations #5708

[HUDI-4420] Fixing table schema delineation on partition/data schema for Spark relations #5708
xushiyan merged 24 commits intoapache:masterfrom
onehouseinc:ak/spk-rel-fix

Commits on Jul 22, 2022

Removed unnessary Catalyst projection

Cleaning up naming for Catalyst expression utils

Employ Catalyst's `UnsafeProjection` when projecting records w/o updates

Consolidated Avro records projections w/in `SafeAvroProjection`

Do unsafe Avro projection after merging to avoid `AvroSafeProjection` init costs

Fixed handling of partition schema extraction for both COW/MOR relations by:

Fixed `BaseFileOnlyRelation`

Fixed `MergeOnReadSnapshotRelation`

Abstracted base-file readers creation for MOR relation behind a single method

Rebased `MergeOnReadIncrementalRelation` onto new `createBaseFileReaders` method

Fixing compilation

Fixing compilation

Typo

Fixing compilation

Converted `BaseFileReader` hierarchy to be case-classes (for serializability)

Fixed base-file reader creation seq to properly project into the required schema

Delineate clearly data-file schema

Fixed `SkipMergeIterator` to properly project into required schema

Fixed MOR relations to delineate schemas w/in `createBaseFileReaders`

Revisited projecting into required-schema to only occur w/in the RDD themselves

Tidying up

Tidying up

Reverting accidental change

Fixing HFile reader creation to match other file-formats;

[HUDI-4420] Fixing table schema delineation on partition/data schema for Spark relations #5708xushiyan merged 24 commits intoapache:masterapache/hudi:masterfrom onehouseinc:ak/spk-rel-fixonehouseinc/hudi:ak/spk-rel-fixCopy head branch name to clipboard

Commits

Commits on Jul 22, 2022

[HUDI-4420] Fixing table schema delineation on partition/data schema for Spark relations #5708
xushiyan merged 24 commits intoapache:masterfrom
onehouseinc:ak/spk-rel-fix