[HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS#4739

Merged

nsivabalan merged 27 commits intoapache:masterfrom

onehouseinc:ak/hdfs-appnd-fix

Mar 7, 2022

Commits on Mar 4, 2022

Refactored MT records composition from `HoodieCommitMetadata` to always operate on full file-sizes
Alexey Kudinkin
committed
Fixed `HoodieMetadataPayload` to properly combine file-sizes
Alexey Kudinkin
committed
Added TODO
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed tests' incorrect `WriteStat` generation
Alexey Kudinkin
committed
Streamlined FS Metadata merging logic (to simply always prefer more recent records)
Alexey Kudinkin
committed
Added validations
Alexey Kudinkin
committed
Fixed validation seq to only validate file listing records
Alexey Kudinkin
committed
Fix `HoodieMetadataPayload` sequence to appropriately handle tombstones
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Bubbled the assertion for positive file-sizes to be conducted upon MT records creation;
Alexey Kudinkin
committed
Fixing Flink tests
Alexey Kudinkin
committed
Fixed `HoodieMetadataPayload` operation to become commutative
Alexey Kudinkin
committed
Fixed compilation
Alexey Kudinkin
committed
Consolidated table init-seq w/in `BaseHoodieWriteClient` to avoid duplication across engines
Alexey Kudinkin
committed
Added `JavaUpgradeDowngradHelper`;
Alexey Kudinkin
committed
Fixed contextual timer setting;
Alexey Kudinkin
committed
Make sure every `WriteClient` operation initializes Hudi Table properly
Alexey Kudinkin
committed
Fixing tests creating invalid base files
Alexey Kudinkin
committed
Revisited rollback related APIs to only invoke `initTable` seq for public facing API;
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Removed table init seq from `rollbackFailedWrite`
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Cleaning up
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS#4739

[HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS#4739
nsivabalan merged 27 commits intoapache:masterfrom
onehouseinc:ak/hdfs-appnd-fix

Commits on Mar 4, 2022

Refactored MT records composition from `HoodieCommitMetadata` to always operate on full file-sizes

Fixed `HoodieMetadataPayload` to properly combine file-sizes

Added TODO

Tidying up

Fixed tests' incorrect `WriteStat` generation

Streamlined FS Metadata merging logic (to simply always prefer more recent records)

Added validations

Fixed validation seq to only validate file listing records

Fix `HoodieMetadataPayload` sequence to appropriately handle tombstones

Tidying up

Bubbled the assertion for positive file-sizes to be conducted upon MT records creation;

Fixing Flink tests

Fixed `HoodieMetadataPayload` operation to become commutative

Fixed compilation

Consolidated table init-seq w/in `BaseHoodieWriteClient` to avoid duplication across engines

Added `JavaUpgradeDowngradHelper`;

Fixed contextual timer setting;

Make sure every `WriteClient` operation initializes Hudi Table properly

Fixing tests creating invalid base files

Revisited rollback related APIs to only invoke `initTable` seq for public facing API;

Fixed tests

Missing license

Fixing compilation

Removed table init seq from `rollbackFailedWrite`

Fixing tests

Cleaning up

Fixing compilation