[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers by alexeykudinkin · Pull Request #6815 · apache/hudi

Updated all usages to reference `HoodieSparkKryoRegistrar`

Alexey Kudinkin

committed

Fixing MT table to be always created in a _reusable_ state by default (ie where it's not flushing its state after every request and instead keeping the caches until it's completely disbanded)

Alexey Kudinkin

committed

Inlined "reuse" flag to be true by default for `HoodieBackedTableMetadata`

Alexey Kudinkin

committed

Added `TransientLazy` utility (`Serializable` counterpart of `Lazy`)

Alexey Kudinkin

committed

Rebased `HoodieTable` to rely on `TransientLazy` to manage lifecycle of non-serializable fields (`ViewManager`, `HoodieTableMetadata`);

Alexey Kudinkin

committed

Made `SparkRDDReadClient` closeable;

Alexey Kudinkin

committed

Fixing compilation

Alexey Kudinkin

committed

Fixing `TransientLazy` to disallow eager init

Alexey Kudinkin

committed

Fixing compilation after rebase

Alexey Kudinkin

committed

Fixed `HoodieTable` to be broadcasted to make sure we have single object per-executor, instead of having one per-task

Alexey Kudinkin

committed

Revisited `Transient` to allow eager initialization

Alexey Kudinkin

committed

Fixed `HoodieTable` to use proper context

Alexey Kudinkin

committed

Revisited `HoodieTable` to hold

Alexey Kudinkin

committed

Make `HoodieBackedTableMetadata` serializable

Alexey Kudinkin

committed

Added `LeakTrackingFSDataInputStream` to track "leaked" (non-closed) streams potentially holding on to a S3 connections

Alexey Kudinkin

committed

Fixing compilation

Alexey Kudinkin

committed

Reduced MT buffer size from 10Mb to 10Kb

Alexey Kudinkin

committed

Fixed `HoodieTable` close to re-throw unchecked

Alexey Kudinkin

committed

Avoid wiring `HoodieSparkSessionExtension` in "hudi-spark-client"

Alexey Kudinkin

committed

Killed superfluous Hadoop's `Configuration`

Alexey Kudinkin

committed

Reverted `HoodieTableFileSystemView` to be serializable in `HoodieBackedTableMetadata`

Alexey Kudinkin

committed

Make sure `ByteBuffer` is being duplicated prior to being read out

Alexey Kudinkin

committed

Added `destroy` method to `Transient` allowing to free up any references held, as well as destroy held objects

Alexey Kudinkin

committed

Revisited `HoodieSparkTable` to hold `Transient` relying on `Broadcast` for ser/de to make sure

Alexey Kudinkin

committed

Avoid holding additional `SerializableConfiguration` refs

Alexey Kudinkin

committed

Reverted `HoodieTable` to hold `HoodieTableMetadata` directly instead of holding the `Broadcast`

Alexey Kudinkin

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers#6815

[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers#6815
alexeykudinkin wants to merge 26 commits intoapache:masterfrom
onehouseinc:ak/mt-repars-fix

Commits on Jan 24, 2023

Updated all usages to reference `HoodieSparkKryoRegistrar`

Fixing MT table to be always created in a _reusable_ state by default (ie where it's not flushing its state after every request and instead keeping the caches until it's completely disbanded)

Inlined "reuse" flag to be true by default for `HoodieBackedTableMetadata`

Added `TransientLazy` utility (`Serializable` counterpart of `Lazy`)

Rebased `HoodieTable` to rely on `TransientLazy` to manage lifecycle of non-serializable fields (`ViewManager`, `HoodieTableMetadata`);

Made `SparkRDDReadClient` closeable;

Fixing compilation

Fixing `TransientLazy` to disallow eager init

Fixing compilation after rebase

Fixed `HoodieTable` to be broadcasted to make sure we have single object per-executor, instead of having one per-task

Revisited `Transient` to allow eager initialization

Fixed `HoodieTable` to use proper context

Revisited `HoodieTable` to hold

Make `HoodieBackedTableMetadata` serializable

Added `LeakTrackingFSDataInputStream` to track "leaked" (non-closed) streams potentially holding on to a S3 connections

Fixing compilation

Reduced MT buffer size from 10Mb to 10Kb

Fixed `HoodieTable` close to re-throw unchecked

Avoid wiring `HoodieSparkSessionExtension` in "hudi-spark-client"

Killed superfluous Hadoop's `Configuration`

Reverted `HoodieTableFileSystemView` to be serializable in `HoodieBackedTableMetadata`

Make sure `ByteBuffer` is being duplicated prior to being read out

Added `destroy` method to `Transient` allowing to free up any references held, as well as destroy held objects

Revisited `HoodieSparkTable` to hold `Transient` relying on `Broadcast` for ser/de to make sure

Avoid holding additional `SerializableConfiguration` refs

Reverted `HoodieTable` to hold `HoodieTableMetadata` directly instead of holding the `Broadcast`