Core: Write partition stats during write operation #8488
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Address new comments remove trailing whitespaces
- Since core module need to write stats in parquet format, to avoid circular dependency, move all the files from iceberg-parquet module to iceberg code. - `TestParquetReadProjection` used to duplicate the test code of iceberg-api module's `TestReadProjection`. Removed the duplicate class and instead directly extend the original class from iceberg-api module. - Update TestParquetReadProjection to skip empty struct testcases as only Avro readers supports it. The testcases are now common for both Avro and Parquet readers.
ajantha-bhat
force-pushed
the
wip_pstats
branch
3 times, most recently
from
c4c8c4b to
0695494
Compare
This was referenced Sep 5, 2023
ajantha-bhat
changed the title
PartitionsTable.Partition will be used between Partitions metadata table and partition stats reader-writer. Hence, move it to a separate class and extend it with Avro's IndexedRecord (for partition stats writing).
Tracking `PartitionStatisticsFile` in a same way as how `StatisticsFile` is already tracked.
ajantha-bhat
force-pushed
the
wip_pstats
branch
from
0695494 to
5517259
Compare
- Add interfaces to SnapshotProducer to write the partition stats (in Parquet format) - Store partition stats file location in snapshot summary - Use the `PartitionStatsUtil.java` from the core module to synchronously write the parquet partition stats for every snapshot produced. - Compute the partition stats based on the table property `write.partition.statistics`. Enabled by default until review to expose issues if exist from all partition table testcases. -Track these partition stats from TableMetadata and Snapshot by registering them during the write operation.
ajantha-bhat
force-pushed
the
wip_pstats
branch
from
5517259 to
ca6fd56
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add interfaces to
SnapshotProducerto write the partition stats (in Parquet format)Store partition stats file location in snapshot summary
Use the
PartitionStatsUtil.javafrom the core module to synchronouslywrite the parquet partition stats for every snapshot produced.
Compute the partition stats based on the table property
write.partition.statistics.Enabled by default until review to expose issues if exist from all partition table testcases.
Track these partition stats from TableMetadata and Snapshot
by registering them during the write operation.
Depends on PR #7105, #8500, #8501, #8502, #8503
Fixes #8458