Core: Refactor and use InternalData for partition stats #12946
Conversation
| * Computes, writes and reads the {@link PartitionStatisticsFile}. Uses generic readers and writers | ||
| * to support writing and reading of the stats in table default format. | ||
| */ | ||
| public class PartitionStatsHandler { |
There was a problem hiding this comment.
This class was just copied from iceberg-data module (deprecated now) and replace datawriter and reader function with InternalData
| return stats; | ||
| } | ||
|
|
||
| private static Collection<PartitionStats> computeStats(Table table, Snapshot snapshot) { |
There was a problem hiding this comment.
Copied the methods from partitionStatsUtil (perviously public, now deprecated) as got a comment that no need to keep too many public interfaces and compute logic can be internal to compute and write.
|
hi @ajantha-bhat, do you think it would make sense to rename |
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java
Outdated
Show resolved
Hide resolved
@deniskuzZ: Thanks for the suggestion. Since we already released the public interface as |
There was a problem hiding this comment.
LGTM +1
thanks @pvary for the proposal and @ajantha-bhat for refactoring and merging the partition stats functionality. Looking forward for the increment partition stats compute support.
data/src/test/java/org/apache/iceberg/TestPartitionStatsHandler.java
Outdated
Show resolved
Hide resolved
|
@pvary: PR is ready for review. |
advancedxy
left a comment
advancedxy
left a comment
There was a problem hiding this comment.
LGTM on my side, looking forward to the incremental support.
core/src/test/java/org/apache/iceberg/PartitionStatsHandlerBase.java
Outdated
Show resolved
Hide resolved
core/src/test/java/org/apache/iceberg/PartitionStatsHandlerBase.java
Outdated
Show resolved
Hide resolved
core/src/test/java/org/apache/iceberg/PartitionStatsHandlerTestBase.java
Outdated
Show resolved
Hide resolved
|
Rebased the PR due to spark flaky test. |
|
@pvary and @gaborkaszab: Anymore comments for this? Thanks for the review. |
gaborkaszab
left a comment
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks for the changes @ajantha-bhat !
I left one minor comment, but other than that the PR LGTM!
core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java
Outdated
Show resolved
Hide resolved
|
Merged to main. |
apache#1581) Co-authored-by: Ajantha Bhat <[email protected]>
5 participants
a) Deprecate
PartitionStatsHandleriniceberg-datab) Copy
PartitionStatsHandlerfromiceberg-datatoiceberg-coreand UseInternalData( also package name had to change to usePartitionStatsUtilllogic as private code). So, users can add dependency oncoreandparquetinstead of wholedatamodule to use partition stats feature.c) Deprecate
PartitionStatsUtilas no need of too many public interface and we can use that logic as private insidePartitionStatsHandler(comment during #12629)d) Deprecate
TestPartitionStatsUtilas the functionality test is covered fromTestPartitionStatsHandlere) Move benchmarks to
iceberg-dataas InternalData needs parquet dependency.