Core: Add a util to read and write partition stats #9170
Conversation
ajantha-bhat
requested review from
RussellSpitzer,
aokolnychyi,
flyrain and
nastra
ajantha-bhat
force-pushed
the
p_util_new
branch
from
6132093 to
f507698
Compare
ajantha-bhat
force-pushed
the
p_util_new
branch
from
f507698 to
899910d
Compare
| import org.apache.iceberg.types.Types; | ||
|
|
||
| public class PartitionEntry implements IndexedRecord { | ||
| private PartitionData partitionData; |
There was a problem hiding this comment.
Fields are based on the spec.
https://github.com/apache/iceberg/blob/main/format/spec.md#partition-statistics-file
| return new PartitionEntry(); | ||
| } | ||
|
|
||
| public PartitionEntry build() { |
There was a problem hiding this comment.
using the builder instead of Immutables because these objects will be mutable in a partition stats map. Will be adding the update functions during impl.
| throw new IllegalArgumentException("getting schema for an unpartitioned table"); | ||
| } | ||
|
|
||
| return new org.apache.iceberg.Schema( |
There was a problem hiding this comment.
Fields are based on the spec.
https://github.com/apache/iceberg/blob/main/format/spec.md#partition-statistics-file
There was a problem hiding this comment.
Note: Even the optional and required fields decision is based on the spec.
| import org.apache.iceberg.avro.AvroSchemaUtil; | ||
| import org.apache.iceberg.types.Types; | ||
|
|
||
| public class PartitionEntry implements IndexedRecord { |
There was a problem hiding this comment.
Extends IndexedRecord to use the existing Parquet/ORC Avro reader and writer from the iceberg-parquet and iceberg-orc module.
| } | ||
|
|
||
| private static void validateFormat(String filePath) { | ||
| if (!filePath.toLowerCase().endsWith(PARQUET_SUFFIX)) { |
There was a problem hiding this comment.
Other formats will be supported in the follow up PR
|
@aokolnychyi: Can this PR be reviewed? I know, I need to rework or analyze about the final spark action to collect the partition stats. So, Please take a look. |
|
closing this in favour of #10176 which has an end to end solution. |
1 participant
PartitionEntryto hold the entries of partition statsiceberg-datamodule.Engines will use these generic writers and readers.
TODO: Support ORC and Avro format as partition stats format in the follow up PR.
Fixes: #8455, #8456