HIVE-28581: Support Partition Pruning stats optimizations for Iceberg tables

[deniskuzZ]

What changes were proposed in this pull request?

Adds support for Iceberg partition prune stats optimizations.

1. Adds functionality to compute, store and fetch basic partition stats per snapshot;
2. Enhances query plan with basic stats, aggregated for pruned partitions;
3. Enables PartitionConditionRemover (PCR) optimization;
4. Improves StatsOptimizer, allows to answers queries like count(*) with partition predicate directly from stats;

Why are the changes needed?

Performance

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

mvn test -Dtest=TestIcebergCliDriver -Dqfile=iceberg_stats_with_ppr.q -Drat.skip=true

[deniskuzZ]

Can we reuse this object or better create another abstraction, like [Virtual/Hidden]Partition?
cc @kasakrisz

[kasakrisz]

Is the functionality of DummyPartition exploited? I saw that when we create DummyPartition instances we return a Partition type reference and never access any DummyPartition defined method. If this is not the case then DummyPartition is fine. The name is misleading though since these objects represent real partitions, aren't they?

[deniskuzZ]

we use overriden getValues and getSpec. And yes, they represent real partitions (not in HMS)

[kasakrisz]

Seems that the current class hierarchy doesn't represent our needs:

• Partition class represents a partition stored on HMS.
• DummyPartition extends Partition so it seems like a special HMS stored partition but this is not the case.

Maybe defining a Partition interface or abstract class with a minimal contract (getName, getValues) would be better. And two class could implement/extend it: one for HMS stored partition and another for non-HMS.

It seems to be a bigger refactor because the current Partition class is widely used.

[deniskuzZ]

DummyPartition is used to represent non-HMS partitions. Before iceberg, it had just two properties: table and name. But I agree the name is not self-explanatory

[sonarqubecloud]

Quality Gate passed

Issues
111 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ayushtkn]

LGTM