Improve Iceberg partition predicate enforcement

[alexjo2144]

Description

Only partition specs which are used by the Snapshot being queried are relevant when determining if partitions can be used to enforce a Predicate.

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Iceberg connector

How would you describe this change to a non-technical end user or system administrator?

Improve filtering on partition columns in Iceberg

Related issues, pull requests, and links

Relates to: #12795

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

[erichwang]

seems reasonable to me

[erichwang]

call me ignorant, but under what conditions would table.getSnapshotId() be empty?

[alexjo2144]

Good question. It's only ever empty during new table creation

[findepi]

add a message so that -- if it throws -- we know what assumption was violated

snapshotId = table.getSnapshotId().orElseThrow(() -> new IllegalStateException("no snapshot id"))

[findepi]

LGTM expect for the call to allManifests().

[findepi]

add a message so that -- if it throws -- we know what assumption was violated

snapshotId = table.getSnapshotId().orElseThrow(() -> new IllegalStateException("no snapshot id"))

[findepi]

i think calling allManifests can be expensive, right?

can we prune them out using the predicates we already know + the new ones?

[findepi]

cc @danielcweeks @rdblue

[alexjo2144]

Getting this list doesn't require reading all of the individual Manifests, just the ManifestList, so it's one Avro file to read

[alexjo2144]

@findepi AC, thanks. Had to make one more small change to TestIcebergProjectionPushdownPlans. Since we're now enforcing all constraints on empty tables I had to add data to the table so that we can test enforced and unenforced constraints.

[findepi]

why this change?

[alexjo2144]

There's an implicit change here that we can now enforce all predicates on empty tables. That broke this test because the table is empty but was not expecting a predicate to be enforced.

[homar]

But doesn't it mean that now iceberg will fail on valid sql ?

[alexjo2144]

Spoke with Konrad offline about this, not a problem

[findepi]

please squash

[alexjo2144]

Squashed