[Iceberg] Refine the partition specs that really need to be checked

[hantangwangd]

Description

Iceberg table may contains empty partition specs with data all be deleted or rewritten to another partition spec, so that it may contains some manifest files with spec id of these partition specs just to record the information about data files deletion. If we don't filter them out during judgement, we might in some cases prohibit metadata deletion or predicate thoroughly pushdown, even if it's feasible.

This PR refine the partition specs that really need to be checked in an Iceberg table when determining whether supports metadata deletion or predicate thoroughly pushdown.

Test Plan

• Test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with empty unsupported partition spec
• Test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with unsupported partition spec whose data files are all deleted
• To do: test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with multiple partition specs after whole table's data files rewriting (optimization)

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

== RELEASE NOTES ==


Iceberg Connector Changes
* Improve the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

[steveburnett]

Thanks for the release note entry! A nit rephrasing suggestion to consider:

== RELEASE NOTES ==


Iceberg Connector Changes
* Refine the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

[hantangwangd]

@steveburnett Thanks for the suggestion, fixed!

[steveburnett]

A final nit: I'm sorry but I didn't think about the Order of changes recommendations in the Release Notes Guidelines when I commented. Here's another revision suggestion, apologies for the churn:

== RELEASE NOTES ==


Iceberg Connector Changes
* Improve the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

[hantangwangd]

@steveburnett So grateful for your help in standardizing the release notes, fixed! I also need to learn the Guidelines :-).

[ZacBlanco]

I think the main change is fine. I do think we should probably add tests for the different table format versions+modes though to verify the expected behavior.

[hantangwangd]

@ZacBlanco, to add test case for V2 MOR table, I found that previously we didn't take delete files into consider when executing metadata-deletion/truncate. So I added a commit at the front to fix the behavior of metadata deletion and table truncate, then add the test cases we mentioned above for V2 MOR table. Please take a look when convenient, thanks!

[ZacBlanco]

Two questions regarding the new change

[tdcmeehan]

Suggested change

	// Create a table with on partition
	// Create a table with no partition

[hantangwangd]

Hi @tdcmeehan, I just special handled some test cases in TestIcebergSmokeRest because of the known Iceberg issue 8756 mentioned by @kiersten-stokes, or else they would fail. Please take a look again when available, thanks!

[tdcmeehan]

Thank you @hantangwangd