Implement metadata_delete_after_commit_enabled and metadata_previous_versions_max for Iceberg table.

[oneonestar]

Description

Implement metadata_delete_after_commit_enabled and metadata_previous_versions_max for Iceberg table.
Without this feature, old metadata files (XXX.metadata.json files) will remains on the file system indefinitely after commit.

This corresponds to write.metadata.delete-after-commit.enabled and write.metadata.previous-versions-max properties (see: https://iceberg.apache.org/docs/latest/maintenance/#remove-old-metadata-files)

The following has been implemented:

CREATE TABLE foo (a bigint) WITH (
    metadata_delete_after_commit_enabled = true,
    metadata_previous_versions_max = 10);

ALTER TABLE foo SET PROPERTIES metadata_delete_after_commit_enabled = true;
ALTER TABLE foo SET PROPERTIES metadata_previous_versions_max = 10;

This PR depends on #20410 which implemented $metadata_log_entries system table for Iceberg table. This system table provides visibility on metadata file. Some testing codes are using $metadata_log_entries.
First 2 commits are from #20410. Only the last commit is for this PR.

Fix #19582
Fix #14128
Supersede #20011

Implementation of metadata_previous_versions_max

metadata_previous_versions_max is implemented by Iceberg's TableMetadata which Trino is using for commit. Trino only needs to set Iceberg table's write.metadata.previous-versions-max property for this to work.

Below is the call stack of an INSERT into Iceberg table.

addPreviousFile:1638, TableMetadata$Builder (org.apache.iceberg)
build:1417, TableMetadata$Builder (org.apache.iceberg)
internalApply:70, SetStatistics (org.apache.iceberg)
commit:56, SetStatistics (org.apache.iceberg)
finishInsert:1163, IcebergMetadata (io.trino.plugin.iceberg)
finishCreateTable:1000, IcebergMetadata (io.trino.plugin.iceberg)
finishCreateTable:585, ClassLoaderSafeConnectorMetadata (io.trino.plugin.base.classloader)
finishCreateTable:658, TracingConnectorMetadata (io.trino.tracing)
finishCreateTable:1129, MetadataManager (io.trino.metadata)
finishCreateTable:610, TracingMetadata (io.trino.tracing)
lambda$createTableFinisher$4:4155, LocalExecutionPlanner (io.trino.sql.planner)
finishTable:-1, LocalExecutionPlanner$$Lambda/0x00007f55b0a068e0 (io.trino.sql.planner)
getOutput:319, TableFinishOperator (io.trino.operator)

Implementation of metadata_delete_after_commit_enabled

Trino is doing TableOperations by itself. The required logic has been added to *TableOperations.
Reference implementation in Iceberg:
https://github.com/apache/iceberg/blob/6a3b2d7c153412b01c746debb018c544516f2bbd/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L412-L440

Release notes

(x) Release notes are required, with the following suggested text:

# Section
* Support the Iceberg table properties `metadata_delete_after_commit_enabled` and `metadata_previous_versions_max` ({issue}`issuenumber`)

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[mosabua]

Can you also review potentially @amogh-jahagirdar

[freerangerooster]

We need this feature badly. Can't really use trino for iceberg for anything serious if we don't have a way to remove old metadata files. In our test case, the data file is around 100KB but the metadata files in total are over 200MB. We tried expire_snapshots & remove_orphan_files (retention_threshold set to 1d). No luck, the old metadata files refuse to go.

[saxcorp]

We also need this feature. Tx for the implementation.:)

[mosabua]

Can you rebase @oneonestar

And can @alexjo2144 @cwsteinbach @ebyhr and others help with review?