Make Iceberg materialized view refresh atomic

[alexjo2144]

Description

Include old file deletion and new file appends in the same transaction.

Non-technical explanation

Avoids a short time period where the MV is in an invalid state.

Release notes

( ) This is not user-visible and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Improve the commit protocol used when refreshing a materialized view.

[alexjo2144]

@raunaqmorarka I wasn't totally sure about this alwaysTrue part. Do we always rewrite the entire content of a MV?

[findepi]

I think incremental refreshes are supported only if io.trino.spi.connector.ConnectorMetadata#delegateMaterializedViewRefreshToConnector returns true (it returns false for Iceberg), and connector knows how to refresh incrementally.

BTW @alexjo2144 will this unregister all old data files, or will it scan any data file contents?

Also, does this preserve table version history correctly?

[alexjo2144]

BTW @alexjo2144 will this unregister all old data files, or will it scan any data file contents?

This will unregister the data files from the manifest and they will not be scanned. It is the same code as is run by executeDelete just attached to the transaction instead of being run separately.

Also, does this preserve table version history correctly?

If we need history to be retained we should upgrade to 0.14.1 first, there's a bug fix in that we'd need. However, I was assuming history for an MV is not useful

[findepi]

If we need history to be retained we should upgrade to 0.14.1 first, there's a bug fix in that we'd need

yeah, just realized it's coming now: #14074 (comment)

However, I was assuming history for an MV is not useful

perhaps; i am fine merging as is, without 0.14.1

[findepi]

@alexjo2144 mind CI

[findepi]

( just squashed )