Iceberg DML operation timeout avoids corrupting the table

[findinpath]

Description

In case of dealing with an CommitFailedException while
commiting an Iceberg transaction, the Iceberg framework
will attempt to retry for COMMIT_NUM_RETRIES times the
operation and if the operation still fails, it will clean up
the metadata file corresponding to the transaction.
In case of a metastore client timeout operation the
Iceberg library can therefore delete metadata files
which eventually get referenced from the configuration
of the table persisted on the metastore for the table
which leaves the table in a corrupt state.

Throw CommitStateUnknownException to ensure that the
table is not left in a corrupt state after the erroneous
completion of the DML operation.

Fixes #14104
probably fixes #12581 too

Non-technical explanation

Avoid deleting metadata files in case of a DML operation timeout because these filesmay eventually get referenced from the Hive metastore configuration of the Iceberg table.

Release notes

( ) This is not user-visible and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Iceberg DML operation timeout avoids corrupting the table

[electrum]

Do you plan to squash the first commit?

[findinpath]

Do you plan to squash the first commit?

The added value of the first commit is only for maintenance reasons. Squashing the commits would leave a naive reader of the code without any clear context related to the problem being addressed in the commit.
I'm fine either way. I wanted to have a test at hand that showcases the issue and not blindly add a fix.

[findepi]

remove try, catch.
createQueryRunner() can throw any Exception

[findepi]

bump

[alexjo2144]

Thanks Marius!

[findepi]

@findinpath under what circumstances should the GlueIcebergTableOperations throw CommitStateUnknownException?

[findinpath]

under what circumstances should the GlueIcebergTableOperations throw CommitStateUnknownException?

com.amazonaws.services.glue.AWSGlue#updateTable

From the javadoc

     * @throws ConcurrentModificationException
     *         Two processes are trying to modify a resource simultaneously.

[findepi]

That indicates that ConcurrentModificationException handling is (probably) right.

Under what circumstances should we throw CommitStateUnknownException in GlueIcebergTableOperations?

[findinpath]

Good, I see where you are probably hinting. I've been too focused on the HMS part and missed the fact that Glue is also affected by the same issue.

Here is the list of exceptions thrown by updateTable operation:

EntityNotFoundException – A specified entity does not exist
InvalidInputException – The input provided was not valid.
InternalServiceException – An internal service error occurred.
OperationTimeoutException – The operation timed out.
ConcurrentModificationException – Two processes are trying to modify a resource simultaneously.
ResourceNumberLimitExceededException – A resource numerical limit was exceeded.
GlueEncryptionException – An encryption operation failed.

Except ConcurrentModificationException, ResourceNumberLimitExceededException (should be happening before hitting the internal metastore service so no risk of corruption here) , InvalidInputException (not very sure here if there may happen that the processing of the input is not half-way through before throwing the exception) and EntityNotFoundException (quite unlikely to be thrown in our specific case) I'd opt to throw CommitStateUnknownException.

So ResourceNumberLimitExceededException & ConcurrentModificationException ask for a retry -> let's throw a CommitFailedException. I'll make a follow-up PR on this topic.