Support sorted writes in the Iceberg connector

[alexjo2144]

Description

Support sorting files during inserts to the Iceberg connector. This reuses the SortingFileWriter from the Hive connector.

Non-technical explanation

Sorting enables better performance during selective read queries, where a small range of values is needed from a high carnality column.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Support for Iceberg table sort orders. Tables can be created or altered to add a list of `sorted_by` columns which will be used to order files written to the table.

[alexjo2144]

Note to self: rework the "Allow updating the sorted_by Iceberg table property" commit to give @osscm author credit.

[findinpath]

Add compatibility tests with spark.

[findinpath]

Sorting enables better performance during selective read queries, where a small range of values is needed from a high cardinality column.

I'd appreciate having a demo test that emphasises on the fact that the number of files being read is smaller when working with sorted_by. Source of inspiration: io.trino.plugin.iceberg.TestIcebergMetadataFileOperations

I don't see sortOrder in the IcebergTableHandle. Will eventually the sorted reads come as a follow-up PR?

[findinpath]

The assertions related to the columns were already made in io.trino.plugin.iceberg.catalog.BaseTrinoCatalogTest#testCreateTable . Are they relevant here as well?

[findinpath]

Pre-existing: It would be beneficial to have the purpose of this property documented (in the code).

[findinpath]

What do we showcase in this test?
I think the test would pass also without sorted_by property specified on the table.

[findinpath]

gentle reminder

[findinpath]

new reminder :)

[alexjo2144]

Still working on test cases and Marius' comments, but added support for sorting during updates and during optimize.

[findinpath]

Naive question: Why not read all the rows from the given parquet file and actually verify that any given row follows the sort order contract in respect to the previous row?

[alexjo2144]

Hmmm, we could it just sounds more complex to set up the test.

[alexjo2144]

@findinpath @findepi @ebyhr I think this is ready for review. Please take a look when you get a chance

[findinpath]

LGTM % comments

[findinpath]

Should we extract fromIdentifierToColumn to a shared utility class?

[alexjo2144]

Probably, but I'd rather leave it until we add transform support and clean it up then. I think there are a few other things that should be central between both partition and sort transform parsing.

#15088

[findinpath]

gentle reminder

[findinpath]

Is this a (temporary) fallback in case that the sorted writing does not work as expected?

[alexjo2144]

It can also be useful if your writes are very small (streaming ingest, for example) such that sorting them would be a waste of time until they are compacted.

[findepi]

Writes being small or not sounds like a query-dependent, so it warrants session toggle more than a catalog config.

Also, can a writer detect that written data is small and not worth sorting?
OTOH, sorting small amount of data sounds like not a big deal (as long as it happens fully in memory and doesn't add latency), so why would we care?

The config should still remain as a kill switch.

[alexjo2144]

@ebyhr @findepi I think I got everything green with this one. Can you kick off a build with serets?

[ebyhr]

/test-with-secrets sha=0f09920b81b690612b026fcfd6e7c4cb252951ee

[github-actions]

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4289597275

[findepi]

Run with secrets failed: #13199 (reopened)

[findepi]

rebased to resolve a conflict

[findepi]

thanks!