Improve file delete performance for expire_snapshots

[tbaeg]

Description

Improve file delete performance for expire_snapshots.

Crude benchmark in a test. 10 per deletes per request (manually set the delete batch size).

OLD:

2025-07-21T13:00:29.337-0600	INFO	ForkJoinPool-1-worker-1	stdout	5024 metadata files found
2025-07-21T13:00:29.338-0600	INFO	ForkJoinPool-1-worker-1	stdout	1002 data files found
2025-07-21T13:00:32.795-0600	INFO	ForkJoinPool-1-worker-1	stdout	Time for expire_snapshots time: 3457
2025-07-21T13:00:32.845-0600	INFO	ForkJoinPool-1-worker-1	stdout	2011 metadata files found after expire_snapshots
2025-07-21T13:00:32.845-0600	INFO	ForkJoinPool-1-worker-1	stdout	2 data files found after expire_snapshots

NEW:

2025-07-22T12:50:13.288-0600	INFO	ForkJoinPool-1-worker-1	stdout	5024 metadata files found
2025-07-22T12:50:13.289-0600	INFO	ForkJoinPool-1-worker-1	stdout	1002 data files found
2025-07-22T12:50:15.741-0600	INFO	ForkJoinPool-1-worker-1	stdout	Time for expire_snapshots time: 2452
2025-07-22T12:50:15.797-0600	INFO	ForkJoinPool-1-worker-1	stdout	2011 metadata files found after expire_snapshots
2025-07-22T12:50:15.797-0600	INFO	ForkJoinPool-1-worker-1	stdout	2 data files found after expire_snapshots

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Iceberg
* Improve performance of expire_snapshots procedure. ({issue}`26230`)

[Copilot]

Pull Request Overview

This PR improves the performance of the expire_snapshots procedure in Iceberg by introducing parallel file deletion capabilities and delegating file cleanup operations to Iceberg's native bulk delete functionality.

Key changes include:

• Introduces a new configurable thread pool for file deletion operations
• Replaces custom file deletion logic with Iceberg's native bulk delete operations
• Updates thread pool sizing defaults with reasonable upper bounds

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ForIcebergFileDelete.java	Adds new binding annotation for file delete executor dependency injection
IcebergConfig.java	Adds configuration for file delete thread pool size
IcebergExecutorModule.java	Creates new executor service for file deletion operations
IcebergMetadata.java	Replaces custom file deletion with Iceberg's native bulk delete operations
IcebergMetadataFactory.java	Integrates file delete executor into metadata factory
TestIcebergConfig.java	Updates tests for new configuration and thread pool defaults
BaseIcebergMinioConnectorSmokeTest.java	Updates test expectations for bulk delete behavior
Various test files	Adds file delete executor parameter to test constructors

[raunaqmorarka]

@tbaeg are you able to share any benchmark results from trying this change ?

[tbaeg]

@tbaeg are you able to share any benchmark results from trying this change ?

At the moment, I do not have any benchmarks to share. For our testing environment I'd first need to back port the change. I can try to get something this week.

[ebyhr]

Looks good except for missing documentation.

[findinpath]

@tbaeg if it is not much trouble, could you add in the description of the PR an informal benchmark with and without parallelism for expire_snapshot.
I'd be interested to see the order of acceleration in terms of time we can achieve when using parallelism.

[tbaeg]

@tbaeg if it is not much trouble, could you add an informal benchmark with and without parallelism for expire_snapshot. I'd be interested to see the order of acceleration in terms of time we can achieve when using parallelism.

I was trying to get something informal together on a test cluster with more substantial data, but it looks like it might not be possible. I can write a simple test and profile it locally, but not sure how worthwhile that is.

[tbaeg]

Did a POC that parallelizes ForwardingFileIo.deleteFiles and introduced a new FileIoFactory so I didn't need to inject the ExecutorService every time a new ForwardingFileIo object was created. If you could take a look again, I'd appreciate it.

@raunaqmorarka @findinpath @chenjian2664 @ebyhr

[raunaqmorarka]

.

[raunaqmorarka]

lgtm overall

[chenjian2664]

Looks good % comments