Use batch deletes in IcebergMetadata

[alexjo2144]

Description

@homar recently added an API to the FileSystem API to delete a batch of files, if the underlying filesystem supports it. Leverage that API in places where many files are deleted at once.

Non-technical explanation

Leverage delete APIs in a more efficient way, if available.

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

[findinpath]

You may want to make the multi-file deletion in batches of DELETE_BATCH_SIZE size.

[homar]

+1

[alexjo2144]

I'll add it, but IMO it's not really necessary. In remove_orphan_files or expire_snapshots it's a good idea because we're loading the list of files to delete incrementally, and batching there prevents us from materializing the full list in memory. Here, we already have the list in memory, so we can rely on the file system to batch in sizes appropriate for the implementation.

[homar]

I agree that the way you did it here is not helpful. Actually I expected you will partition it on higher level -

fullyDeletedFiles.values().stream()
                        .flatMap(Collection::stream)
                        .map(CommitTaskData::getPath)

without collecting it to list first. But now, after I learnt that Lists.partition() is not lazy, I am not sure if this can be done...

[findepi]

What's the benefit?
if the FS needs batches, why not let it do on its own?

[findepi]

i agree with 2c87d6b#r1015558983 that the data list is already in memory (in some form)

[findepi]

keep deleteBatch.add(location + "/" + fileName); and if (deleteBatch.size() >= DELETE_BATCH_ lines together

[findepi]

Build is green. Will squash and merge.