Reduce writers retained memory utilization

[sopel39]

Whenever writer is closed, it's instance was
retained so that rollback could be performed
in case of an error. However, this was retaining
excessive amount of memory which was not needed
anymore.

This commit introduces RollbackAction that can
be retained after driver is closed which should
significantly reduce memory usage during long
inserts.

Description

Non-technical explanation

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

[electrum]

Which writers are holding onto memory after they are closed? Can we fix that directly instead of making the code more complicated with this change?

[sopel39]

Which writers are holding onto memory after they are closed? Can we fix that directly instead of making the code more complicated with this change?

it's OrcWriter and ParquetWriter. I decided that it's too cumbersome and not really elegant to make multiple fields inside these writers @Nullable and mutable. IMO it's cleaner and simpler just to drop writer when it's not needed.

[electrum]

Alternative approach: #14824

[sopel39]

ac

[raunaqmorarka]

To me making the rollback action a Closeable looks weird, there is no resource that we are releasing.
You could use Runnable instead if the problem is that you want to avoid extra return null.

[sopel39]

You could use Runnable instead if the problem is that you want to avoid extra return null.

I actually use it in try-with-resource, e.g:

        try (rollbackAction) {
            avroWriter.close();
        }
        catch (Exception e) {
            throw new TrinoException(ICEBERG_WRITER_CLOSE_ERROR, "Error rolling back write to Avro file", e);
        }

so I don't have to deal with syntax sugar myself. I think using native Java try-with-resource is actually better than relying on Guava closer.