Enable optimized Parquet writer by default in Delta Lake

[ebyhr]

Description

Enable optimized Parquet writer by default in Delta Lake

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Delta Lake
* Enable optimized Parquet writer by default. ({issue}`12757`)

[ebyhr]

Looking CI failure.

Error:  Failures: 
Error:    TestDeltaLakeCreateTableStatisticsLegacyWriter>AbstractTestDeltaLakeCreateTableStatistics.testTimestampMilliRecords:378 expected [Optional[5536338739703808]] but found [Optional.empty]

[raunaqmorarka]

The code changes look ok.
Do the Databricks product tests run automatically ?
Could you also add some rough benchmark results confirming improvement with new writer ?

[findepi]

Could you also add some rough benchmark results confirming improvement with new writer ?

New writer is an improvement regardless of perf numbers, because it avoids JNI for compression, and thus avoids OOM due to GCLocker.

Do the Databricks product tests run automatically ?

They still don't exist, @findinpath @alexjo2144 are working on this.
@ebyhr we need to verify this internally for now.

[homar]

Is there a reason to leave the possibility to use the old writer ?

[findepi]

Is there a reason to leave the possibility to use the old writer ?

just in case. we can remove it later.

[findepi]

Thank you!