Stream large transaction log jsons instead of storing in-memory

[raunaqmorarka]

Description

Operations fetching metadata and protocol entries can skip reading the rest of the json file after those entries are found

Additional context and related issues

On a example transaction log json of 1.5GB, the time taken for simple operations
like register table, DESCRIBE and SELECTs which don't use table statistics (or any read with set session delta.statistics_enabled=false) reduces from 18s to under 1s on local machine.
Such large transaction log jsons were observed to have been produced by CLONE operation from Apache Spark.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Delta Lake
* Improve performance of queries on tables with large transaction log jsons. ({issue}`24491`)

[Pluies]

@raunaqmorarka very interesting! I've skimmed the code, would you consider this an alternative approach to the metadata/protocol caching we were trying to push in #20437, or more as a separate improvement? I assume they'll impact the same sort of time-consuming operations.

(Asking as I'm keen on getting out fork closer to upstream; it doesn't really matter which approach ends up being used as long as performance improves 👍 )

[raunaqmorarka]

@raunaqmorarka very interesting! I've skimmed the code, would you consider this an alternative approach to the metadata/protocol caching we were trying to push in #20437, or more as a separate improvement? I assume they'll impact the same sort of time-consuming operations.

(Asking as I'm keen on getting out fork closer to upstream; it doesn't really matter which approach ends up being used as long as performance improves 👍 )

Thanks for pointing out that PR, I hadn't looked at it before. For me the priority was to deal gracefully with transaction log jsons that are GBs in size. I've tweaked the PR a bit to be better about caching metadata/protocol entries. Feel free to try this out on your workloads or add review comments.

[findinpath]

Looks promissing to me.