Cache collected file statistics

[mateuszkj]

Which issue does this PR close?

Closes #871 (but I'm not sure about that).

Rationale for this change

Reduce I/O by collecting statistics for files (parquet) only once in ListingTable.

What changes are included in this PR?

Store collected statistics in cache per file location.

Cache is invalided when:

• File size has changed
• File last modification has changed

Are there any user-facing changes?

No.

Or maybe mention that sometimes when collect_stats is enabled first query can be much slower due to increased I/O while collecting statistics. Cached statistics are invalidated in next query when table file has changed.

[mateuszkj]

JYI, collecting statistics is disabled when creating table from SQL (CREATE EXTERNAL TABLE ..), it's different behavior when using function SessionContext.register_parquet(..) where collecting statistics is enabled.

https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/context.rs#L507 -> collect_stat: false

https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/options.rs#L195 -> collect_stat: true

I'm not sure if it's intended.

[codecov-commenter]

Codecov Report

Merging #3649 (b6c7608) into master (b4c0601) will increase coverage by 0.00%.
The diff coverage is 87.50%.

@@           Coverage Diff           @@
##           master    #3649   +/-   ##
=======================================
  Coverage   86.01%   86.01%           
=======================================
  Files         300      300           
  Lines       56543    56595   +52     
=======================================
+ Hits        48633    48682   +49     
- Misses       7910     7913    +3     

Impacted Files	Coverage Δ
datafusion/core/src/datasource/listing/table.rs	90.40% <87.50%> (+0.19%)	⬆️
datafusion/expr/src/window_frame.rs	92.43% <0.00%> (-0.85%)	⬇️
datafusion/expr/src/logical_plan/plan.rs	77.26% <0.00%> (ø)
datafusion/optimizer/src/reduce_outer_join.rs	98.35% <0.00%> (+0.15%)	⬆️
datafusion/core/src/physical_plan/metrics/value.rs	87.56% <0.00%> (+0.49%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[alamb]

Looks good to me -- thank you @mateuszkj

[ursabot]

Benchmark runs are scheduled for baseline = f706902 and contender = 85c11c1. 85c11c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java