Reuse buffered reader with overflow across full accounts storage scans

[kskalski]

Problem

Currently full accounts scan creates buffered reader with overflow capacity for each storage and discards it after that storage is done processing. This also drops any additional state or work that might have been done by the reader implementation - none is done by current BufferedReader, but it prevents us from using multi-file read-ahead implementation of FileBufRead there.

Summary of Changes

• remove &File from mandatory args of BufferedReader constructore, use Option<&file> in reader state
• add set_file(file: &File, read_limit: usize) function to FileBufRead trait - this activates the file to be read from and resets buffer state and file offset
• update scan iterations structure to create single buffered reader per work unit (usually this is dynamically defined by rayon par_iter) and pass it around to each storage scan

Reusing the reader saves some allocations made when reading large accounts data, but it is in-frequent enough that this PR is performance neutral:

master
rebuild bank from snapshots took 186.3s
startup_verify_accounts total_us=113899487i verify_accounts_lt_hash_us=113899484i

pr
rebuild bank from snapshots took 183.0s
startup_verify_accounts total_us=113460880i verify_accounts_lt_hash_us=113460878i

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.52542% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.2%. Comparing base (228732f) to head (5e764a9).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #7080     +/-   ##
=========================================
- Coverage    83.2%    83.2%   -0.1%     
=========================================
  Files         796      796             
  Lines      361923   361996     +73     
=========================================
+ Hits       301346   301394     +48     
- Misses      60577    60602     +25     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #7226. Mergify cannot evaluate rules on this PR. ⚠️

[HaoranYi]

lgtm. Reusing buffers instead of creating new ones for each storage seems reasonable.

[brooksprumo]

Looks good! One nit then

[brooksprumo]

Also, if you can avoid force-pushing/rebasing, it makes reviewing easier.

Here's an example. With the force-push, GitHub doesn't remember what commits I've already seen, so they are all marked as new:

This PR wasn't too bad, but it can make some reviews much harder, as then I have to go back and double check more stuff to ensure nothing else was changed. My default is to not force-push/rebase unless absolutely necessary (e.g. to resolve a merge conflict, or pick up a change from another PR that this PR depends on).

[brooksprumo]

[kskalski]

Also, if you can avoid force-pushing/rebasing, it makes reviewing easier.

True, I try to avoid it, but with some of these changes conflicts pop up quite often and today there was a really painful issue with CI, so I rebased on pick up a fix and let the CI pass.