v2.3: Do not sanitize append vecs at startup (backport of #6552)

[mergify]

Problem

At startup from a snapshot, we must rebuild/reconstruct all the AppendVecs. This is basically relinking all the storage files on disk to the AccountStorageEntrys that were serialized into the bank snapshot (aka snapshot manifest). We use the storage file paths from the snapshot manifest to reconstruct the AppendVecs.

We also do some sanitization of the storage files when reconstructing the new AppendVecs. One step is to sanitize each account in the storage file. This involves scanning (i.e. reading) the whole file, and checking every single account to make sure the accounts are valid. Scanning every account in every storage file is rather time consuming. On mainnet-beta, we now have over 900 million accounts. Doing this sanitization step takes over 100 seconds.

An observation is that we also read all the accounts/storages two other times as part of startup. One is for index generation, and the other is for startup accounts verification. If the storages are wrong/invalid, then one (or both) of these other tasks will catch it. Thus, we can avoid the sanitization step during startup reconstruction.

Speeding up reconstructing the storages, speeds up startup. And if startup is faster, that means validators begin replaying faster. If validators begin replaying faster, they catch up faster, AND that also means there are fewer blocks they have to repair (vs getting through turbine).

Summary of Changes

Skip AppendVec accounts sanitization during startup when reconstructing storages.

Testing against mnb, I saw times drop from ~100-120 seconds to ~2 seconds for reconstructing storages at startup.

Justification to Backport

@alessandrod asked me to.

---

This is an automatic backport of pull request #6552 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 3804f39 has failed:

On branch mergify/bp/v2.3/pr-6552
Your branch is up to date with 'origin/v2.3'.

You are currently cherry-picking commit 3804f3934.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   accounts-db/src/accounts_file.rs
	modified:   accounts-db/src/append_vec.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   runtime/src/serde_snapshot.rs

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[brooksprumo]

This backport PR is blocked until #6580 is merged.

[brooksprumo]

@Mergifyio rebase

[mergify]

rebase

☑️ Nothing to do, the required conditions are not met

Details

• -conflict [📌 rebase requirement]
• -closed [📌 rebase requirement]
• queue-position = -1 [📌 rebase requirement]
• any of:
  • #commits-behind > 0 [📌 rebase requirement]
  • #commits > 1 [📌 rebase requirement]
  • -linear-history [📌 rebase requirement]

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.95652% with 3 lines in your changes missing coverage. Please review.

Project coverage is 82.8%. Comparing base (3e1c3a2) to head (c888147).

Additional details and impacted files

@@            Coverage Diff            @@
##             v2.3    #6579     +/-   ##
=========================================
- Coverage    82.8%    82.8%   -0.1%     
=========================================
  Files         848      848             
  Lines      379542   379563     +21     
=========================================
- Hits       314575   314544     -31     
- Misses      64967    65019     +52     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[HaoranYi]

lgtm.