AppendVec::new_from_storage() does sanitization if len and capacity do not match

[brooksprumo]

Problem

Minimized snapshots created with older versions of ledger-tool can have storages with a capacity (file size) larger than their len (number of bytes) (see #6797 for an example). This is no longer an issue in master, as shrink (which is used to write the new, minimized storages) now sets the correct file size for new storages.

After merging #6552, it was no longer possible to open those snapshots with storages that have mismatched capacity and len.

Summary of Changes

At startup when rebuilding storages, if the capacity and len mismatch, fall back to the old/slow impl that does the full sanitization of the storage file.

Fixes #6797

Note, since #6552 was backported to v2.3, I think we should also backport this to v2.3.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 13 lines in your changes missing coverage. Please review.

Project coverage is 83.3%. Comparing base (ebfb7c2) to head (fd0abf1).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6803   +/-   ##
=======================================
  Coverage    83.3%    83.3%           
=======================================
  Files         853      853           
  Lines      378181   378191   +10     
=======================================
+ Hits       315281   315293   +12     
+ Misses      62900    62898    -2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[HaoranYi]

nice debugging.
lgtm.

[roryharr]

When was shrink fixed to store the correct size?

And how long will backwards compatibility be required? Is it worth adding a test to ensure this path doesn't break again?

[brooksprumo]

When was shrink fixed to store the correct size?

There's been multiple fixes over the years. I thought it had been fixed for a while. Firedancer reported the issue here #6797, so it depends on what version they used to create that minimized snapshot.

---

And how long will backwards compatibility be required?

We only need to guarantee compatibility between adjacent versions. But, no need to break folks unnecessarily. Since the firedancer minimized snapshot worked in v2.2, we should let it continue to work at least for v2.3.

Is it worth adding a test to ensure this path doesn't break again?

Maybe? We'd have to manually create invalid storage files to do that (invalid w.r.t. cap vs len). I'm currently adding a fix to the snapshot minimization that will include a new test that will actually create a minimized snapshot and then load it.

[brooksprumo]

@anza-xyz/backport-reviewers FYI, intending to backport this one. Tagging to get eyes on it before merging to master.

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.