v2.0: Marks old storages as dirty in clean_accounts() (backport of #3702)

[mergify]

Problem

We do not clean up old storages.

More context: when calculating a full accounts hash, we call mark_old_slots_as_dirty() as a way to ensure we do not forget or miss cleaning up really old storages (i.e. ones that are older than an epoch old). But, when we enable skipping rewrites, we don't want to clean up those old storages, as they'll intentionally be treated as ancient append vecs. So inside mark_old_slots_as_dirty() we conditionally mark old slots as dirty. This is based on the value of ancient_append_vec_offset, which should be None unless ancient append vecs are enabled.

Unfortunately, normal running validators, we end up never marking old slots as dirty, because the ancient append vec offset is always Some. And thus we don't clean up old storages.

Summary of Changes

Mark old storages as dirty in clean_accounts().

We still check if ancient append vecs are enabled, but not with the ancient_append_vec_offset. Instead we look at the skipping rewrites feature gate and the cli arg.

By moving this marking into clean_accounts(), we also decouple it from accounts hash calculation, which is not necessary anymore. This also removes behavioral differences based on if snapshots are enabled or not.

Justification to Backport

Without this fix, nodes may never clean up old account storage files, leading to eventual crashes due to running out of file descriptors. There's also the general performance regression that occurs as these old account storage files are unexpectedly kept around forever.

---

This is an automatic backport of pull request #3702 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of bf33b8c has failed:

On branch mergify/bp/v2.0/pr-3702
Your branch is up to date with 'origin/v2.0'.

You are currently cherry-picking commit bf33b8c4d8.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   accounts-db/src/accounts_db.rs
	deleted by us:   accounts-db/src/accounts_db/tests.rs
	both modified:   runtime/src/bank.rs

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[jeffwashington]

lgtm

[HaoranYi]

lgtm.

[t-nelson]

this needs held until #3707 has baked on testnet for awhile

[brooksprumo]

this needs held until #3707 has baked on testnet for awhile

Agreed. I'm also running this on PR (and the original) on some nodes to get more datapoints to ensure this fix is correct and sufficient.

[brooksprumo]

Closing these backport PRs as they do not actually solve the underlying issue.