Shuffle order of scanning account storages in calculate_accounts_lt_hash_at_startup

[kskalski]

Problem

We scan storages using rayon work splitting heuristic that treats each collection element as relatively similar in cost / processing time. However account storages have vastly differing sizes and their default order actually has clusters of large vs small storages.
This causes work to be split into a unit batch that apparently has mostly large storages and may run for >100s longer than other theads that sit idle without being able to pick more work.

Summary of Changes

• refactor code used to order storages for placement in snapshot archive to account_storage.rs module as more generic AccountStoragesOrderer util wrapper
• add an additional strategy for simple randomized order of storages (doesn't require magic numbers and works well when we only need uniform total size per batch of items)
• use randomized order in calculate_accounts_lt_hash_at_startup

At current snapshot (slot 356410350) unpacking and verification:

startup_verify_accounts total_us=117597515i verify_accounts_lt_hash_us=117597513i

vs baseline (master for the same data):

startup_verify_accounts total_us=372777563i verify_accounts_lt_hash_us=372777560i

[kskalski]

clippy wanted me to implement IntoIterator trait, but then compiler complained I can't use impl Iterator as associated type value, bacause it's unstable :(

[brooksprumo]

Is that why we need this to be a Boxed dyn?

[brooksprumo]

We could call the method something other than into_iter() and then clippy should be OK.

For example one of these:

    pub fn my_iter(&self) -> impl ExactSizeIterator<Item = &'a AccountStorageEntry> + use<'a, '_> {
        self.indices.iter().map(|i| self.storages[*i].as_ref())
    }

    pub fn my_into_iter(self) -> impl ExactSizeIterator<Item = &'a AccountStorageEntry> + use<'a> {
        self.indices.into_iter().map(|i| self.storages[i].as_ref())
    }

And then we can remove the impl IntoIterator until the impl trait is allowed in associated types here.

(or fn into_seq_iter(), etc...)

[kskalski]

yap, without impl_trait_in_assoc_type I need to name the iterator type explicitly, other approaches:

• create a struct that encapsulate the map operation and implement Iterator trait, such that IntoIterator can just use the struct's type here
• work-around the clippy insistence on having into_iter() as actual IntoIterator implementation, e.g. use a non-standard naming like make_iter
• a deeper refactor such that we don't use map to create iterator, then we can probably name the type being used as iterator implementation

Similar concern is with IntoParallelIterator, but for it there is no clippy check

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.2%. Comparing base (d76c437) to head (6c0a181).
⚠️ Report is 2712 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #7226    +/-   ##
========================================
  Coverage    83.1%    83.2%            
========================================
  Files         850      850            
  Lines      369949   369978    +29     
========================================
+ Hits       307739   307849   +110     
+ Misses      62210    62129    -81     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[brooksprumo]

Is that why we need this to be a Boxed dyn?

[brooksprumo]