feat(persistence): batch write hashed_state

[duyquang6]

As discussed #19739 (comment)
Batch write of hashed_state is safe, so I created this PR to cherry-pick old reverted commit

Changes

• batch write hashed_state

*Note: The after result here is after we improve extend_sorted_vec, which is upcoming PR

before

erc20 transfers spam: ~100ms

native transfers spam: ~50ms

after

erc20 transfers spam: ~80ms

native transfers spam: ~30ms

[mattsse]

pedantic doc nit

[mediocregopher]

The previous implementation was specifically designed to avoid having to do a big collect like this; the resulting memory allocation from this collect dwarfs any ostensible speedup you get from not having to sort. I just did a bench comparing your implementation to the previous and this new one is about 2x slower for synthetic datasets:

You can see the bench here if you're curious

[duyquang6]

very useful bench @mediocregopher
when I first use old version with aggregated hashed state to bench, the result is not good. This is why I think there something wrong with extend_ref or extend_sorted_vec

when I use your bench compare with custom own merge version (not used merge_join_by), it only shine at other target size smaller than other size, but overall case, old version still win. That give me some hint to use this function better, is keep target size and other size similar or larger so might benefit old version

shine case (new better)

but overall (size similar or target size > other) old still better

[duyquang6]

I dump the raw data of HashedPostStateSorted when bench with native-transfer

here is bench result of extend_ref, new version of both is better than in this testcase

can double check the bench here - already attach hashed state raw data
Could be raw data might have properties that benchmark doesn’t fully cover 🤔 ?

[mediocregopher]

@duyquang6 these are interesting results, your benches for extend_sorted_vec_comparison/t10_o1000 conflict with what I originally saw in mine, but now I'm able to replicate, so there's some inconsistency there that I still need to figure out.

If yours is faster for t10_o1000 I expect it's because it's doing the full allocation up-front, whereas mine is likely doing two larger allocations at the end with the extend calls.

What do you think about trying out something like:

    // Where "50" is a made up number that needs to be tuned
    if other.len() > target.len() * 50 {
        return extend_sorted_vec_custom(target, other);
    }

    extend_sorted_vec(target, other)

This way maybe we better cover all cases.

[duyquang6]

I dump the raw data of HashedPostStateSorted when bench with native-transfer

here is bench result of extend_ref, new version of both is better than in this testcase

can double check the bench [here](https://github.com/duyquang6/reth/blob/bench-sorted-extend/crates/trie/common/benches/extend_ref.rs#L286) - already attach hashed state raw data Could be raw data might have properties that benchmark doesn’t fully cover 🤔 ?

draft temporarily. I will invest more time on finding root cause why there is different, will update later

[mediocregopher]

Closes #20609

[duyquang6]

Hi, I'm currently busy with other work and won't have time for this PR for a few weeks. If anyone wants to take this over to unblock #20609, feel free - otherwise I'll get back to it when I have bandwidth
cc sir @mediocregopher

nvm, i got some bandwidth today, will work on this

[duyquang6]

I benchmarked with Vec<B256> instead of Vec<u64> since that matches the real use case. Here is results from M1 Pro, differ significantly from u64 benchmarks:

The custom merge version is ~30% faster than the current extend_sorted_vec for B256 data.

Result of Vec<u64>:

---

Summary:

Tested three approaches for merging sorted vectors:

1. default: In-place overwrites for duplicates, collects new items, sorts at end
2. merge: Classic single-pass merge into new vector, O(n+m)
3. itertool_merge: Uses itertools::merge_join_by

Key findings:

Scenario	Winner	Speedup
Large B256 keys (100k entries)	custom_merge	~30-40% faster
Small u64 keys with high overlap	default	~20% faster
Small datasets (<100 entries)	All similar	negligible

Why merge wins for B256 (use case on HashedPostState):

• B256 comparision is more expensive that u64, that make difference
• At 100k accounts: merge ~0.8ms vs default ~1.2ms

Benchmark code: https://github.com/duyquang6/reth/blob/bench-sorted-extend/crates/trie/common/benches/extend_sorted_vec.rs

Implementation: https://github.com/duyquang6/reth/blob/bench-sorted-extend/crates/trie/common/src/utils.rs

should I split 2 PR - since with batch write, we can resolve #20609 first ?:

• Batch write hashed state (this PR)
• Improve extend_sorted_vec (if needed)

sir @mediocregopher @mattsse

[cliff0412]

may i know what tool u used to do erc20/native transfers spam?

[duyquang6]

may i know what tool u used to do erc20/native transfers spam?

Hi, we wrote a custom Rust script for benchmarking transaction throughput

[mediocregopher]

We've gone with a different approach in #21422 and confirmed small perf improvement based on that. Further improvements can be based on that work, going to close this for now