revert: "perf(persistence): improve write batch for HashedPostState & TrieUpdatesSorted"

[mattsse]

Reverts #19739

pending more testing

[duyquang6]

Hi @mattsse, I saw that PR #19739 was reverted.
Could you share the reasoning or issues behind the revert?
I’d like to understand the cause and help with a follow-up fix if needed.

[yongkangc]

@duyquang6 this was the issue caused by the pr

Hoodi nightly is stuck

Nov 17 18:57:59.616 WARN  Invalid execution payload                     validation_error: Some("mismatched block state root: got 0x24e282c4ede32948b27aa3f152d324ec1f64cec3501675e6be840a2ced163ca7, expected 0x3f2ff081aa727d37bfdf3fab7cf9259aebd0af1cd0da6f44247bd2e411ac06be"), latest_valid_hash: Some(0x927af68e134029b9253dcf3118602eec26147b13dc8331803d98244afbc096a7), execution_block_hash: 0x7fb903bb8efbc1a4aabcc3a43a3ac48816bcf9675c2e045b14b291ed56e631e9, root: 0xaee37919f70b6f78f333ec23fbc0dfabc86e99545701654890fa4ee8e96bc7dd, graffiti: "chungus-lighthouse-geth-1", proposer_index: 103812, slot: 1765983, method: "new_payload"

we suspect the problem is that it batches the call to write_hashed_state which happens immediately after. write_state depends on the call to write_hashed_state for the previous block to have been called, but now write_hashed_state for all blocks in the batch happens at the end.

This PR aggregates the updates until the end of the batch, so changesets aren't written correctly.
So potentially these tables are corrupted:

• AccountsChangeSets
• StoragesChangeSets
• AccountsTrieChangeSets
• StoragesTrieChangeSets
• AccountsTrie/StoragesTrie (if a block was validated using reverts)
  I don't think the plain/hashed state tables themselves would be affected, unless maybe there's a reorg.

[duyquang6]

@duyquang6 this was the issue caused by the pr

Hoodi nightly is stuck

Nov 17 18:57:59.616 WARN  Invalid execution payload                     validation_error: Some("mismatched block state root: got 0x24e282c4ede32948b27aa3f152d324ec1f64cec3501675e6be840a2ced163ca7, expected 0x3f2ff081aa727d37bfdf3fab7cf9259aebd0af1cd0da6f44247bd2e411ac06be"), latest_valid_hash: Some(0x927af68e134029b9253dcf3118602eec26147b13dc8331803d98244afbc096a7), execution_block_hash: 0x7fb903bb8efbc1a4aabcc3a43a3ac48816bcf9675c2e045b14b291ed56e631e9, root: 0xaee37919f70b6f78f333ec23fbc0dfabc86e99545701654890fa4ee8e96bc7dd, graffiti: "chungus-lighthouse-geth-1", proposer_index: 103812, slot: 1765983, method: "new_payload"

we suspect the problem is that it batches the call to write_hashed_state which happens immediately after. write_state depends on the call to write_hashed_state for the previous block to have been called, but now write_hashed_state for all blocks in the batch happens at the end.

This PR aggregates the updates until the end of the batch, so changesets aren't written correctly. So potentially these tables are corrupted:

* AccountsChangeSets

* StoragesChangeSets

* AccountsTrieChangeSets

* StoragesTrieChangeSets

* AccountsTrie/StoragesTrie (if a block was validated using reverts)
  I don't think the plain/hashed state tables themselves would be affected, unless maybe there's a reorg.

interesting, I will try reproduce on my side and see what actually broken
thank @yongkangc and @mediocregopher