chore(trie): fully reveal sparse tries prior to leaf updates/removals

[mediocregopher]

Fixes #17571

Background

The Serial/ParallelSparseTrie's have their nodes revealed using the DecodedMultiProofs generated based on the previous database state and the changed leafset. Once revealed, the leaf updates/removals are applied to the sparse tries and root hashes are calculated.

There are specific edge-cases during leaf adding/removal where a node which is outside the changeset is required to complete the operation. In these cases we were falling back to a singular database lookup to fill in the gap.

By modifying the generation of the DecodedMultiProofs to include those extra required nodes we can simplify the sparse trie implementations significantly, as well as the surrounding engine code which supports these one-off lookups.

Changes

This primarily relies on a change to the ProofRetainer in alloy-trie, made here:

alloy-rs/trie#109

These changes are enabled by a flag which the payload validator enables; other places where proofs are generated remain unnaffected.

There is a further change here to the trie walker to not skip over branch nodes when they might be involved in a leaf removal. A change to support leaf addition is not required, because that case involves children of extension nodes, and we don't ever skip over extension nodes anyway.

The sparse trie code to fallback to the db for missing nodes remains in place for now, but we emit a warning when it happens. This will let us track if there's any edge-cases we're missing.

Benchmarks - 2k blocks on main

No significant change

Timestamp: 2025-07-28 10:37:12 UTC
Baseline: main
Feature:  origin/mediocregopher/17571-leaf-updates-removals

Performance Changes:
  NewPayload Latency: -1.24%
  FCU Latency:        -1.74%
  Total Latency:      -1.26%
  Gas/Second:         +1.27%
  Blocks/Second:      +1.27%

Baseline Summary:
  Blocks: 2000, Gas: 36118922440, Duration: 74.02s
  Avg NewPayload: 35.69ms, Avg FCU: 1.29ms, Avg Total: 36.99ms
  Started: 2025-07-28 10:48:20 UTC, Ended: 2025-07-28 10:56:39 UTC

Feature Summary:
  Blocks: 2000, Gas: 36118922440, Duration: 73.08s
  Avg NewPayload: 35.25ms, Avg FCU: 1.27ms, Avg Total: 36.52ms
  Started: 2025-07-28 10:58:35 UTC, Ended: 2025-07-28 11:06:55 UTC

[Rjected]

The sparse trie code to fallback to the db for missing nodes remains in place for now, but we emit a warning when it happens. This will let us track if there's any edge-cases we're missing.

Assuming there are no warnings from the benches? Are there any changes in the metrics (nodes fetched for ex)

[mediocregopher]

The sparse trie code to fallback to the db for missing nodes remains in place for now, but we emit a warning when it happens. This will let us track if there's any edge-cases we're missing.

Assuming there are no warnings from the benches? Are there any changes in the metrics (nodes fetched for ex)

I swear there were no warnings when I ran this against mainnet on monday, but now I'm seeing two related to leaf removal, so there's a bit more investigation to do here 🤦

Regarding metrics, I just reran the bench (2k blocks on mainnet) and this time it came out 3% faster 🤷 there were indeed more multiproofs fetched and more db txs generally though.

Summary:
Total multiproof account nodes: +20.4118%
Total multiproof storage nodes: +19.1292%
Opened Read-Write Transactions: +13.9773%
Opened Read-Only Transactions: +12.8246%

[mattsse]

I believe I was able to follow along here

all of this made sense, lgtm but also want @Rjected to take a final look at this

[mattsse]

ah I see, this way this works for both, proofretainer and addedremovedkeys

makes sense

[Rjected]

LGTM, nice

[Rjected]

needs docs fixes