perf(storage): add bloom filter for empty storage slots#21185
Draft
perf(storage): add bloom filter for empty storage slots#21185
Conversation
Implements a bloom filter to short-circuit storage reads for empty slots, potentially avoiding RocksDB lookups for 30-40% of storage reads. Based on analysis from Nethermind's FlatDB work: - Mainnet has ~500M-1B non-empty storage slots - A bloom filter can definitively say 'not present' for empty slots - Potential 10-20% mgas/s improvement on some block ranges Architecture: - New reth-storage-bloom crate with GrowableBloom filter - BloomStateProvider wrapper that checks bloom before DB access - Metrics for bloom hits, misses, and false positives - Persistence support for saving/loading bloom to disk Trade-offs: - Memory: ~1-2GB for mainnet scale at 1% false positive rate - Write amplification: Every storage write updates bloom - Feature-gated behind 'storage-bloom' flag Closes #21184 Amp-Thread-ID: https://ampcode.com/threads/T-019bd355-619a-769b-9eb0-40d1dc2d47ba Co-authored-by: Amp <amp@ampcode.com>
Member
Author
Benchmark Started 🚀The CodSpeed benchmark workflow is running: https://github.com/paradigmxyz/reth/actions/runs/21133685427 Note: This is the standard What to measure:
cc @brian @alexey - will need to coordinate a manual benchmark run on reth3/reth5 to get meaningful numbers. |
- Replace FPR abbreviation with 'false positive rate' - Merge match arms with identical bodies - Apply nightly rustfmt formatting
Amp-Thread-ID: https://ampcode.com/threads/T-019bd355-619a-769b-9eb0-40d1dc2d47ba Co-authored-by: Amp <amp@ampcode.com>
mediocregopher
requested changes
Jan 19, 2026
Member
mediocregopher
left a comment
There was a problem hiding this comment.
Integrate into the payload validator in engine tree
- Remove file persistence (load_from_file, save_to_file) from bloom filter - Remove BloomError type and related test - Integrate storage bloom filter into BasicEngineValidator in engine tree - Add storage-bloom feature flag to engine-tree crate
- Remove file persistence code (load_from_file, save_to_file) per review - Integrate bloom filter into payload validator in engine tree - Add storage-bloom feature flag to reth-engine-tree - Export BloomStateProvider from reth-provider The bloom filter now starts fresh on each node restart and is integrated into the payload validation path where it wraps the state provider to short-circuit empty storage slot reads.
This fixes the API breakage when storage-bloom feature is enabled. The bloom filter is now created inside the BasicEngineValidator::new constructor with default configuration, keeping the constructor signature stable regardless of feature flags.
Per review feedback, removed the feature flag and made the storage bloom filter the default behavior. The bloom filter is now always enabled and integrated into the payload validator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a bloom filter to short-circuit storage reads for empty slots, potentially avoiding RocksDB lookups for 30-40% of storage reads.
Based on analysis from Nethermind's FlatDB work (PR #9854):
Architecture
reth-storage-bloomcrate withGrowableBloomfilterBloomStateProviderwrapper that checks bloom before DB accessTrade-offs
From Nethermind's experience:
How to Test
This is feature-gated behind
storage-bloomflag. To enable:Benchmark Request
Need to benchmark on mainnet block re-execution to measure:
cc @brian @alexey - please review benchmark results when available
Closes
Closes #21184