feat: implement in-memory storage backend#183
Merged
dhyaniarun1993 merged 21 commits intounstablefrom Oct 12, 2025
Merged
Conversation
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
from
d7fe4bc to
56afdf7
Compare
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
2 times, most recently
from
955e4f1 to
7731223
Compare
meyer9
force-pushed
the
meyer9/166-finalize-storage-interface
branch
from
3e311c7 to
5ef8999
Compare
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
from
7731223 to
a737cf2
Compare
meyer9
force-pushed
the
meyer9/166-finalize-storage-interface
branch
from
5ef8999 to
84c813c
Compare
meyer9
requested review from
BioMark3r,
dhyaniarun1993,
emhane,
itschaindev,
sadiq1971 and
theochap
as code owners
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
from
a737cf2 to
1eef699
Compare
meyer9
force-pushed
the
meyer9/166-finalize-storage-interface
branch
from
84c813c to
9ca3987
Compare
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
from
1eef699 to
1c7a7a3
Compare
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
meyer9
force-pushed
the
meyer9/176-in-memory-storage-backend
branch
from
1c7a7a3 to
f336fc3
Compare
| let mut collected_entries: Vec<(Nibbles, BranchNodeCompact)> = | ||
| if let Some(addr) = hashed_address { | ||
| // Storage trie cursor | ||
| for ((block, address, path), branch) in &storage.storage_branches { |
Collaborator
There was a problem hiding this comment.
Shouldn't we merge the changes from trie_updates for all blocks up to the specified max_block_number?
Collaborator
Author
There was a problem hiding this comment.
I think this should merge them all already. The BTreeMap is sorted by block number ASC, so when we collect all the entries, it will choose the latest block <= max_block_number.
Collaborator
There was a problem hiding this comment.
If I understand correctly, we are storing live updates only in the trie_updates BTreeMap. We aren't saving data to storage_branches BTreeMap in the live syncing.
Hence live updates changes won't be visible to the cursor since cursor is only reading storage_branches BTreeMap
Collaborator
Author
There was a problem hiding this comment.
ah I see, that should be saved to *_branches and hashed_*. The trie_updates and post_states are just indexes from block number to the slots/trie nodes changed for pruning/reorgs.
Collaborator
There was a problem hiding this comment.
Thanks. Will review it.
| .collect() | ||
| } else { | ||
| // Account trie cursor | ||
| for ((block, path), branch) in &storage.account_branches { |
|
does this pr reference any issue? |
meyer9
added a commit
that referenced
this pull request
Oct 31, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 11, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 11, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
meyer9
added a commit
that referenced
this pull request
Nov 13, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
meyer9
added a commit
that referenced
this pull request
Nov 13, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 18, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 18, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 25, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 25, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 8, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 8, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 10, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 10, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 5, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 5, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 13, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 13, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 15, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 15, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 20, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 20, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 26, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 26, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 29, 2026
Ref #176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 29, 2026
Based on #183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 2, 2026
Based on op-rs/op-reth#183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref #176
This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend.