feat: implement live state collector using external provider#198
Merged
dhyaniarun1993 merged 18 commits intounstablefrom Oct 16, 2025
Merged
feat: implement live state collector using external provider#198dhyaniarun1993 merged 18 commits intounstablefrom
dhyaniarun1993 merged 18 commits intounstablefrom
Conversation
meyer9
force-pushed
the
meyer9/169-1-state-provider
branch
from
0383453 to
1d82611
Compare
meyer9
force-pushed
the
meyer9/169-2-state-provider
branch
from
20c8bab to
10b4952
Compare
meyer9
force-pushed
the
meyer9/169-1-state-provider
branch
from
5f2e461 to
74d1ac9
Compare
meyer9
force-pushed
the
meyer9/169-2-state-provider
branch
from
10b4952 to
e4d42f9
Compare
meyer9
requested review from
dhyaniarun1993,
emhane,
itschaindev and
sadiq1971
as code owners
emhane
added
K-feature
meyer9
force-pushed
the
meyer9/169-2-state-provider
branch
2 times, most recently
from
dc0c939 to
6e00453
Compare
meyer9
force-pushed
the
meyer9/169-1-state-provider
branch
from
067c64e to
c365ff3
Compare
meyer9
force-pushed
the
meyer9/169-2-state-provider
branch
from
334ce08 to
c77c060
Compare
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
meyer9
force-pushed
the
meyer9/169-2-state-provider
branch
from
d008bcb to
a7ac239
Compare
Collaborator
Author
|
Metrics issue tracked #246 |
Collaborator
Author
|
Would love 1 more review on this before merging! @dhyaniarun1993 @itschaindev @sadiq1971 |
dhyaniarun1993
approved these changes
Oct 16, 2025
Collaborator
dhyaniarun1993
left a comment
dhyaniarun1993
left a comment
There was a problem hiding this comment.
Looks good to go.
emhane
added a commit
that referenced
this pull request
Oct 20, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
meyer9
added a commit
that referenced
this pull request
Oct 31, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 11, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
meyer9
added a commit
that referenced
this pull request
Nov 13, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 18, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Nov 25, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 8, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 10, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Dec 18, 2025
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 5, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 13, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 15, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 20, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 26, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
emhane
added a commit
that referenced
this pull request
Jan 29, 2026
Based on #197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
theochap
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 13, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
theochap
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 13, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
theochap
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 14, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
theochap
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 17, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
theochap
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 17, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
github-merge-queue bot
pushed a commit
to ethereum-optimism/optimism
that referenced
this pull request
Feb 18, 2026
…19194) * refactor(db): derive Clone for DatabaseEnv (paradigmxyz/reth#21641) Co-authored-by: Amp <amp@ampcode.com> * chore(deps): breaking bumps (paradigmxyz/reth#21584) Co-authored-by: Georgios Konstantopoulos <me@gakonst.com> Co-authored-by: Amp <amp@ampcode.com> * refactor: add with_* compressor utility methods (paradigmxyz/reth#21680) * refactor: use alloy_primitives::map for all HashMap/HashSet types (paradigmxyz/reth#21686) Co-authored-by: Amp <amp@ampcode.com> * fix: support EIP-1559 params configuration for Optimism dev mode (paradigmxyz/reth#21855) * chore(rust/op-reth): pull latest changes and update dependencies * chore(op-exex): Add crate `reth-optimism-exex` and `reth-optimism-trie` (op-rs/op-reth#204) - Adds new crates `reth-optimism-exex` and `reth_optimism_trie` - Moves `reth_exex::external_proofs` -> `reth_optimism_exex` - Moves `reth_exex::external_proofs::storage` -> `reth_optimism_trie` * fix: ci lint and doc (op-rs/op-reth#234) Seems like when `reth-optimism-trie` depends on `reth-db-api` indirectly, the doctest crate couldn’t resolve `reth_db_api::…` and failing. Closes op-rs/op-reth#233 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: separate storage and account cursors (op-rs/op-reth#229) This PR separates storage and account cursors which makes sense because these two cursors may have very different seek/next implementations. In the case of in-memory, a single cursor impl can still handle both, but for MDBX, it makes sense to implement separate cursors. Closes op-rs/op-reth#236 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add external state provider implementation (op-rs/op-reth#197) Based on op-rs/op-reth#203 , op-rs/op-reth#204 This PR implements `StateProvider` given a `OpProofsStorage` instance. It reads most data from the external database, but falls back on the latest provider for block hashes and code by hash similar to the existing historical provider in Reth. This is an important part to implementing live syncing since we're running the sync process on the DB being created. In the Reth implementation, `proof.rs` is contained in `trie/db`, so I think it makes sense to go in our DB crate. The `provider.rs` is in the `reth-provider` crate, but I don't think a separate provider crate helps us here, so I think we should also include that in the trie crate as well. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: add deletion tests for op proofs storage (op-rs/op-reth#230) Based on op-rs/op-reth#229 Adds tests to ensure that storing TrieUpdates that include deletions actually deletes the nodes at that block height, and that updates take precedence over deletions (same as `write_trie_updates` in Reth). Closes op-rs/op-reth#238 * feat: implement live state collector using external provider (op-rs/op-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add binary entry point for external proofs in OP (op-rs/op-reth#222) Closes op-rs/op-reth#164 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: store hashed account and storage (op-rs/op-reth#235) Closes op-rs/op-reth#211 * feat: implement cursor read/write metrics (op-rs/op-reth#232) Fixes op-rs/op-reth#225 (does not track errors as these are fatal currently) * feat: store account and storage branches (op-rs/op-reth#255) Closes op-rs/op-reth#210 * feat: Concrete error variants for `reth-optimism-trie` (op-rs/op-reth#251) Fixes op-rs/op-reth#237 Adds more concrete error variants for `OpProofsStorageError` Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Remove block_number from initial sync store methods (op-rs/op-reth#248) Fixes op-rs/op-reth#247 Removes the `block_number` parameter from: - store_account_branches - store_storage_branches - store_hashed_accounts - store_hashed_storages Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(MDBX): Implement earliest and latest block tracking in storage (op-rs/op-reth#253) Fixes op-rs/op-reth#215 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(deps): Add `serde-bincode-compat` feature to `reth-trie` (op-rs/op-reth#250) Ref op-rs/op-reth#241 - Fixes comment from deps fix in op-rs/op-reth#229 (fix correct but comment misleading) - `serde-bincode-compat` is a feature which must be enabled when the `bincode` dependency is used. Asides for dev-deps, this only happens in `reth-stages`. `reth-exex` runs in `reth-stages`. In this wokrstream we use the `TrieUpdate` in exex. `serde-bincode-compat` is already implemented for `TrieUpdates` in `reth-trie-common`, this PR simply makes that feature accessible via `reth-trie` and in turn `reth-optimism-trie`, in order to enable it in `reth-optimism-exex`. * feat: store trie updates (op-rs/op-reth#258) Closes: op-rs/op-reth#214 * feat: implemented mdbx trie cursor (op-rs/op-reth#249) Closes op-rs/op-reth#212 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(deps): Rm redundant dep (op-rs/op-reth#260) Ref op-rs/op-reth#205 Removes redundant dep `reth-db-api` from `reth-optimism-trie`, since used types are re-exported via `reth-db` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * refactor: replace tuple with `BlockNumHash` (op-rs/op-reth#254) Resolves op-rs/op-reth#239 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> * feat: Return ref to B256 (op-rs/op-reth#272) Fixes op-rs/op-reth#270 * chore(exex): Add missing crate header to `reth-optimism-exex` (op-rs/op-reth#271) Resolves op-rs/op-reth#243 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented hashed cursors (op-rs/op-reth#268) Closes op-rs/op-reth#213 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add support for `eth_getProof` (op-rs/op-reth#257) Adds support for `eth_getProof` RPC method. This required reworking the launch command to still work with an alternative DB provider. Closes op-rs/op-reth#173 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implement proof collection e2e tests (op-rs/op-reth#231) Implements end to end tests for proof collection. These tests create a blockchain, run the backfill job, then run the live collector. The live collector executes on top of the backfilled state and it already checks that the state root matches, so the tests just ensure that it runs without errors. Fixes op-rs/op-reth#170 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * perf(trie): feature gate `reth-optimism-trie` metrics (op-rs/op-reth#282) Closes op-rs/op-reth#281 - Feature gates `reth-optimism-trie` metrics - Moves cursor impls out of proofs module into new module `cursor` - Moves cursor factory impls into new module `cursor_factory` - Updates cursor factory impls to return cursor types with metrics wrapper if metrics feature is enabled * chore(docs): Add missing doclinks (op-rs/op-reth#277) Adds doc links to all types to make sure they are caught by lint if names of types change in the future * fix(docs): Fixes docs of `reth_optimism_trie::cursor_factory` (op-rs/op-reth#286) Fixes module docs of `reth_optimism_trie::cursor_factory`, which weren't updated on moving the code out of the proofs module * fix(test): Enable live collector tests with metrics feature (op-rs/op-reth#291) Closes op-rs/op-reth#283 Enable live collector tests when metrics feature is enabled * fix(docs): Fixes doc links (op-rs/op-reth#292) Fixes some doc links from op-rs/op-reth#277 Several places should have linked to `OpProofsStore` which linked to `OpProofsStorage` * feat: Track eth_getProof metrics (op-rs/op-reth#285) Adds latency, total requests, successful responses, and error responses metrics to the eth_getProof RPC method. closes op-rs/op-reth#226 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: devnet setup (op-rs/op-reth#295) Closes op-rs/op-reth#200 * feat: implement `debug_executePayload` (op-rs/op-reth#276) Closes op-rs/op-reth#189 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: storage trie cursor (op-rs/op-reth#301) Closes op-rs/op-reth#299 * feat: latest block updates in `store_trie_updates` (op-rs/op-reth#304) Closes op-rs/op-reth#302 * feat: live collector integration (op-rs/op-reth#306) Closes op-rs/op-reth#296 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add mdbx to storage tests (op-rs/op-reth#308) Fixes a small issue where storage slots with a zero value weren't skipped. I implemented this above the BlockVersionedCursor since it only applies to the storage cursor. Also enables MDBX in the common storage tests and ensures that no storage tests insert block numbers out of order or insert duplicate keys. * fix: merge conflict caused by storage tests (op-rs/op-reth#313) Fix merge conflicts caused by storage API change. * feat: create index table to efficiently prune history tables (op-rs/op-reth#280) Closes op-rs/op-reth#219 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(trie): Move `OpProofsStorageError` to own module (op-rs/op-reth#317) Ref op-rs/op-reth#316 Moves `OpProofsStorageError` into own module for readability Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Named fields for OpProofsStorageError variants (op-rs/op-reth#319) Fixes op-rs/op-reth#315 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement `debug_executionWitness` (op-rs/op-reth#312) Add support for `debug_executionWitness` API. This hasn't been tested, but I'll be able to test it soon once the backfill job completes. This code is mostly copied from here and adapted to work with our state provider: https://github.com/op-rs/op-reth/blob/1470a9cc77af798c999da05d9541a7025e5d4924/crates/rpc/rpc/src/debug.rs#L643-L695 Fixes op-rs/op-reth#190 * chore(trie): Rename metrics scope (op-rs/op-reth#322) Ref op-rs/op-reth#321 Renames scope for `BlockMetrics` and `OperationMetrics` from `external_proofs` to `optimism_trie` * feat: implemented wiped storage handling on `store_trie_updates` (op-rs/op-reth#300) Closes op-rs/op-reth#259 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore(trie): Rename `OpProofsStorageError` variant (op-rs/op-reth#324) Ref op-rs/op-reth#315 Renames `OpProofsStorageError::BlockUpdateFailed` to `OpProofsStorageError::MissingParentBlock` which better matches the error message Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix(db): `is_parent check` in `store_trie_update` (op-rs/op-reth#309) Fixes op-rs/op-reth#305 * feat: fetch trie updates (op-rs/op-reth#327) Closes: op-rs/op-reth#256 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> * feat: implemented mdbx `replace_updates` (op-rs/op-reth#329) Closes op-rs/op-reth#193 Closes op-rs/op-reth#216 * feat: apply `BlockStateDiff` to the `prune_earliest_state` method (op-rs/op-reth#332) Closes op-rs/op-reth#217 Also fixed a bug where the `set_earliest_block_number_hash` was using `append` instead of `upsert`. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: refactor storage trie cursor (op-rs/op-reth#338) Closes op-rs/op-reth#337 * feat: integrate metrics into live sync process (op-rs/op-reth#330) Fixes op-rs/op-reth#246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com> * test: basic storage proof (op-rs/op-reth#336) Part 1 for op-rs/op-reth#289 * test: `eth_getProof` for account proof validation (op-rs/op-reth#341) Closes op-rs/op-reth#288 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * test: `debug_executePayload` RPC (op-rs/op-reth#340) Closes op-rs/op-reth#325 * test: multi slot and zero testing (op-rs/op-reth#343) Closes op-rs/op-reth#289 * test: resyncing (op-rs/op-reth#350) Closes op-rs/op-reth#349 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: e2e for `debug_executionWitness` (op-rs/op-reth#348) Closes op-rs/op-reth#326 Fixed the `nonce too low` issue with the way `execution_witness` was implemented for op. * test: doc test eg for exex setup (op-rs/op-reth#352) Closes op-rs/op-reth#342 * chore: enable remaining mdbx tests (op-rs/op-reth#354) Fixes op-rs/op-reth#252 * fix: ensure db tx is closed after backfill (op-rs/op-reth#357) The db tx isn't dropped ever for the backfill job, so create a temp scope to ensure it's dropped. * refactor: `MdbxProofStorage` (op-rs/op-reth#351) Closes op-rs/op-reth#335 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: rebase conflicts (op-rs/op-reth#367) Fix conflicts rebasing onto latest upstream main * feat: handle reorg (op-rs/op-reth#356) Closes op-rs/op-reth#191 --------- * feat: add target debug logs to reth_optimism_exex (op-rs/op-reth#368) Fixes op-rs/op-reth#366 * test: basic reorg (op-rs/op-reth#369) Part of op-rs/op-reth#174 * fix: missing block after reorg (op-rs/op-reth#382) Closes op-rs/op-reth#379 * feat: add sync status RPC (op-rs/op-reth#353) Adds `debug_proofsSyncStatus` for fetching the proof window when enabled. Also adds metrics for the earliest and latest block numbers available. * feat: replace variant OpProofsStorageError::Other (op-rs/op-reth#355) Fixes op-rs/op-reth#316 I also noticed a few `DatabaseError` that were being mapped to `OpProofsStorageError::Other` and changed them to point to `OpProofsStorageError::DatabaseError` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Implemented `OpProofStoragePruner` (op-rs/op-reth#360) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: handle `ChainReverted` notification (op-rs/op-reth#380) Closes op-rs/op-reth#192 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: small lint (op-rs/op-reth#390) Find a small typo here. * test: reorg test scenarios (op-rs/op-reth#384) Closes op-rs/op-reth#174 * test: add new devnet for op-reth as sequencer (op-rs/op-reth#391) Closes op-rs/op-reth#389 * chore(trie): Simplify cursor interface (op-rs/op-reth#388) Ref op-rs/op-reth#371 Removes redundant `OpProofsTrieCursorRO` and `OpProofsHashedCursorRO` traits by enabling a `DatabaseError` variant that wraps an error trait object. This way we can have a bijective relation between `DatabaseError` and `OpProofsStorageError` in a type safe way, i.e. conversion both ways without matching on string. cc @dhyaniarun1993 * feat: add panels for `eth_getProof` in grafana dashboard (op-rs/op-reth#392) Closes op-rs/op-reth#318 <img width="2208" height="672" alt="Screenshot 2025-11-21 at 15 35 16" src="https://github.com/user-attachments/assets/62ba3b36-fd4f-498a-9bbc-7e4f33c9e3d2" /> The OP-Reth dashboard is not loaded by default when running the kurtosis devnet for e2e historical proofs. Once op-rs/op-reth#391 is merged, I'll update the devnet config to load it by default. * fix: add custom TrieType to separate metrics for op proofs storage (op-rs/op-reth#397) Closes op-rs/op-reth#321 * feat: implement methods for new traits for `inmemory` cursor (op-rs/op-reth#396) Closes op-rs/op-reth#400 * fix: use loop instead of recursing in hashed next (op-rs/op-reth#402) Fixes a potential DoS vector of overflowing the stack by writing zero values to storage. Resolves op-rs/op-reth#401, op-rs/op-reth#394 * fix: store block updates metric fixed (op-rs/op-reth#409) Closes op-rs/op-reth#410 * feat: implemented `OpProofStorage` Database metrics (op-rs/op-reth#407) Closes op-rs/op-reth#224 Closes op-rs/op-reth#387 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add `initialize-op-proofs` command (op-rs/op-reth#377) Adds `initialize-op-proofs` which is now required before running the node with the proof ExEx. Fixes op-rs/op-reth#376 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: integrate `initialize-op-proofs` to devnet (op-rs/op-reth#412) Closes op-rs/op-reth#411 --------- Co-authored-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: dashboard for proof history related metrics (op-rs/op-reth#414) Closes op-rs/op-reth#413 Closes op-rs/op-reth#405 The new dashboard looks like following: <img width="2199" height="1086" alt="Screenshot 2025-11-27 at 18 19 36" src="https://github.com/user-attachments/assets/8d8f4f32-f1c3-4704-8ee4-ed67921ae039" /> <img width="2199" height="959" alt="Screenshot 2025-11-27 at 18 20 04" src="https://github.com/user-attachments/assets/33714ecc-5853-4a19-a519-4a35efff5cd9" /> --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: update metric on startup (op-rs/op-reth#419) Closes op-rs/op-reth#416 Issue: The issue was that CLI command `initialize-op-proofs` set the earliest block number but when the node starts, the metrics initialized are at default values. All other metrics get called soon after the start but not the earliest block number. Solution: Set the metric on startup, if available. * chore: code coverage for historical proof exex (op-rs/op-reth#415) Closes op-rs/op-reth#406 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: Add test for Display impl for PrunerOutput (op-rs/op-reth#435) Fixes op-rs/op-reth#432 * feat: add exex sync status in grafana (op-rs/op-reth#429) Closes op-rs/op-reth#426 <img width="1661" height="422" alt="Screenshot 2025-12-03 at 10 59 58" src="https://github.com/user-attachments/assets/91aeb374-a40a-45db-9c38-f9f36215747e" /> * feat: Consume TrieUpdates and HashedState from notification (op-rs/op-reth#439) ### Description This PR consumes trie updates and hashed state directly from notifications, falling back to full block execution when notification data is missing. ### Current limitations: There are known serialization/deserialization issues with the WAL. Before starting the node, ensure the WAL is cleaned. The Proof History ExEx is designed to tolerate missing notification data, but if you are running any additional ExEx components, ensure they can also safely handle gaps in notification data. Due to the above limitation, the resyncing test has been disabled. Closes op-rs/op-reth#420 Based on op-rs/op-reth#425 --------- Co-authored-by: Brian Picciano <me@mediocregopher.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: fall back to block execution if state updates missing from notification (op-rs/op-reth#455) Closes op-rs/op-reth#453 * feat: Implemented `OpProofStoragePrunerTask` (op-rs/op-reth#375) Closes op-rs/op-reth#361 Closes op-rs/op-reth#395 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented pruner metrics (op-rs/op-reth#454) Closes op-rs/op-reth#363 * fix: new earliest block after pruning (op-rs/op-reth#470) Fixes: op-rs/op-reth#469 * chore(exex): Add metrics feature in `reth-optimism-exex` (op-rs/op-reth#438) Closes op-rs/op-reth#427 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> * fix(trie): Add missing `ValueWithSubkey` impl for `VersionedValue<T>` (op-rs/op-reth#475) Closes op-rs/op-reth#474 * chore: add prune e2e test back (op-rs/op-reth#477) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: sysgo integration (op-rs/op-reth#478) Closes op-rs/op-reth#297 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: enable code coverage for e2e (op-rs/op-reth#485) Closes op-rs/op-reth#442 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(chain-state): Receive sorted updates to exex (op-rs/op-reth#464) Closes op-rs/op-reth#460 Makes use of the sorted updates which are now already available in the `ExecutedBlock`, instead of sorting out removed accounts/slots later in `reth-optimism-trie` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(trie): Cover error branches of `execute_and_store_block_updates` (op-rs/op-reth#437) Fixes op-rs/op-reth#436 * feat: unwind command added (op-rs/op-reth#499) Closes op-rs/op-reth#451 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(test): use `reth_optimism_chainspec::BASE_SEPOLIA` in tests (op-rs/op-reth#505) Update to use `reth_optimism_chainspec::BASE_SEPOLIA` in engine tests, now that the jovian timestamp is known * feat: prune cli added (op-rs/op-reth#507) Closes op-rs/op-reth#452 * test: add unit test for OpProofsStateProviderRef Debug impl (op-rs/op-reth#493) Adds unit test for the `Debug` implementation of `OpProofsStateProviderRef`. Closes op-rs/op-reth#434 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> * test: extended prune e2e with `getProof` consistency (op-rs/op-reth#514) Closes op-rs/op-reth#461 * feat: used `OpProofsStorageWithMetrics` provider on pruner (op-rs/op-reth#498) Closes op-rs/op-reth#472 * feat: pruning progress and metrics improvements (op-rs/op-reth#504) Closes op-rs/op-reth#362 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement prune batching (op-rs/op-reth#530) Closes op-rs/op-reth#364 * feat: address lookup table (op-rs/op-reth#509) Closes op-rs/op-reth#486 * feat: backfill address map (op-rs/op-reth#512) Closes op-rs/op-reth#510 * fix(trie): Fix broken `Debug` impl unit test (op-rs/op-reth#539) Fix broken unit test for verifying `Debug` impl, add missing map `address_mappings` to expected test output, introduced in op-rs/op-reth#512 * refactor(trie): return `OpProofsStorageError` from `store_block_updates` (op-rs/op-reth#537) Closes op-rs/op-reth#524 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: metrics panel for debug api ext (op-rs/op-reth#533) Closes op-rs/op-reth#515 Closes op-rs/op-reth#516 <img width="1607" height="277" alt="Screenshot 2025-12-18 at 20 44 35" src="https://github.com/user-attachments/assets/d135ff3c-c837-4dc8-a097-b9f818a08a35" /> --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * refactor(trie): return `OpProofsStorageError` from `execute_and_store_block_updates` (op-rs/op-reth#535) Closes op-rs/op-reth#523 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: initialize-op-proofs needs chain parameter (op-rs/op-reth#544) * fix: support Send in test_op_proofs_state_provider_ref_debug (op-rs/op-reth#546) * chore(trie): return `OpProofsStorageError` from `unwind_history` (op-rs/op-reth#548) Cherry picks commit from op-rs/op-reth#540 Co-authored-by: Himess <semihcvlk53@gmail.com> * refactor(trie): return OpProofsStorageError from unwind_and_store_block_updates (op-rs/op-reth#541) Closes op-rs/op-reth#525 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: integrated storing address mapping (op-rs/op-reth#534) Closes op-rs/op-reth#487 * chore: refactor op-proofs cli (op-rs/op-reth#554) Closes op-rs/op-reth#511 * feat: added startup safety check for massive pruning operation (op-rs/op-reth#556) Closes op-rs/op-reth#555 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix reorg e2e (op-rs/op-reth#558) Attempts to fix op-rs/op-reth#528 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: improve `append_or_delete_dup_sorted` (op-rs/op-reth#552) Closes op-rs/op-reth#501 * chore(trie): replace `eyre` with `OpProofsStorageError` (op-rs/op-reth#564) Closes op-rs/op-reth#527 * chore: revert hybrid storage experiment (op-rs/op-reth#568) Revert all the changes made as part of the hybrid storage experiment. * refactor(trie): extract save_hashed_accounts closure to method1 (op-rs/op-reth#570) Closes op-rs/op-reth#566 (1/5) Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: optimized history deletions (op-rs/op-reth#565) Closes op-rs/op-reth#311 Performace comparison under load test: ``` previous: 4.873750666s sorted delete: 3.278444083s sorted delete + seperate read (current as per the PR): 2.980065583s ``` * ci: restrict codecov workflows to merge_group and workflow_dispatch (op-rs/op-reth#513) ## Summary - Remove `push` and `pull_request` triggers from coverage workflows - Add `merge_group` trigger to run codecov only in merge queue - Keep `workflow_dispatch` for manual triggering when needed This reduces CI resource usage by running codecov only when necessary. Closes op-rs/op-reth#500 * feat: optimised initial state update for pruning (op-rs/op-reth#563) Closes op-rs/op-reth#562 Validated the optimization by rewriting an initial state with N=50,000 updated entries. - Throughput: Increased by ~20.9% (326k → 394k entries/sec). - Latency: Reduced by ~17.2% (153ms → 126ms). ``` Old approach: Rewrite Duration: 153.26ms Throughput: 326,248 entries/sec New approach: Rewrite Duration: 126.77ms Throughput: 394,410 entries/sec ``` bench: op-rs/op-reth@274e544 * docs: OpProofStorage DB Schema (op-rs/op-reth#583) Closes op-rs/op-reth#240 * chore: push devstack support for op-reth upstream (op-rs/op-reth#585) Part of op-rs/op-reth#483. First wait for theo's PR [op-rs/op-reth#18754](#18754) to be merged. Then wait for our PR [op-rs/op-reth#18772](#18772) to be merged. After that, in this repo, remove the current submodule and replace it with optimism itself. * feat: add verification interval for integrity check (op-rs/op-reth#577) Closes op-rs/op-reth#449 The approach used is to perform full block verification after every N blocks to ensure the state is still correct. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: backfill refactor (op-rs/op-reth#578) Closes op-rs/op-reth#566 --------- Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: update optimism (op-rs/op-reth#590) * chore(tire): Improve test coverage for `OpProofsStorageError` (op-rs/op-reth#589) Ref op-rs/op-reth#588 Adds tests for conversions to `OpProofsStorageError` * fix: db_stat after upstream pull (op-rs/op-reth#601) * test: unit test for exex (op-rs/op-reth#599) Closes op-rs/op-reth#458 * fix: `MdbxStorageCursor` non first key issue (op-rs/op-reth#602) Closes op-rs/op-reth#600 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> * chore: added exex missing tests (op-rs/op-reth#605) Closes op-rs/op-reth#588 * chore: getProof benchmark utility added (op-rs/op-reth#550) Utility for op-rs/op-reth#446 * feat: refactor and optimise pruning (op-rs/op-reth#587) Closes op-rs/op-reth#567 This PR improves the performance of the historical state pruning logic. By refining how state updates are handled and written during pruning, we observe a significant reduction in write latency. Performance Benchmarks: Comparison run using [load_test](https://github.com/op-rs/op-reth/blob/8f0b61039a1af5f07793d5ea38e21e0439b7f0b3/crates/optimism/trie/src/db/store.rs#L3462) Metric | Current (Baseline) | Optimised (This PR) | Improvement -- | -- | -- | -- Write Time | ~107ms | ~50ms | ~2.1x Faster Total Duration | ~129ms | ~81ms | ~1.6x Faster * fix: prune redundant tombstones at history horizon (op-rs/op-reth#620) Closes op-rs/op-reth#619 * feat: backfill job restart-safe and idempotent (op-rs/op-reth#594) Closes op-rs/op-reth#244 * chore(trie): Use `reth-trie-common` to enable `serde-bincode-compat` in v1.10.1 (op-rs/op-reth#617) Ref op-rs/op-reth#611 Uses `reth-trie-common` dep explicitly in `reth-optimism-trie` to make sure `serde-bincode-compat` can be enabled while using reth release v1.10.1 * feat: backfill renamed to initialize (op-rs/op-reth#622) Closes op-rs/op-reth#606 * feat: refactored store (op-rs/op-reth#623) Closes op-rs/op-reth#626 * chore: mv proof args to rollup node (op-rs/op-reth#625) Closes op-rs/op-reth#613 * chore(trie): Revert redundant diff (op-rs/op-reth#610) Ref op-rs/op-reth#611 Uses getters instead of direct field access for trie updates to remove redundant diff with upstream * chore: use reth versioned deps (op-rs/op-reth#627) Closes op-rs/op-reth#621 This PR updates the workspace configuration to use reth as a git dependency instead of a local crate, including necessary fixes for the compiler, CI workflows, and test configurations. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(trie): Fix lint (op-rs/op-reth#636) Lint fix. Clippy is catching since yesterday's new nightly release. * fix: removed unused pruner metrics (op-rs/op-reth#638) Closes op-rs/op-reth#607 * chore(trie): Remove redundant generic lifetime for `InitializationJob` (op-rs/op-reth#632) Ref op-rs/op-reth#631 - Changes nested database tx type to owned type as has no need to be ref - Uses `AsyncFnOnce`(new in Rust 1.85) instead of generic future return type * chore(trie): Make `initialize` a method of `InitializationJob` (op-rs/op-reth#635) Ref op-rs/op-reth#631 - Removes `safe_fn` closure as arg from `initialize` by making `initialize` a method of `InitializationJob` - Moves save functions out of `InitializationJob` into impls of trait method of new trait `InitTable::save_entries` - Uses `size_hint` to avoid cost of heap re-allocations - Removes redundant clone * refactor(trie): use parking_lot::RwLock in InMemoryProofsStorage (op-rs/op-reth#637) Closes op-rs/op-reth#633 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: async notification processing (op-rs/op-reth#614) This PR improves the robustness of the OP-Reth Proofs ExEx by process notifications asynchronously(to avoid WAL backpressure) when the gap between proof storage and the tip is big. This change ensures that notification pipeline is able to keep up and gracefully handle the gap. Closes op-rs/op-reth#597 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: simplify OpProofsStore (op-rs/op-reth#639) Closes: op-rs/op-reth#634 Blocked on: op-rs/op-reth#637 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: ExEx config builder (op-rs/op-reth#642) Closes op-rs/op-reth#641 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore(trie): Fix redundant type annotations (op-rs/op-reth#645) Fixes lint `clippy::redundant_type_annotations` * chore(trie): Replace redundant clone of entries in `InitializationJob` (op-rs/op-reth#644) Closes op-rs/op-reth#631 - Puts off assigning variables until right when needed - Optimizes away redundant clone of entries on batch insertion - Updates symbol name to better show initialising is a batching algorithm * chore: moved proof initialization to `reth-optimism-node` (op-rs/op-reth#640) Closes op-rs/op-reth#612 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: separate initial state store trait (op-rs/op-reth#647) Closes op-rs/op-reth#643 * chore(test): support reth as in sequencer and validator (op-rs/op-reth#650) This PR added support to run op-reth in sequencer and op-reth(with proof) in validator for the e2e test. Note: In this setup pruning test isn't supported(at this moment) since this test is falling outside the proof window of reth. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix flaky e2e test (op-rs/op-reth#653) Closes op-rs/op-reth#528 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(test): dep dump (op-rs/op-reth#654) Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(op-reth/historical-proofs): fix compilation errors --------- Signed-off-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Georgios Konstantopoulos <me@gakonst.com> Co-authored-by: Amp <amp@ampcode.com> Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com> Co-authored-by: Xzavier <xingqiang.yuan@okg.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Varun Doshi <doshivarun202@gmail.com> Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: einar-oplabs <oplabs@einar.io> Co-authored-by: Einar Rasmussen <einar@oplabs.co> Co-authored-by: op-will <232669456+op-will@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <emiliaha95@gmail.com> Co-authored-by: itschaindev <jagrutk@protonmail.com> Co-authored-by: Jonas Warlike <mikmillarog@protonmail.com> Co-authored-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Brian Picciano <me@mediocregopher.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: Sebastian Stammler <seb@oplabs.co>
danyalprout
pushed a commit
to base/base
that referenced
this pull request
Feb 20, 2026
…p-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #197
This PR implements the live state collector on top of the
OpProofsStateProvidercreated in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip.