feat(op-reth-hist-proofs): Import op-reth historical proofs code from op-rs fork#19044
Draft
emhane wants to merge 219 commits intoemhane/import-historical-proofsfrom
Draft
feat(op-reth-hist-proofs): Import op-reth historical proofs code from op-rs fork#19044emhane wants to merge 219 commits intoemhane/import-historical-proofsfrom
emhane wants to merge 219 commits intoemhane/import-historical-proofsfrom
Conversation
emhane
changed the base branch from
develop
to
emhane/import-historical-proofs
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
Fixes #177 I added a few methods that should allow atomically pruning and reorging blocks. Each function should represent a single transaction. We can store trie nodes, reorg, prune in a single transaction now. I kept the bulk insert methods available for storing the current state. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
Ref op-rs/op-reth#176 This PR adds an in-memory storage backend and tests that work with any storage backend. We can easily add more test cases to the file to test against a future SQLite backend. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Based on op-rs/op-reth#183 This PR adds a backfill job that accepts a DB transaction and copies the current state to the database. The transaction ensures we see a consistent view of the database at the current block, even if the node is syncing. This requires `--db.read-transaction-timeout 0`. This currently doesn't handle interrupting the job because the state may update while syncing and may read a different version of the database upon restart. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Closes #209
Description: - This PR adds scaffold MDBX proofs storage with impl to unblock parallel work. - Unblock multiple developers and avoid merge conflicts while the MDBX backend is being built. - Provide a compilable stub that wires types and env init, without committing to final storage logic.
…e` (#204) - Adds new crates `reth-optimism-exex` and `reth_optimism_trie` - Moves `reth_exex::external_proofs` -> `reth_optimism_exex` - Moves `reth_exex::external_proofs::storage` -> `reth_optimism_trie`
emhane
force-pushed
the
emhane/import-hist-proofs
branch
from
305b47d to
ad3e0a1
Compare
Seems like when `reth-optimism-trie` depends on `reth-db-api` indirectly, the doctest crate couldn’t resolve `reth_db_api::…` and failing. Closes #233 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
This PR separates storage and account cursors which makes sense because these two cursors may have very different seek/next implementations. In the case of in-memory, a single cursor impl can still handle both, but for MDBX, it makes sense to implement separate cursors. Closes #236 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
Based on op-rs/op-reth#203 , op-rs/op-reth#204 This PR implements `StateProvider` given a `OpProofsStorage` instance. It reads most data from the external database, but falls back on the latest provider for block hashes and code by hash similar to the existing historical provider in Reth. This is an important part to implementing live syncing since we're running the sync process on the DB being created. In the Reth implementation, `proof.rs` is contained in `trie/db`, so I think it makes sense to go in our DB crate. The `provider.rs` is in the `reth-provider` crate, but I don't think a separate provider crate helps us here, so I think we should also include that in the trie crate as well. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Closes #164 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
Fixes #225 (does not track errors as these are fatal currently)
Fixes #237 Adds more concrete error variants for `OpProofsStorageError` Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
Fixes #247 Removes the `block_number` parameter from: - store_account_branches - store_storage_branches - store_hashed_accounts - store_hashed_storages Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
Ref op-rs/op-reth#241 - Fixes comment from deps fix in op-rs/op-reth#229 (fix correct but comment misleading) - `serde-bincode-compat` is a feature which must be enabled when the `bincode` dependency is used. Asides for dev-deps, this only happens in `reth-stages`. `reth-exex` runs in `reth-stages`. In this wokrstream we use the `TrieUpdate` in exex. `serde-bincode-compat` is already implemented for `TrieUpdates` in `reth-trie-common`, this PR simply makes that feature accessible via `reth-trie` and in turn `reth-optimism-trie`, in order to enable it in `reth-optimism-exex`.
Closes #212 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Ref op-rs/op-reth#611 Uses getters instead of direct field access for trie updates to remove redundant diff with upstream
Closes op-rs/op-reth#621 This PR updates the workspace configuration to use reth as a git dependency instead of a local crate, including necessary fixes for the compiler, CI workflows, and test configurations. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Lint fix. Clippy is catching since yesterday's new nightly release.
#632) Ref op-rs/op-reth#631 - Changes nested database tx type to owned type as has no need to be ref - Uses `AsyncFnOnce`(new in Rust 1.85) instead of generic future return type
Ref op-rs/op-reth#631 - Removes `safe_fn` closure as arg from `initialize` by making `initialize` a method of `InitializationJob` - Moves save functions out of `InitializationJob` into impls of trait method of new trait `InitTable::save_entries` - Uses `size_hint` to avoid cost of heap re-allocations - Removes redundant clone
Closes #633 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
This PR improves the robustness of the OP-Reth Proofs ExEx by process notifications asynchronously(to avoid WAL backpressure) when the gap between proof storage and the tip is big. This change ensures that notification pipeline is able to keep up and gracefully handle the gap. Closes #597 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com>
Closes: #634 Blocked on: op-rs/op-reth#637 --------- Co-authored-by: Himess <semihcvlk53@gmail.com>
Closes #641 --------- Co-authored-by: Himess <semihcvlk53@gmail.com>
Fixes lint `clippy::redundant_type_annotations`
#644) Closes op-rs/op-reth#631 - Puts off assigning variables until right when needed - Optimizes away redundant clone of entries on batch insertion - Updates symbol name to better show initialising is a batching algorithm
Closes #612 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com>
This PR added support to run op-reth in sequencer and op-reth(with proof) in validator for the e2e test. Note: In this setup pruning test isn't supported(at this moment) since this test is falling outside the proof window of reth. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Closes #528 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com>
emhane
force-pushed
the
emhane/import-hist-proofs
branch
from
b63bc89 to
2870c9a
Compare
Contributor
Author
|
status update: all code from op-rs/op-reth stable should now be on this branch. next up is to translate GitHub workflows to circle ci. |
opgitgovernance
added
the
S-stale
Contributor
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.