op-chain-ops: Add parallel migration#5151
Merged
mergify[bot] merged 1 commit intodevelopfrom Mar 16, 2023
Merged
Conversation
|
mslipper
force-pushed
the
feat/parallel-migration
branch
4 times, most recently
from
af443ee to
e904a02
Compare
mslipper
force-pushed
the
feat/parallel-migration
branch
2 times, most recently
from
c63da3f to
cd013d5
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## develop #5151 +/- ##
===========================================
- Coverage 40.45% 36.59% -3.86%
===========================================
Files 353 211 -142
Lines 21570 17716 -3854
Branches 776 0 -776
===========================================
- Hits 8727 6484 -2243
+ Misses 12175 10592 -1583
+ Partials 668 640 -28
Flags with carried forward coverage won't be shown. Click here to find out more.
|
mslipper
force-pushed
the
feat/parallel-migration
branch
from
cd013d5 to
acd8d4d
Compare
✅ Deploy Preview for opstack-docs ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
tynes
reviewed
Mar 15, 2023
tynes
reviewed
Mar 15, 2023
tynes
approved these changes
Mar 15, 2023
mslipper
force-pushed
the
feat/parallel-migration
branch
from
acd8d4d to
b2376eb
Compare
Contributor
|
Semgrep found 1
Generic API Key detected Created by detected-generic-api-key. |
After some research, I discovered that we can iterate over Geth's storage in parallel as long as we don't share the state database. This PR updates the OVM ETH check script to parallelize state iteration across 64 workers. The parallelization works by partitioning storage keyspace based on the number of workers. To optimize further, I also moved the state balance checking into the main iteration loop to avoid unnecessary iteration. Note that state mutation must be done serially. Overall, this should safe about 40 minutes during the migration. This PR was tested by running it against a mainnet data directory. The entire process took approximately one hour. As part of this testing I discovered an invalid check in `check.go`, which was verifying the wrong storage slot in the withdrawals check function. This has been fixed.I confirmed with Mark that the updated `MessageSender` value is the correct one to be checking for. The filtering code was correct from the beginning.
mslipper
force-pushed
the
feat/parallel-migration
branch
from
b2376eb to
47dada2
Compare
Contributor
|
This PR has been added to the merge queue, and will be merged soon. |
Contributor
|
This PR is next in line to be merged, and will be merged as soon as checks pass. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After some research, I discovered that we can iterate over Geth's storage in parallel as long as we don't share the state database. This PR updates the OVM ETH check script to parallelize state iteration across 64 workers. The parallelization works by partitioning storage keyspace based on the number of workers. To optimize further, I also moved the state balance checking into the main iteration loop to avoid unnecessary iteration. Note that state mutation must be done serially. Overall, this should safe about 40 minutes during the migration.
This PR was tested by running it against a mainnet data directory. The entire process took approximately one hour. As part of this testing I discovered an invalid check in
check.go, which was verifying the wrong storage slot in the withdrawals check function. This has been fixed.I confirmed with Mark that the updatedMessageSendervalue is the correct one to be checking for. The filtering code was correct from the beginning.Builds on #5147.