Bedrock: new batch-derivation changes#2871
Merged
Conversation
|
Contributor
|
This PR changes implementation code, but doesn't include a changeset. Did you forget to add one? |
ops-bedrock/rollup.json
Outdated
|
|
||
| "seq_window_size": 2, | ||
|
|
||
| "channel_timeout": 10, |
Contributor
There was a problem hiding this comment.
10 here, 100s above (in docker compose)
protolambda
force-pushed
the
batch-derivation
branch
from
d752252 to
50c8967
Compare
Contributor
trianglesphere
left a comment
trianglesphere
left a comment
There was a problem hiding this comment.
I left a couple comments on packages other than op-node
I wrote most of op-batcher so I don't want to do most of the review
op-e2e lgtm
op-proposer lgtm
ops-bedrock has a slight inconsistency
Will help with splitting out refactor code from op-node & then start reviewing it
Contributor
|
Hey @protolambda! This PR has merge conflicts. Please fix them before continuing review. |
feat: bedrock inwards/outwards batch deriv draft v2 skip timed out frames, keep reading tx data ignore frame if it already exists, do not stop reading tx update pruning fix channel closing fix reorg recover func misc reorg func fixes channels for multiplexing submissions, inclusion for ordering of processing, decoded batches can be ordered as necessary after pulling from the stream ignore timed out frames fix maxBlocksPerChannel name fix var name, stop producing output data if no blocks are left implement channel out reader, start testing, renaming, structure etc. rename pipeline to channel-in-reader, fix old l2 package imports close compression stream improve channel out reader add de compression and rlp reading to channel-in-reader channel in reader: l1 origin update channel in reader updates move new deriv code into derive package work in progress integration of batch derivation changes work in progress, l2 derivation stepper fix rlp dependency channel in reader is broken, left todo update work in progress derivation pipeline with todo spec per function engine queue todo work in progress integration with driver fix channel in reader init driver event loop and derive loop separation (WIP) derive: Implement BatchQueue with full window This implements a simple algorithm for the batch queue. It waits until it has a full sequence window and then runs the historical batch derivation process over that data. The WIP part is that it needs more data that it does not yet have. derive: Fully derive payload attributes Also properly slices the queue. Remove batch bundle, split of reading of data from txs move engine update/consolidation into derive package tag channel bank with l1 origin as whole, read frame data may not revert to older l1 origin because out-of-order channel frames read full channel, forward L1 origin changes in channel-in-reader, don't block on batch reading engine queue engine queue work driver updates carry data between pipeline stages log sync progress wip init pipeline fetch l1 data as part of derivation pipeline init fix work in progress channel bank reset change channel bank resetting as part of pipeline define interfaces for stages, clean up l1 interface usage less trigger happy derivation pipeline resets, just reset when the pipeline says we need to test utils update driver snapshot usage, move L1Info interface to eth package, misc driver cleanup use channel emitter for api, fix build issues update batch submitter (work in progress, needs more testing) engine queue fix (@trianglesphere) find sync start reduce args, just get l2 head directly fix channel reader: don't attempt to read when there's no channel data yet log batcher and proposer in e2e channel emitter / channel out reader fixes fix channel emitter timeout fix channel reading end fix unexpected eof when buffer is not filled due to compressing layer also buffering add logging to batch filtering fix batch inputs, don't derive attributes before reading all batches of the origin all derivation pipeline stages now have the same Step and ResetStep interfaces misc open/close origin fixes and sync work fix test lint improve testutils, fix l1 mock, implement calldata source tests more mocking/testing utils, split l1 source/traversal, test first few stages improve mock test utils, don't use bignum in l2 api test pipeline per stage channel timeout config param, test channel bank fix batcher channel timeout flag new op-batcher new batcher in the op-node logging / disable parts of the op-node for testing fix off by one in batcher Close l1src stage Note: may want to pass the close further out / have more complex logic about open/close. logging + hacks to make the sequencer work & verifier half work change open/close origin api, fix genesis seq nr bug, e2e passing fix progress/origin naming, avoid engine api linear unwind in consolidation, fix batcher process closing remove old ChannelEmitter, remove ChannelOutReader in favor of ChannelOut, fix tests, clean up unused l2 engine change, clean up op-batcher flags fix op-batcher flags / docker compose update clean up logging lint test valid -> if err == nil, not err != nil L1Source -> L1Retrieval, fix receiver names wait for derivation to be idle before sequencing new block implement verifier and sequencer confirmation depth op-node: Add Epoch Hash to batch This commits a batch to a specific L1 origin block by hash rather than just by number. This help in the case of L1 reorgs by stopping batches from being applied in weird ways. fix missing epoch block hash batcher: Handle multiple frames per channel The batcher is still very simple generating a new channel full of L2 blocks since the last channel that it created, but it is just a tad smarter now in that if will handle the case of multiple frames per channel. This is the bare minimum functionality to handle happy case batching on a real network. The only other thing that it can't handle is reorgs, but it can now handle very larger L2 transactions and blocks.
protolambda
force-pushed
the
batch-derivation
branch
from
50c8967 to
adec5b4
Compare
Co-authored-by: Joshua Gutow <jgutow@optimism.io>
…seq window size to 4
protolambda
force-pushed
the
batch-derivation
branch
from
476291d to
dc42213
Compare
We were creating 65k predeploys, which made a huge genesis file. This PR reduces that to 2048.
Enables easy access to the logs of a running devnet.
theochap
added a commit
that referenced
this pull request
Dec 10, 2025
Fixes design inconsistency where sequencer keys could only be passed via CLI args while regular P2P keys support file loading. This adds `--p2p.sequencer.key.path` parameter to match the existing `--p2p.priv.path` pattern. This resolves the[ TODO mentioned](https://github.com/ethereum-optimism/optimism/blob/44d3429d2913c8c90fbb0848e14c1022b8ca2267/op-devstack/sysgo/l2_cl_kona.go#L196) in optimism's devstack where they noted the missing file support for sequencer keys. --------- Co-authored-by: theo <80177219+theochap@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR tracks all batch-derivation changes, starting a squashed version of the earlier dev branch
batch-deriv-v2(#2714).Op-node and op-e2e tests are passing, except an unrelated e2e fees test which is being looked at separately.
We'll split out some of the
eth/l2/testutilschanges into separate PRs to reduce the size of this one. And then likely review in smaller PRs that isolate changes more.