feat: basic staged sync crate by onbjerg · Pull Request #6 · paradigmxyz/reth

onbjerg · 2022-09-29T13:29:25Z

Basic crate intended to house staged syncing primitives (stage trait etc.) for use in reth stages

Some details are left out for now (for example stages take a &mut dyn Transaction) since it's an unknown quantity to me what the database abstraction will look like

Additionally I've left out the impl of Pipeline::run for a later PR

onbjerg · 2022-09-29T13:30:03Z

crates/staged-sync/src/lib.rs

using U64 here since BlockId in the primitives crate is either a hash or a number (we just need a number) and BlockNumber is either a tag or a number (and we just need a number)

We should probably start adding our own type in primitive crate, there is a difference between types for RPC and types for execution and I am not sure how much of them can be reused other than for jsonrpc. Can be done later

onbjerg · 2022-09-29T13:31:25Z

crates/staged-sync/src/lib.rs

Left out pruning stuff since we talked about seeing if it could fit in other components that are "closer" to the database (e.g. the database layer itself)

crates/staged-sync/Cargo.toml

crates/staged-sync/src/lib.rs

crates/staged-sync/Cargo.toml

gakonst · 2022-09-30T16:30:33Z

crates/stages/src/pipeline.rs

a bit sensitive of the dynamic dispatch here but maybe it's ok

gakonst · 2022-09-30T16:31:16Z

crates/stages/src/pipeline.rs

How is unwind priority determined? In the opposite order of normal stage priority?

What does require_tip do?

Looking at silkworm it seems two stages switched place on unwind. I am not sure if this is silkworm specific:
https://github.com/torquem-ch/silkworm/blob/3aedb286efe536062c9ffb4b5f5de9856866abab/node/silkworm/stagedsync/sync_loop.cpp#L88-L116

How is unwind priority determined? In the opposite order of normal stage priority?

We have to set it manually. Essentially stages may unwind in a different order than they are executed, and some stages may explicitly require unwinding before others and this helps that. Unwind priority works like this in Akula:

Unwind priority > 0: Higher unwind priority are unwound first

Unwind priority = 0: Unwound in the order that they were executed

An example:

Stage A (priority 0)
Stage B (priority 0)
Stage C (priority 2)
Stage D (priority 1)

Unwind order: Stage C, Stage D, Stage A, Stage B

I'm wondering if there is a better way to encapsulate the special handling of priority 0 though since it is a bit opaque

What does require_tip do?

A stage that has that flag set needs us to reach the tip of the chain before it will execute, otherwise it is skipped and the next stage is performed (permitted it also does not require the tip). My understanding is that currently the stages in Akula that require reaching the tip are every stage after the execution stage, but only if pruning is not enabled.

Added some more docs in the crate that hopefully clarifies it, otherwise I'm OK with adding more.

gakonst · 2022-09-30T16:32:34Z

crates/stages/src/pipeline.rs

Does unwinding mean: I am at block 10 and unwind_to is Some(5), it will undo blocks 10, 9 ... 5 in that order? It's like canceling each stage and undoing its progress? Do we understand when this happens?

Yes that is correct. It happens whenever a stage encounters some sort of validation failure (e.g. invalid hash, we're on the wrong fork, state is invalid etc etc), so essentially any time ExecOutput::Unwind is returned

crates/stages/Cargo.toml

rkrasiuk · 2022-10-03T09:30:02Z

crates/stages/src/pipeline.rs

+/// The unwind priority is set with [Pipeline::push_with_unwind_priority].
+#[derive(Default)]
+pub struct Pipeline {
+    stages: Vec<QueuedStage>,


shall we use VecDeque here?

Don't think that would make sense since we never remove stages from here, and we only access in either a forwards fashion or sorted by unwind priority

crates/stages/src/lib.rs

Resolution checkpoint Resolution checkpoint paradigmxyz#2 Resolution checkpoint paradigmxyz#3 x Resolution checkpoint paradigmxyz#4 Resolution checkpoint paradigmxyz#5 Resolution checkpoint paradigmxyz#6 Resolution checkpoint paradigmxyz#7 Resolution checkpoint paradigmxyz#8 Resolve checkpoint paradigmxyz#9 (transaction primitive) Resolve checkpoint paradigmxyz#10 (rpc api transactions) Resolve checkpoint paradigmxyz#11 (building w/o feature flag) Start review Compiling with and without `optimism` feature flag Remove `DepositTx` from txpool mock tests, they never go into the txpool fmt code lint fix signature tests Co-authored-by: nicolas <48695862+merklefruit@users.noreply.github.com> Use free CI runners (revert before upstream) Co-authored-by: refcell <abigger87@gmail.com> Signature test fixes Co-authored-by refcell <abigger87@gmail.com> Fix Receipt proptest Co-authored-by BB <brian.t.bland@gmail.com> lint Fix variable-length compact for txtype/transaction Co-authored-by: Brian Bland <brian.t.bland@gmail.com> Fix basefee tests Remove unnecessary rpc deps Co-authored-by: Brian Bland <brian.t.bland@gmail.com> Co-authored-by: refcell <abigger87@gmail.com> Co-authored-by: nicolas <48695862+merklefruit@users.noreply.github.com> Co-authored-by: Roberto <bayardo@alum.mit.edu>

* add feature 'enable_execution_duration_record' * set 'enable_execution_duration_record' feature for util mod

…xyz#6) Separate the cache layer for scheduler and partition, and implement fall back to sequential execute.

* fix: adjust ci workflows * ci: pin clippy to working version (paradigmxyz#11237) * ci: pin clippy to working version (paradigmxyz#11401) * fix integration tests --------- Co-authored-by: Federico Gimenez <fgimenez@users.noreply.github.com> Co-authored-by: nk_ysg <nk_ysg@163.com>

…igmxyz#6)

…unwind loop - Documented P0 bug: Execution stage unwinds to 0 after ~500K blocks - Pipeline successfully syncs Headers/Bodies/SenderRecovery - Execution starts, reaches ~500K, then unwinds and restarts - Added investigation steps and potential causes

…aradigmxyz#7 slow re-download - Issue paradigmxyz#6 downgraded P0→P3: bad FCU feeder was the cause (all 3 hashes same) - Fixed reth-fcu-feeder.py to use genesis as finalizedBlockHash - Pipeline completes ALL 13 stages with single FCU (1M+ blocks!) - Added Issue paradigmxyz#7: reverse headers downloader extremely slow after restart - Updated status and strategy

onbjerg commented Sep 29, 2022

View reviewed changes

mattsse reviewed Sep 29, 2022

View reviewed changes

crates/staged-sync/Cargo.toml Outdated Show resolved Hide resolved

crates/staged-sync/src/lib.rs Outdated Show resolved Hide resolved

crates/staged-sync/src/lib.rs Outdated Show resolved Hide resolved

rakita reviewed Sep 29, 2022

View reviewed changes

crates/staged-sync/Cargo.toml Outdated Show resolved Hide resolved

gakonst approved these changes Sep 30, 2022

View reviewed changes

onbjerg and others added 5 commits September 30, 2022 14:19

feat: basic staged sync crate

3a080f5

refactor: rename to reth-stages

176e446

feat: remove eyre

2d7cabf

feat: impl Error for StageError

42969a2

chore: bump cargo lock

7e6f228

gakonst force-pushed the onbjerg/ssync-crate branch from 3967724 to 7e6f228 Compare September 30, 2022 21:20

rakita reviewed Oct 3, 2022

View reviewed changes

crates/stages/Cargo.toml Outdated Show resolved Hide resolved

rkrasiuk approved these changes Oct 3, 2022

View reviewed changes

rakita reviewed Oct 3, 2022

View reviewed changes

crates/stages/src/lib.rs Outdated Show resolved Hide resolved

onbjerg added 2 commits October 3, 2022 14:38

chore: nits

b5e6a60

docs: clarify unwind

d76b368

onbjerg merged commit 6868dda into master Oct 3, 2022

onbjerg deleted the onbjerg/ssync-crate branch October 3, 2022 12:40

anonymousGiga pushed a commit to anonymousGiga/reth that referenced this pull request Feb 20, 2024

add feature 'enable_execution_duration_record' (paradigmxyz#6)

e26e1b0

* add feature 'enable_execution_duration_record' * set 'enable_execution_duration_record' feature for util mod

AshinGau added a commit to AshinGau/reth that referenced this pull request Sep 11, 2024

grevm: separate the cache layer for scheduler and partition (paradigm…

b9dab64

…xyz#6) Separate the cache layer for scheduler and partition, and implement fall back to sequential execute.

AshinGau added a commit to AshinGau/reth that referenced this pull request Oct 13, 2024

grevm: separate the cache layer for scheduler and partition (paradigm…

51823fb

…xyz#6) Separate the cache layer for scheduler and partition, and implement fall back to sequential execute.

anitaltius pushed a commit to Altius-Labs/1.2.2_db that referenced this pull request May 17, 2025

perf(trie): reduce iops during parallel multiproof calculation (parad…

37432f6

…igmxyz#6)

pycckuu mentioned this pull request Jun 5, 2025

engine_forkchoiceUpdatedV{X} returns ⁠VALID with ⁠payloadId: null when targeting a valid historical block with ⁠payloadAttributes, preventing reorg block production. #16690

Closed

1 task

yongkangc mentioned this pull request Feb 11, 2026

perf: collect-based fast path for small block tx iterator #22078

Closed

joohhnnn mentioned this pull request Mar 5, 2026

perf(cli): use sliding window for download ETA calculation #22816

Open

Conversation

onbjerg commented Sep 29, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

onbjerg Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

onbjerg Oct 3, 2022 •

edited

Loading