Skip to content

feat(engine): support dual-frontier partial trie persistence#23592

Closed
mediocregopher wants to merge 78 commits into
mainfrom
mediocregopher/partial-persistence
Closed

feat(engine): support dual-frontier partial trie persistence#23592
mediocregopher wants to merge 78 commits into
mainfrom
mediocregopher/partial-persistence

Conversation

@mediocregopher
Copy link
Copy Markdown
Member

@mediocregopher mediocregopher commented Apr 16, 2026 •
edited
Loading

Tracks partial trie durability in FinishCheckpoint.partial_state_trie and teaches engine persistence to plan around separate non-trie and trie frontiers. Updates provider save, read, and unwind paths to anchor overlays to the durable trie frontier, only mask trie writes for deferred blocks whose non-trie data is durably written, and preserve the partial trie frontier across truncation. This keeps historical state reconstruction, overlay caching, and engine unwind behavior correct when trie persistence lags the fully durable Finish checkpoint.

@mediocregopher
feat(trie): add disjoint_by_keys for sorted overlays
9883b71 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9639-935f-73a4-ba50-7682a7b9aca0
Co-authored-by: Amp <amp@ampcode.com>
@github-project-automation github-project-automation Bot moved this to Backlog in Reth Tracker Apr 16, 2026
@mediocregopher mediocregopher self-assigned this Apr 16, 2026
@mediocregopher mediocregopher added A-trie Related to Merkle Patricia Trie implementation C-perf A change motivated by improving speed, memory usage or disk footprint labels Apr 16, 2026
@mediocregopher mediocregopher marked this pull request as draft April 16, 2026 12:49
mediocregopher and others added 21 commits April 16, 2026 12:55
@mediocregopher
feat(stages): add partial finish checkpoint field
037828f 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9639-935f-73a4-ba50-7682a7b9aca0
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
feat(stages): move partial trie progress into finish checkpoint
b97544a 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d968b-970d-735e-844c-b83ff245ce05
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(node): unwind startup to partial trie checkpoint
761acad 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d968b-970d-735e-844c-b83ff245ce05
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(provider): simplify save_blocks trie masking API
e2bd518 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9728-846b-7418-8a69-ddbba65ce656
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
docs(provider): explain save_blocks trie masking range
84b5205 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9728-846b-7418-8a69-ddbba65ce656
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(engine): raise persistence defaults
ca35283 
Raise the default persistence threshold to 10 and derive the default backpressure threshold from the persistence and in-memory thresholds. Enable alloy getrandom for reth-engine-primitives tests so the existing B256::random() test coverage keeps compiling.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9a2b-716e-774d-8db7-b0308fa96a23
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): persist partial trie finish checkpoint
9e38dde 
Make masked save_blocks persistence track partial trie progress in the Finish checkpoint and switch the trie-masking API to a masked-suffix start index so the final block is always covered by the mask when partial trie persistence is used.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9a2b-716e-774d-8db7-b0308fa96a23
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(engine): retain partial trie suffix after persistence
ca20cc1 
Return finish-stage partial trie progress from the persistence thread and keep blocks above that trie boundary resident in memory after persistence completes.

This preserves the old prune-through-tip behavior when the partial trie matches the persisted tip or is absent.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9ab4-1eab-713f-9d5d-71903b7bc724
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
feat(engine): add dual-frontier persistence planning
c757a31 
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
@mediocregopher
refactor(chain-state): derive lazy overlay anchor from blocks
be4e8cd 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab40-7d65-709b-90d6-98965c5c6a65
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(chain-state): address lazy overlay review feedback
cacb69a 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab40-7d65-709b-90d6-98965c5c6a65
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(chain-state): cache lazy overlays by anchor
ebfaa6f 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab6f-7cee-755e-9f9c-309ae0b8517c
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(provider): resolve lazy overlay anchors at use time
5041d55 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab6f-7cee-755e-9f9c-309ae0b8517c
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(provider): separate overlay anchors from revert state
812e479 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(provider): infer overlay anchors from sources
5036eb5 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(provider): thread explicit requested anchors
b5ad001 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): anchor overlay state providers by hash
d92ad5a 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafa5-05ee-739e-a0b6-037cc30e4a0e
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): pass overlay anchors via constructor
134a7f3 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(engine): anchor state-root test overlay factory
7db14d0 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(trie): initialize test overlay anchors
45db5e0 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): satisfy overlay lint checks
ffb0587 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
mediocregopher and others added 22 commits May 4, 2026 10:56
@mediocregopher
fix(trie): stop tracing post-drop merkle rebuild
c7d9fd6 
Keep restart trace capture on the failed unwind path, but run the post-drop Merkle rebuild at info level without a separate trace artifact.
This keeps the repro focused on restart-trace.log while preserving the existing post-drop verification step.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df291-fde0-73c9-ad81-f516190fc3ad
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore(provider): improve save_blocks trie debug logging
3c667b1 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df3d9-cf72-713e-8a8e-9d3ab36f83f0
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(trie): preserve equivalent masked trie nodes
ecbf6ba 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df406-d380-704b-8376-32b9bbcbce7c
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(scripts): make Hoodi persistence repro marker detection robust
9040aae 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df406-d380-704b-8376-32b9bbcbce7c
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(scripts): randomize Hoodi crash target block
72b0666 
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df4c4-1820-728c-a2c2-bae7d3a17ae3
@mediocregopher
feat(trie): add walker option for matching branch children
f9e3da0 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df7cd-9370-777b-9be4-b151c795829b
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
refactor(trie): rename walker branch child option
3fab9bf 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df7cd-9370-777b-9be4-b151c795829b
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
feat(stages): enable full changed-branch walks during merkle unwind
91c25f4 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df7cd-9370-777b-9be4-b151c795829b
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(trie): simplify disjointed merge masking
8db3ac6 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df84d-0048-705b-97b4-e199ea9ddf6f
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Merge remote-tracking branch 'origin/main' into mediocregopher/partia…
eb2c8b8 
…l-persistence

Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
@mediocregopher
test(scripts): add hoodi reorg repro
a0dc39c 
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
@mediocregopher
chore(engine): log overlay construction details
636d8f7 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): revert overlays from finish frontier
92e4a15 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): reject overlays ahead of partial trie
6e597b5 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore(engine): log payload builder overlay frontiers
33f55b9 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(provider): allow lazy overlays from partial trie
53a6b9f 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore(engine): log payload builder state providers
a27db17 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore(scripts): capture provider overlay logs
4f1f8e3 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(scripts): narrow reorg mismatch detection
08e61b1 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore: expand overlay repro logging
ad06261 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019df965-c32c-7188-831c-c5d6ea37e036
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
chore: add Hoodi reorg diagnostics
6e574bf 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e02e7-d893-749b-9ba7-fde95abec62f
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Merge origin/main into pr-23592
70f157c 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e02e7-d893-749b-9ba7-fde95abec62f
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Copy link
Copy Markdown
Member Author

mediocregopher commented May 7, 2026

derek bench big-blocks blocks=500 slack=on-win baseline-args='--engine.persistence-threshold=13' feature-args='--engine.persistence-threshold=13 --engine.deferred-trie-blocks=10'

@decofe
Copy link
Copy Markdown
Member

decofe commented May 7, 2026 •
edited
Loading

cc @mediocregopher

✅ Benchmark complete! View job

Benchmark Results

⚠️ Feature is 1 commit behind main. Consider rebasing for accurate results.

Metric main mediocregopher/partial-persistence Change
Mean 657.29ms 703.13ms +6.97% ❌ (±0.37%)
StdDev 116.45ms 123.70ms
P50 627.66ms 673.48ms +7.30% ❌ (±0.90%)
P90 801.54ms 858.31ms +7.08% ❌ (±1.55%)
P99 1080.32ms 1166.67ms +7.99% ❌ (±3.40%)
Mgas/s 1588.53 1483.92 -6.59% ❌ (±0.36%)
Wall Clock 665.24s 711.30s +6.93% ❌ (±0.37%)
Persist Wait 295.08ms 729.43ms +147.20% ❌ (±33.52%)

500 big blocks, driver: reth-bench (fallback: big-blocks), 20 warmup

Wait Time Breakdown

Persistence Wait

Metric main mediocregopher/partial-persistence
Mean 295.08ms 729.43ms
P50 0.05ms 0.05ms
P95 3908.63ms 9927.24ms

Trie Cache Update Wait

Metric main mediocregopher/partial-persistence
Mean 0.09ms 0.60ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Execution Cache Update Wait

Metric main mediocregopher/partial-persistence
Mean 0.00ms 0.00ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Latency, Throughput & Diff

Wait Time Breakdown

Wait Time Breakdown

Gas vs Latency

Gas vs Latency

Grafana Dashboard

View real-time metrics

mediocregopher and others added 3 commits May 8, 2026 18:18
@mediocregopher
fix(reth-bench): preserve replay payload topology
eb3f527 
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
@mediocregopher
chore: replay Hoodi reorg artifacts in repro script
3327f0d 
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
@mediocregopher
fix(engine): catch up trie state before disk reorg removal
694b6b4 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e108f-c77e-712f-a377-a0148ea5feb9
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Copy link
Copy Markdown
Member Author

mediocregopher commented May 10, 2026

Closed in favor of the cleaned replacement PR: #24100

@github-project-automation github-project-automation Bot moved this from Backlog to Done in Reth Tracker May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Rjected Rjected Awaiting requested review from Rjected Rjected is a code owner

@shekhirin shekhirin Awaiting requested review from shekhirin shekhirin is a code owner

@joshieDo joshieDo Awaiting requested review from joshieDo joshieDo will be requested when the pull request is marked ready for review joshieDo is a code owner

@mattsse mattsse Awaiting requested review from mattsse mattsse will be requested when the pull request is marked ready for review mattsse is a code owner

@klkvr klkvr Awaiting requested review from klkvr klkvr will be requested when the pull request is marked ready for review klkvr is a code owner

@gakonst gakonst Awaiting requested review from gakonst gakonst will be requested when the pull request is marked ready for review gakonst is a code owner

Assignees

@mediocregopher mediocregopher

Labels

A-trie Related to Merkle Patricia Trie implementation C-perf A change motivated by improving speed, memory usage or disk footprint

Projects

Reth Tracker
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mediocregopher @decofe