Skip to content

fix(storage): reth db migrate-v2 for pruned nodes#23716

Merged
shekhirin merged 7 commits into
mainfrom
alexey/fix-update-index-no-prior-segment
Apr 27, 2026
Merged

fix(storage): reth db migrate-v2 for pruned nodes#23716
shekhirin merged 7 commits into
mainfrom
alexey/fix-update-index-no-prior-segment

Conversation

@shekhirin
Copy link
Copy Markdown
Member

@shekhirin shekhirin commented Apr 24, 2026 •
edited
Loading

  1. For a static file that has a range 0..=10 we can't write block 2 without writing blocks 0..=1 first. The fix is to write empty data for these blocks, we do it already in Execution stage for receipts / changesets.
  2. When we open a static files writer on block 500_001, it creates a new empty jar for range 500_000..=999_999, and tries to update the index with the max existing block being 499_999 (thinking that it exists), but it doesn't because it's a pruned node that shouldn't have this static file by design, so it crashes. The fix here is to not crash, skipping the index update.
  3. When migrating storage changesets, we initialized the walker as cursor.walk(Some(Default::default()))?.peekable(), which is not correct if the first block is non-zero.

Drive-by: add count field to receipts migration log

@shekhirin
fix(storage): fix migrate-v2 for pruned nodes
ab10772 
writer.update_index() synthesized segment_max_block = expected_block_start - 1
for empty jars, causing manager.update_index() to NippyJar::load a previous
range file that may not exist. Check that the previous file actually exists
before referencing it; if not, pass None (no-op for the index).

Also pad empty changesets/blocks in migrate-v2 when the prune checkpoint is
past the static file range boundary.
@github-project-automation github-project-automation Bot moved this to Backlog in Reth Tracker Apr 24, 2026
@shekhirin shekhirin marked this pull request as ready for review April 24, 2026 15:57
joshieDo
joshieDo reviewed Apr 24, 2026
View reviewed changes
Comment thread crates/cli/commands/src/db/migrate_v2.rs Outdated
joshieDo
joshieDo reviewed Apr 24, 2026
View reviewed changes
Comment thread crates/cli/commands/src/db/migrate_v2.rs Outdated
joshieDo
joshieDo reviewed Apr 24, 2026
View reviewed changes
Comment thread crates/cli/commands/src/db/migrate_v2.rs Outdated
@shekhirin shekhirin force-pushed the alexey/fix-update-index-no-prior-segment branch 3 times, most recently from e54f4ab to 1660da3 Compare April 24, 2026 19:49
@shekhirin shekhirin force-pushed the alexey/fix-update-index-no-prior-segment branch from 1660da3 to 756d367 Compare April 24, 2026 19:50
@shekhirin shekhirin added C-bug An unexpected or incorrect behavior A-db Related to the database labels Apr 25, 2026
@shekhirin
fix(db): start StorageChangeSets walker from first_block in migrate-v2
9404e1a 
Previously the walker started from Default::default() (block 0, address 0x0),
which produced count=0 when a StorageHistory prune checkpoint existed because
the for loop started at first_block while the walker was stuck at block 0.
@shekhirin shekhirin force-pushed the alexey/fix-update-index-no-prior-segment branch from d6eb888 to 9404e1a Compare April 25, 2026 10:17
@shekhirin shekhirin mentioned this pull request Apr 25, 2026
Closed
@DaniPopes DaniPopes requested a review from joshieDo April 25, 2026 14:24
@github-project-automation github-project-automation Bot moved this from Backlog to In Progress in Reth Tracker Apr 27, 2026
@shekhirin shekhirin added this pull request to the merge queue Apr 27, 2026
Merged via the queue into main with commit db17c89 Apr 27, 2026
36 checks passed
@shekhirin shekhirin deleted the alexey/fix-update-index-no-prior-segment branch April 27, 2026 07:24
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Reth Tracker Apr 27, 2026
sieniven pushed a commit to okx/reth that referenced this pull request Apr 28, 2026
@shekhirin @sieniven
fix(storage): reth db migrate-v2 for pruned nodes (paradigmxyz#23716)
ccbef3f
@decofe decofe mentioned this pull request Apr 29, 2026
Merged
umeeSthein pushed a commit to umeeSthein/tempo that referenced this pull request Apr 30, 2026
@decofe @klkvr
deps: update reth from main (2026-04-29) (tempoxyz#3714)
135ecef 
Automated nightly update of reth dependencies from `paradigmxyz/reth`
main branch.

## Upstream reth changes


[`7839f3d...73ec2c9`](paradigmxyz/reth@7839f3d...73ec2c9)

🔗 Amp thread:
https://ampcode.com/threads/T-019dd89b-f659-75eb-a3f2-3e769d63297f
**Engine**
- Spawn BAL hashed state before storage prefetch
([#23761](paradigmxyz/reth#23761)) and disable
BAL parallel execution by default
([#23764](paradigmxyz/reth#23764))
- Add CLI flag to disable BAL storage prefetch
([#23770](paradigmxyz/reth#23770))
- Log transient invalid header cache skips
([#23711](paradigmxyz/reth#23711))
- Add `DecodedBal` in `ExecutionEnv`
([#23675](paradigmxyz/reth#23675))

**RPC**
- Clean up eth state cache reorg entries
([#23683](paradigmxyz/reth#23683))
- Pass block timestamp to txn
([#23700](paradigmxyz/reth#23700))
- Include block numbers in `BlockRangeExceedsHead` error
([#23720](paradigmxyz/reth#23720))

**Network / P2P**
- Add Basic in-memory BAL store and BAL request to block access list
requests ([#23682](paradigmxyz/reth#23682),
[#23710](paradigmxyz/reth#23710))
- Enforce BAL response soft limit and apply count cap to handler
([#23725](paradigmxyz/reth#23725),
[#23754](paradigmxyz/reth#23754))
- Avoid RLP-decoding `NewBlock` payloads
([#23712](paradigmxyz/reth#23712)) and bound
memory footprint of p2p messages
([#23718](paradigmxyz/reth#23718))
- Memory-bound channel between network and tx manager
([#23802](paradigmxyz/reth#23802))
- Track unknown tx types in announcement metrics
([#23688](paradigmxyz/reth#23688))
- Retain active session buffer capacity
([#23702](paradigmxyz/reth#23702)) and respect
peer requirements for fetch followups
([#23706](paradigmxyz/reth#23706))
- Support binding discv5 and discv4 to the same port
([#23613](paradigmxyz/reth#23613)) and use Weak
reference in discv5 kbuckets bg task to release port on shutdown
([#23282](paradigmxyz/reth#23282))

**Trie**
- Skip DB seek on exact overlay hits
([#23559](paradigmxyz/reth#23559))
- Account for heap-allocated blinded hashes in `SparseNode::memory_size`
([#23726](paradigmxyz/reth#23726))

**DB / Storage**
- `reth db migrate-v2` for pruned nodes
([#23716](paradigmxyz/reth#23716))
- Pass `ExecutedBlocks` to `OverlayBuilder`, reduce reverts queried
([#23657](paradigmxyz/reth#23657))
- Add empty request check to storage values
([#23714](paradigmxyz/reth#23714))
- Derive `Eq` for `IntegerList`
([#23709](paradigmxyz/reth#23709)) and reorder
unix deps after strum
([#23697](paradigmxyz/reth#23697))

**Provider / EVM**
- Use overlay builders in historical state paths
([#23667](paradigmxyz/reth#23667))
- Return gas output from block builder
([#23744](paradigmxyz/reth#23744))
- Expose executor transaction result type
([#23759](paradigmxyz/reth#23759))

**Perf**
- Configurable rocksdb block cache size and re-use of mdbx provider in
re-execute ([#23701](paradigmxyz/reth#23701))
- Enable revm `p256-aws-lc-rs` feature
([#23721](paradigmxyz/reth#23721))
- Add platform-specific RUSTFLAGS to Dockerfile
([#23738](paradigmxyz/reth#23738))

**Bench**
- Add reorg mode to `new-payload-fcu`
([#23666](paradigmxyz/reth#23666))
- Enable `keccak-cache-global` in `reth-bb`
([#23723](paradigmxyz/reth#23723)) and fix
multi-executor support
([#23763](paradigmxyz/reth#23763))
- Dedupe merged BAL storage reads
([#23758](paradigmxyz/reth#23758))

**Re-execute**
- Verify reverts against changesets
([#23717](paradigmxyz/reth#23717))

**Payload**
- Track Amsterdam block gas in builders
([#23743](paradigmxyz/reth#23743))

**CLI**
- Use `TxTy`/`ReceiptTy` for static-file db get
([#23692](paradigmxyz/reth#23692)) and node
types in execution stage dump
([#23705](paradigmxyz/reth#23705))
- Preserve `trusted_nodes_only` from config when `--trusted-only` is
unset ([#23703](paradigmxyz/reth#23703))
- Avoid u64 underflow in `setup_without_evm` for genesis header
([#23728](paradigmxyz/reth#23728))

**Testing**
- Add BAL request e2e coverage
([#23727](paradigmxyz/reth#23727))
- Cover admin node info discv5 port
([#23781](paradigmxyz/reth#23781))

**Misc**
- Bump reth core to v0.3.1
([#23707](paradigmxyz/reth#23707)) and alloy-evm
to 0.33.3 ([#23778](paradigmxyz/reth#23778))
- Align ERA1 export with spec
([#23693](paradigmxyz/reth#23693))

## Migrations

🔗 Amp thread:
https://ampcode.com/threads/T-019dd89c-63d3-752f-ac22-7d4aef2ac3e3
- **Bumped reth git rev** from `7839f3d` to `73ec2c9` across all
`reth-*` workspace dependencies in
[Cargo.toml](file:///home/runner/work/tempo/tempo/Cargo.toml), and
bumped `reth-primitives-traits` from `0.3.0` to `0.3.1` and `alloy-evm`
from `0.33.0` to `0.34.0` to track upstream releases.
- **Switched p2p proxy transactions channel** from
`tokio::sync::mpsc::unbounded_channel` to
`reth_metrics::common::mpsc::memory_bounded_channel` with a 32 MiB
budget in
[bin/tempo/src/p2p_proxy.rs](file:///home/runner/work/tempo/tempo/bin/tempo/src/p2p_proxy.rs),
matching reth's new default `tx_channel_memory_limit_bytes` to prevent
unbounded memory growth; added `reth-metrics` to
[bin/tempo/Cargo.toml](file:///home/runner/work/tempo/tempo/bin/tempo/Cargo.toml).
- **Changed `BlockExecutor::commit_transaction` signature** in
[crates/evm/src/block.rs](file:///home/runner/work/tempo/tempo/crates/evm/src/block.rs)
from returning `Result<GasOutput, BlockExecutionError>` to returning
`GasOutput` directly (upstream removed the fallible return); replaced
the missing-tx error with an `expect`, and updated all tests to drop
`.unwrap()`.
- **Added `Send + 'static` bound to `TxResult` impl** for
`TempoTxResult<H>` to satisfy the new upstream trait bounds on
associated execution-result types.
- **Exposed `TempoBlockExecutor` and `TempoTxResult` as `pub`**
(previously `pub(crate)`) and re-exported them from
[crates/evm/src/lib.rs](file:///home/runner/work/tempo/tempo/crates/evm/src/lib.rs)
so they can be named in the new associated types.
- **Added associated types `TxExecutionResult` and `Executor` to
`BlockExecutorFactory` impl** for `TempoEvmConfig`, and changed
`create_executor` to return the concrete `Self::Executor<'a, DB, I>`
instead of `impl BlockExecutorFor<'a, Self, DB, I>`, tracking upstream's
move from RPIT to named associated types; dropped the now-unused
`BlockExecutorFor` import and the `'a` lifetime bounds on `DB`/`I`.
- **Implemented new `HeaderMut` methods** (`set_mix_hash`,
`set_extra_data`, `set_parent_beacon_block_root`) for `TempoHeader` in
[crates/primitives/src/reth_compat/header.rs](file:///home/runner/work/tempo/tempo/crates/primitives/src/reth_compat/header.rs)
to satisfy the expanded upstream trait, delegating to the inner Ethereum
header.

[GitHub
Workflow](https://github.com/tempoxyz/tempo/actions/runs/25101489247)

---------

Co-authored-by: Centaur AI <ai@centaur.local>
Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: klkvr <klkvrr@gmail.com>
@Vui-Chee Vui-Chee mentioned this pull request May 6, 2026
Merged
@shekhirin shekhirin mentioned this pull request May 6, 2026
Closed
1 task
This was referenced May 27, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joshieDo joshieDo joshieDo approved these changes

@mattsse mattsse Awaiting requested review from mattsse mattsse is a code owner

@Rjected Rjected Awaiting requested review from Rjected Rjected is a code owner

Assignees

No one assigned

Labels

A-db Related to the database C-bug An unexpected or incorrect behavior

Projects

Reth Tracker
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shekhirin @joshieDo