Skip to content

fix(bench): require local benchmark data#23679

Merged
mediocregopher merged 7 commits into
mainfrom
mediocregopher/local-bench-data
Apr 22, 2026
Merged

fix(bench): require local benchmark data#23679
mediocregopher merged 7 commits into
mainfrom
mediocregopher/local-bench-data

Conversation

@mediocregopher
Copy link
Copy Markdown
Member

@mediocregopher mediocregopher commented Apr 22, 2026

Bench workflows now validate that the local schelk snapshot is already populated instead of checking MinIO and downloading it on demand. Big-block benchmarks likewise require the local payload archive and use its manifest to determine which weekly snapshot must already be present, and the benchmark build helper now builds locally instead of querying the MinIO binary cache.

Co-Authored-By: Brian Picciano 933154+mediocregopher@users.noreply.github.com

Prompted by: mediocregopher

@mediocregopher
fix(bench): require local benchmark data
9fce582 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db48c-8b95-744f-aad9-140ad31275fc
Co-authored-by: Amp <amp@ampcode.com>
@github-project-automation github-project-automation Bot moved this to Backlog in Reth Tracker Apr 22, 2026
@mediocregopher mediocregopher self-assigned this Apr 22, 2026
@mediocregopher mediocregopher added A-ci Related to github workflows or other build and lint tools C-benchmark A change that impacts how or what we benchmark labels Apr 22, 2026
@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

❌ Benchmark failed while validating local snapshot. View logs

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

❌ Benchmark failed while validating local snapshot. View logs

@mediocregopher
fix(bench): stop requiring jwt in local snapshots
b31d1bc 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db48c-8b95-744f-aad9-140ad31275fc
Co-authored-by: Amp <amp@ampcode.com>
@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

❌ Benchmark failed while running baseline benchmark (1/2). View logs

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

⚠️ Benchmark cancelled. View logs

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

✅ Benchmark complete! View job

Benchmark Results

⚠️ Feature is 1 commit behind mediocregopher/local-bench-data. Consider rebasing for accurate results.

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data Change
Mean 41.09ms 41.99ms +2.17% ⚪ (±9.11%)
StdDev 47.90ms 41.99ms
P50 28.91ms 31.31ms +8.29% ⚪ (±9.66%)
P90 68.24ms 63.88ms -6.39% ⚪ (±44.46%)
P99 355.94ms 329.29ms -7.49% ⚪ (±20.65%)
Mgas/s 1187.46 1135.75 -4.36% ❌ (±3.25%)
Wall Clock 4.21s 4.31s +2.44% ⚪ (±9.03%)
Persist Wait 17.03ms 16.76ms -1.56% ⚪ (±96.88%)

100 blocks, 10 warmup

Wait Time Breakdown

Persistence Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 17.03ms 16.76ms
P50 0.02ms 0.02ms
P95 201.88ms 212.32ms

Trie Cache Update Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 0.43ms 0.39ms
P50 0.00ms 0.00ms
P95 2.66ms 1.86ms

Execution Cache Update Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 0.02ms 0.06ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Latency, Throughput & Diff

Wait Time Breakdown

Wait Time Breakdown

Gas vs Latency

Gas vs Latency

Grafana Dashboard

View real-time metrics

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

❌ Benchmark failed while validating local snapshot. View logs

@mediocregopher mediocregopher marked this pull request as ready for review April 22, 2026 10:12
@mediocregopher mediocregopher requested a review from gakonst as a code owner April 22, 2026 10:12
@mediocregopher
fix(ci): drop stale cargo-deny ignores
62d12ac 
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db48c-8b95-744f-aad9-140ad31275fc
Co-authored-by: Amp <amp@ampcode.com>
@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

❌ Benchmark failed while validating local snapshot. View logs

mediocregopher and others added 2 commits April 22, 2026 11:05
@mediocregopher
fix(ci): ignore rustls-webpki CRL advisory
c8a3f8f 
RUSTSEC-2026-0104 only affects CRL parsing before signature
verification. reth does not parse CRLs, so ignore it to unblock
cargo deny on this benchmark workflow PR.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db4da-109e-74b8-938e-c5fb27b8f275
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
fix(bench): tolerate stale schelk mount state
1d21fd5 
Big-bench jobs can hit a stale schelk state after host reboots where
incremental recover fails and mount reports the volume is already
mounted. Fall back to full recovery and validate the mounted datadir
instead of failing on that mount exit code alone.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db4da-109e-74b8-938e-c5fb27b8f275
Co-authored-by: Amp <amp@ampcode.com>
@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 22, 2026 •
edited
Loading

cc @mediocregopher

✅ Benchmark complete! View job

Benchmark Results

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data Change
Mean 748.50ms 810.17ms +8.24% ❌ (±5.82%)
StdDev 178.34ms 139.98ms
P50 701.04ms 809.79ms +15.51% ❌ (±4.87%)
P90 1007.50ms 986.79ms -2.06% ⚪ (±11.99%)
P99 1409.84ms 1226.74ms -12.99% ⚪ (±13.81%)
Mgas/s 1425.07 1296.46 -9.02% ❌ (±4.87%)
Wall Clock 76.39s 82.34s +7.79% ❌ (±5.98%)
Persist Wait 0.06ms 0.35ms +511.01% ⚪ (±999.44%)

100 big blocks, 10 warmup, BAL: feature

Wait Time Breakdown

Persistence Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 0.06ms 0.35ms
P50 0.05ms 0.06ms
P95 0.08ms 0.09ms

Trie Cache Update Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 0.00ms 0.00ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Execution Cache Update Wait

Metric mediocregopher/local-bench-data mediocregopher/local-bench-data
Mean 0.00ms 0.00ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Latency, Throughput & Diff

Wait Time Breakdown

Wait Time Breakdown

Gas vs Latency

Gas vs Latency

Grafana Dashboard

View real-time metrics

mediocregopher and others added 2 commits April 22, 2026 11:18
@mediocregopher
merge origin/main into mediocregopher/local-bench-data
1676572 
Bring in origin/main, including the rustls-webpki lockfile bump,
and drop the temporary RUSTSEC-2026-0104 ignore that becomes stale
after the merge.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db4da-109e-74b8-938e-c5fb27b8f275
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
test(ci): restore deny.toml from origin/main
05e2545 
Restore the advisory ignore list to match origin/main exactly so we can
verify whether cargo deny still passes on this branch after the merge.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db4da-109e-74b8-938e-c5fb27b8f275
Co-authored-by: Amp <amp@ampcode.com>
@github-project-automation github-project-automation Bot moved this from Backlog to In Progress in Reth Tracker Apr 22, 2026
@mediocregopher mediocregopher added this pull request to the merge queue Apr 22, 2026
Merged via the queue into main with commit 122c5b3 Apr 22, 2026
35 checks passed
@mediocregopher mediocregopher deleted the mediocregopher/local-bench-data branch April 22, 2026 12:26
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Reth Tracker Apr 22, 2026
This was referenced Apr 22, 2026
Closed
Merged
crazywriter1 pushed a commit to crazywriter1/tempo that referenced this pull request Apr 23, 2026
@decofe @shekhirin
deps: update reth from main (2026-04-23) (tempoxyz#3698)
328c734 
Automated nightly update of reth dependencies from `paradigmxyz/reth`
main branch.

## Upstream reth changes


[`98ebc34...7839f3d`](paradigmxyz/reth@98ebc34...7839f3d)

🔗 Amp thread:
https://ampcode.com/threads/T-019db884-a7c3-738b-8f38-cc04f8942d8a
**Engine**
- Suppress persistence during payload building
([#23618](paradigmxyz/reth#23618))
- Align Amsterdam endpoint validation
([#23625](paradigmxyz/reth#23625))
- Revert [#23541](paradigmxyz/reth#23541) and
[#23578](paradigmxyz/reth#23578)
([#23646](paradigmxyz/reth#23646))
- Let consensus impls control which errors are transient
([#23668](paradigmxyz/reth#23668))
- Configure invalid header cache hit eviction
([#23670](paradigmxyz/reth#23670))

**Perf**
- Relax executor reset thresholds for re-execute
([#23617](paradigmxyz/reth#23617))
- Replace `BTreeMap` with `imbl::OrdMap` in `BestTransactions`
([#23621](paradigmxyz/reth#23621))
- Avoid reopening `.csoff` on every changeset lookup
([#23687](paradigmxyz/reth#23687))
- Disable read tx timeout during re-execute
([#23680](paradigmxyz/reth#23680))

**P2P / Net**
- Add snap/2 wire helpers and messages
([#23611](paradigmxyz/reth#23611))
- Optionally fetch BAL with full blocks
([#23629](paradigmxyz/reth#23629))
- Discv5 enabled by default
([#23686](paradigmxyz/reth#23686))

**DB**
- Add `reth db migrate-v2` for v1→v2 storage migration
([#23422](paradigmxyz/reth#23422))
- Detect and warn about ZFS
([#23685](paradigmxyz/reth#23685))

**BAL**
- Scaffold BAL store abstraction
([#23596](paradigmxyz/reth#23596))
- Enable BAL building in ethereum payload
([#23597](paradigmxyz/reth#23597))
- Add parallelization and batch IO flags
([#23663](paradigmxyz/reth#23663))

**Refactor**
- Make `WorkerPool` lazy by default
([#23627](paradigmxyz/reth#23627))
- Encapsulate state fetching in db provider
([#23656](paradigmxyz/reth#23656))
- Remove `TrieNodeProvider`
([#23658](paradigmxyz/reth#23658))
- Unify opaque consensus error helpers
([#23669](paradigmxyz/reth#23669))

**Payload**
- Add gas limit and slot number to `BlockOrPayload`
([#23624](paradigmxyz/reth#23624),
[#23626](paradigmxyz/reth#23626))

**Bench**
- Add CLI flag to fetch balances by default; require local benchmark
data ([#23655](paradigmxyz/reth#23655),
[#23679](paradigmxyz/reth#23679))

**Deps**
- Bump alloy crates to 2.0.1
([#23677](paradigmxyz/reth#23677)),
rustls-webpki
([#23681](paradigmxyz/reth#23681)), weekly
`cargo update`
([#23628](paradigmxyz/reth#23628))

**Testing**
- Remove unsafe `env::set_var(RUST_LOG)` from tests
([#23672](paradigmxyz/reth#23672))
- Address nightly clippy warnings
([#23630](paradigmxyz/reth#23630))

## Migrations

🔗 Amp thread:
https://ampcode.com/threads/T-019db884-dc46-71f5-a823-00c3a16191d4
- **Reth dependency bump**: All `reth-*` git dependencies updated from
rev `98ebc34` to `7839f3d`
- **Alloy version bump**: `alloy-*` crates updated from `2.0.0` to
`2.0.1`; `alloy-evm` changed from `0.33.2` to `0.33.0`
- **`ConsensusError::Other` → `ConsensusError::msg`**: All
`ConsensusError::Other(...)` calls migrated to
`ConsensusError::msg(...)`, which accepts `&str`/`impl Display` directly
instead of requiring `String` (removes `.to_string()` calls for string
literals)
- **`deny.toml` license exceptions**: Added MPL-2.0 exceptions for
`bitmaps`, `imbl`, and `imbl-sized-chunks` (new transitive dependencies)

[GitHub
Workflow](https://github.com/tempoxyz/tempo/actions/runs/24816009191)

---------

Co-authored-by: Alexey Shekhirin <github@shekhirin.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shekhirin shekhirin shekhirin approved these changes

@emmajam emmajam emmajam approved these changes

@gakonst gakonst Awaiting requested review from gakonst gakonst is a code owner

Assignees

@mediocregopher mediocregopher

Labels

A-ci Related to github workflows or other build and lint tools C-benchmark A change that impacts how or what we benchmark

Projects

Reth Tracker
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mediocregopher @decofe @shekhirin @emmajam