Skip to content

perf(engine): include backpressure in newPayload latency metric#23541

Merged
mediocregopher merged 2 commits into
mainfrom
mediocregopher/bench-latency-includes-backpressure
Apr 15, 2026
Merged

perf(engine): include backpressure in newPayload latency metric#23541
mediocregopher merged 2 commits into
mainfrom
mediocregopher/bench-latency-includes-backpressure

Conversation

@mediocregopher
Copy link
Copy Markdown
Member

The reth_newPayload latency timing previously only measured on_new_payload execution, starting the clock after backpressure and persistence waits had already completed. This hid real-world engine responsiveness issues from benchmark results — a change that caused 6x persistence backpressure could still show as a 7% improvement in all headline metrics.

Now latency = enqueued_at.elapsed() - explicit_persistence_wait, so backpressure (time the message spent queued because the engine was busy) is included in all headline metrics (Mean, P50, P90, P99, Wall Clock) while explicit persistence wait (the deliberate wait for previous block to persist) remains excluded.

Also adds Persist Wait as a row in the main benchmark comparison table with significance testing, so regressions in persistence throughput are visible at a glance instead of buried in a collapsed section.

Prompted by: mediocregopher

The reth_newPayload latency timing previously only measured on_new_payload
execution, starting the clock after backpressure and persistence waits
had already completed. This hid real-world engine responsiveness issues
from benchmark results.

Now latency = enqueued_at.elapsed() - explicit_persistence_wait, so
backpressure is included in all headline metrics (Mean, P50, P90, P99,
Wall Clock) while explicit persistence wait remains excluded.

Also adds persistence wait as a row in the main benchmark comparison
table with significance testing, so regressions in persistence
backpressure are visible at a glance.

Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9155-a132-7554-8e20-9766d1f0752d
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Copy link
Copy Markdown
Member Author

derek bench blocks=10 warmup=0 abba=false

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 15, 2026

cc @mediocregopher

❌ Benchmark failed while running baseline benchmark (1/2). View logs

@mediocregopher
Copy link
Copy Markdown
Member Author

derek bench blocks=10 warmup=1 abba=false

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 15, 2026

cc @mediocregopher

✅ Benchmark complete! View job

Benchmark Results

Metric main mediocregopher/bench-latency-includes-backpressure Change
Mean 78.13ms 77.51ms -0.79% ⚪ (±1.58%)
StdDev 43.90ms 43.49ms
P50 62.70ms 58.78ms -6.24% ✅ (±4.83%)
P90 179.62ms 177.67ms -1.09% ⚪ (±1.16%)
P99 179.62ms 177.67ms -1.09% ⚪ (±1.16%)
Mgas/s 526.82 532.36 +1.05% ⚪ (±2.58%)
Wall Clock 0.80s 0.79s -0.77% ⚪ (±1.51%)
Persist Wait 0.03ms 0.03ms -10.63% ⚪ (±15.07%)

10 blocks, 1 warmup

Wait Time Breakdown

Persistence Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.03ms 0.03ms
P50 0.03ms 0.03ms
P95 0.06ms 0.06ms

Trie Cache Update Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.00ms 0.20ms
P50 0.00ms 0.00ms
P95 0.00ms 1.58ms

Execution Cache Update Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.00ms 0.00ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Latency, Throughput & Diff

Wait Time Breakdown

Wait Time Breakdown

Gas vs Latency

Gas vs Latency

Grafana Dashboard

View real-time metrics

Adds the Persist Wait metric to the shared metricRows() function in
bench-utils.js, which feeds both the Slack notification table and the
GitHub Actions job summary.

Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9155-a132-7554-8e20-9766d1f0752d
Co-authored-by: Amp <amp@ampcode.com>
@mediocregopher
Copy link
Copy Markdown
Member Author

derek bench blocks=10 warmup=1 abba=false

@decofe
Copy link
Copy Markdown
Member

decofe commented Apr 15, 2026

cc @mediocregopher

✅ Benchmark complete! View job

Benchmark Results

Metric main mediocregopher/bench-latency-includes-backpressure Change
Mean 81.50ms 77.57ms -4.82% ⚪ (±5.32%)
StdDev 45.15ms 44.20ms
P50 75.18ms 63.52ms -15.51% ✅ (±14.25%)
P90 186.14ms 178.28ms -4.22% ✅ (±2.57%)
P99 186.14ms 178.28ms -4.22% ✅ (±2.57%)
Mgas/s 511.30 537.54 +5.13% ⚪ (±7.82%)
Wall Clock 0.83s 0.79s -4.75% ⚪ (±5.36%)
Persist Wait 0.03ms 0.03ms -4.83% ⚪ (±13.04%)

10 blocks, 1 warmup

Wait Time Breakdown

Persistence Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.03ms 0.03ms
P50 0.03ms 0.03ms
P95 0.06ms 0.08ms

Trie Cache Update Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.13ms 0.06ms
P50 0.00ms 0.00ms
P95 1.29ms 0.61ms

Execution Cache Update Wait

Metric main mediocregopher/bench-latency-includes-backpressure
Mean 0.00ms 0.00ms
P50 0.00ms 0.00ms
P95 0.00ms 0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Latency, Throughput & Diff

Wait Time Breakdown

Wait Time Breakdown

Gas vs Latency

Gas vs Latency

Grafana Dashboard

View real-time metrics

@github-project-automation github-project-automation Bot moved this from Backlog to In Progress in Reth Tracker Apr 15, 2026
@mediocregopher mediocregopher added this pull request to the merge queue Apr 15, 2026
Merged via the queue into main with commit cf29b3f Apr 15, 2026
36 checks passed
@mediocregopher mediocregopher deleted the mediocregopher/bench-latency-includes-backpressure branch April 15, 2026 17:26
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Reth Tracker Apr 15, 2026
mediocregopher added a commit that referenced this pull request Apr 20, 2026
This reverts commit 93b2201 and commit cf29b3f.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daa9e-42a5-74da-a301-2dcd4474bf17
Co-authored-by: Amp <amp@ampcode.com>
github-merge-queue Bot pushed a commit that referenced this pull request Apr 20, 2026
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Co-authored-by: Amp <amp@ampcode.com>
crazywriter1 pushed a commit to crazywriter1/tempo that referenced this pull request Apr 23, 2026
Automated nightly update of reth dependencies from `paradigmxyz/reth`
main branch.

## Upstream reth changes


[`98ebc34...7839f3d`](paradigmxyz/reth@98ebc34...7839f3d)

🔗 Amp thread:
https://ampcode.com/threads/T-019db884-a7c3-738b-8f38-cc04f8942d8a
**Engine**
- Suppress persistence during payload building
([#23618](paradigmxyz/reth#23618))
- Align Amsterdam endpoint validation
([#23625](paradigmxyz/reth#23625))
- Revert [#23541](paradigmxyz/reth#23541) and
[#23578](paradigmxyz/reth#23578)
([#23646](paradigmxyz/reth#23646))
- Let consensus impls control which errors are transient
([#23668](paradigmxyz/reth#23668))
- Configure invalid header cache hit eviction
([#23670](paradigmxyz/reth#23670))

**Perf**
- Relax executor reset thresholds for re-execute
([#23617](paradigmxyz/reth#23617))
- Replace `BTreeMap` with `imbl::OrdMap` in `BestTransactions`
([#23621](paradigmxyz/reth#23621))
- Avoid reopening `.csoff` on every changeset lookup
([#23687](paradigmxyz/reth#23687))
- Disable read tx timeout during re-execute
([#23680](paradigmxyz/reth#23680))

**P2P / Net**
- Add snap/2 wire helpers and messages
([#23611](paradigmxyz/reth#23611))
- Optionally fetch BAL with full blocks
([#23629](paradigmxyz/reth#23629))
- Discv5 enabled by default
([#23686](paradigmxyz/reth#23686))

**DB**
- Add `reth db migrate-v2` for v1→v2 storage migration
([#23422](paradigmxyz/reth#23422))
- Detect and warn about ZFS
([#23685](paradigmxyz/reth#23685))

**BAL**
- Scaffold BAL store abstraction
([#23596](paradigmxyz/reth#23596))
- Enable BAL building in ethereum payload
([#23597](paradigmxyz/reth#23597))
- Add parallelization and batch IO flags
([#23663](paradigmxyz/reth#23663))

**Refactor**
- Make `WorkerPool` lazy by default
([#23627](paradigmxyz/reth#23627))
- Encapsulate state fetching in db provider
([#23656](paradigmxyz/reth#23656))
- Remove `TrieNodeProvider`
([#23658](paradigmxyz/reth#23658))
- Unify opaque consensus error helpers
([#23669](paradigmxyz/reth#23669))

**Payload**
- Add gas limit and slot number to `BlockOrPayload`
([#23624](paradigmxyz/reth#23624),
[#23626](paradigmxyz/reth#23626))

**Bench**
- Add CLI flag to fetch balances by default; require local benchmark
data ([#23655](paradigmxyz/reth#23655),
[#23679](paradigmxyz/reth#23679))

**Deps**
- Bump alloy crates to 2.0.1
([#23677](paradigmxyz/reth#23677)),
rustls-webpki
([#23681](paradigmxyz/reth#23681)), weekly
`cargo update`
([#23628](paradigmxyz/reth#23628))

**Testing**
- Remove unsafe `env::set_var(RUST_LOG)` from tests
([#23672](paradigmxyz/reth#23672))
- Address nightly clippy warnings
([#23630](paradigmxyz/reth#23630))

## Migrations

🔗 Amp thread:
https://ampcode.com/threads/T-019db884-dc46-71f5-a823-00c3a16191d4
- **Reth dependency bump**: All `reth-*` git dependencies updated from
rev `98ebc34` to `7839f3d`
- **Alloy version bump**: `alloy-*` crates updated from `2.0.0` to
`2.0.1`; `alloy-evm` changed from `0.33.2` to `0.33.0`
- **`ConsensusError::Other` → `ConsensusError::msg`**: All
`ConsensusError::Other(...)` calls migrated to
`ConsensusError::msg(...)`, which accepts `&str`/`impl Display` directly
instead of requiring `String` (removes `.to_string()` calls for string
literals)
- **`deny.toml` license exceptions**: Added MPL-2.0 exceptions for
`bitmaps`, `imbl`, and `imbl-sized-chunks` (new transitive dependencies)

[GitHub
Workflow](https://github.com/tempoxyz/tempo/actions/runs/24816009191)

---------

Co-authored-by: Alexey Shekhirin <github@shekhirin.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants