feat(engine): slow block logs by shekhirin · Pull Request #21433 · paradigmxyz/reth

shekhirin · 2026-01-26T16:26:54Z

Ref: https://hackmd.io/dg7rizTyTXuCf2LSa2LsyQ, https://ethresear.ch/t/data-driven-analysis-on-eip-7907/23850

Reference implementations in other clients: NethermindEth/nethermind#10288, ethereum/go-ethereum#33655

Adds configurable slow block logging via --engine.slow-block-threshold that emits a structured warn! log with detailed timing, state-operation counts, and cache hit-rate metrics for blocks whose total processing time exceeds the threshold.

When the threshold is configured, block validation instruments the state provider to collect fetch counts and latencies, and tracks cache hits/misses. After persistence commits, blocks exceeding the threshold emit a ConsensusEngineEvent::SlowBlock carrying an ExecutionTimingStats snapshot. The node events handler formats this into a single structured log line with execution time, state root time, commit time, mgas/s, account/storage/code fetch counts, and cache hit rates.

When no threshold is set, no stats are collected and no additional allocations occur.

Key types introduced:

ExecutionTimingStats (reth-chain-state): 24-field struct capturing block number/hash, execution/state-root/commit durations, state operation counts, and cache metrics
SlowBlockInfo (reth-engine-primitives): wrapper carrying stats + commit duration + total duration
CacheStats (engine-tree): atomic counters for cache hit/miss tracking
StateProviderStats (engine-tree): atomic counters for state provider fetch counts and latencies
ValidationOutcome (engine-tree): result type alias for block validation with optional timing stats

Also refactors PersistenceResult to carry commit duration alongside the last persisted block, replacing the separate SaveBlocksResult type.

mattsse

this is quite a lot of stuff -.-

but I guess all of these make sense, I'd like to avoid having this leak as part of the block type

also have some suggestions for how to avoid adding this to the persistence task as well.

mattsse · 2026-01-27T18:28:47Z

    }
 }
+
+/// A handle to [`InstrumentedStateProvider`] that can be used to fetch recorded metrics.


i find handle here a bit confusing, isnt this now just a container for metrics?

mattsse · 2026-01-27T18:30:15Z

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        if s.is_empty() {
-            return Ok(Self(Default::default()))
+            return Ok(Self(Default::default()));


mattsse · 2026-01-27T18:30:22Z

        let Self { builder, task_executor } = self;

        let engine_tree_config = builder.config.engine.tree_config();
-


mattsse · 2026-01-27T18:30:38Z

+reth-evm = { workspace = true, features = ["test-utils", "metrics"] }
+reth-revm.workspace = true
+reth-execution-types.workspace = true
 secp256k1.workspace = true
 alloy-genesis.workspace = true
+revm-bytecode.workspace = true
+revm-database.workspace = true
+revm-primitives.workspace = true
+tracing-subscriber = { workspace = true, features = ["env-filter"] }


why do we need all of this here now?

To enable testing. I needed all of the dev depencencies in order to be able to run the tests.

mattsse · 2026-01-27T18:31:59Z

+/// Trie context needed for [`BasicEngineValidator::spawn_deferred_trie_task`]
+#[derive(Debug)]
+struct TrieCtx<P> {
+    hashed_state: Arc<HashedPostState>,
+    trie_output: Arc<TrieUpdates>,
+    overlay_factory: OverlayStateProviderFactory<P>,
+}
+
+impl<P> TrieCtx<P> {
+    fn new(
+        hashed_state: HashedPostState,
+        trie_output: TrieUpdates,
+        overlay_factory: OverlayStateProviderFactory<P>,
+    ) -> Self {
+        Self {
+            hashed_state: Arc::new(hashed_state),
+            trie_output: Arc::new(trie_output),
+            overlay_factory,
+        }
+    }
+}


im losing it with amp, every single type it puts new types at the wrong place, this does not belong above the main type

This wasn't in my original PR. @shekhirin I assume is your addition?

mattsse · 2026-01-27T18:37:41Z

+        // Emit unified slow block logs after commit completes
+        for stats in timing_stats {
+            let total_duration = stats.execution_duration +
+                stats.state_read_duration +
+                stats.state_hash_duration +
+                commit_duration;
+
+            // Check if total time exceeds threshold
+            if let Some(threshold) = self.slow_block_threshold &&
+                total_duration > threshold
+            {
+                self.log_slow_block(stats, commit_duration, total_duration);
+            }
+        }


can we extract this to a helper fn perhaps?

mattsse · 2026-01-27T18:38:12Z

+        self.metrics.save_blocks_duration_seconds.record(total_commit_duration);
+
+        // Emit unified slow block logs after commit completes
+        for stats in timing_stats {


this loop is a noop if no slow_block_threshold is set right?

Implements execution metrics following the cross-client specification: https://github.com/ethereum/execution-specs/blob/main/docs/execution-metrics-spec.md - Add slow block logging method to ExecutorMetrics - Output structured JSON log entries for cross-client analysis - Include timing, throughput, and state access counts - Threshold configurable via SLOW_BLOCK_THRESHOLD_MS constant

Implement standardized JSON format for slow block logging to enable cross-client performance analysis and protocol research. This change is part of the Cross-Client Execution Metrics initiative proposed by Gary Rong: https://hackmd.io/dg7rizTyTXuCf2LSa2LsyQ The standardized metrics enabled data-driven analysis like the EIP-7907 research: https://ethresear.ch/t/data-driven-analysis-on-eip-7907/23850 JSON format includes: - block: number, hash, gas_used, tx_count - timing: execution_ms, total_ms - throughput: mgas_per_sec - state_reads: accounts, storage_slots, code, code_bytes - state_writes: accounts, storage_slots - cache: account/storage/code hits, misses, hit_rate

Convert timing parameters from u64 to f64 to preserve sub-millisecond precision in slow block JSON output. Changes: - log_slow_block() timing params: u64 -> f64 - is_slow_block(): u64 -> f64 with threshold cast - Tracing format: add {:.3} for 3 decimal places - Update tests for f64 values

…ring

Remove CacheStats from CachedStateMetrics and pass it as an optional Arc<CacheStats> to CachedStateProvider only when slow_block_threshold is configured. This avoids unnecessary atomic increments on every cache hit/miss when slow block logging is disabled. Replace InstrumentedStateProviderHandle/InstrumentedStateProviderHandleInner with a flat StateProviderStats struct shared via Arc. The provider now holds metrics and stats directly, removing a layer of indirection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…r for cache_stats Use with_cache_stats() builder method instead of adding a parameter to the constructor. Also fix pre-existing const fn breakage on nightly for ExecutionCache::new and SavedCache::new. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Eliminates PersistenceRx enum and SaveBlocksResult by using a single PersistenceResult type for all persistence operations, simplifying wait_for_event from duplicated per-variant select blocks to a single unified select_biased block. Fixes total_duration double-counting (state_read_duration is a subset of execution_duration). Removes premature purge_timing_stats call in on_new_persisted_block that was clearing stats before the threshold check could run. Fixes divide-by-zero in mgas/s calculation. Corrects misleading doc comments on Duration fields in ExecutionTimingStats. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ock logging Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…_complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…rics-standardization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When CacheStats is present (slow block threshold configured), record only to CacheStats during execution and flush to prometheus in a single batch at the end, avoiding double recording on every cache access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

This reverts commit 2bbc78b.

Resolved conflicts in config.rs and engine.rs where sparse_trie_max_storage_tries was replaced by sparse_trie_max_hot_slots + sparse_trie_max_hot_accounts on main. Kept both main's LFU pruning renames and the PR's slow_block_threshold additions. Amp-Thread-ID: https://ampcode.com/threads/T-019cc34f-c511-71b7-b6bd-ad8600e06a51 Co-authored-by: Amp <amp@ampcode.com>

github-project-automation Bot added this to Reth Tracker Jan 26, 2026

github-project-automation Bot moved this to Backlog in Reth Tracker Jan 26, 2026

shekhirin mentioned this pull request Jan 26, 2026

feat(evm): standardize slow block JSON output for cross-client metrics #21237

Closed

shekhirin force-pushed the feature/execution-metrics-standardization branch 9 times, most recently from 3edfd77 to 57b9e54 Compare January 27, 2026 15:50

shekhirin added C-enhancement New feature or request A-observability Related to tracing, metrics, logs and other observability tools labels Jan 27, 2026

shekhirin marked this pull request as ready for review January 27, 2026 18:01

shekhirin requested review from Rjected, fgimenez, gakonst, klkvr, mattsse, mediocregopher and yongkangc as code owners January 27, 2026 18:01

mattsse requested changes Jan 27, 2026

View reviewed changes

github-project-automation Bot moved this from Backlog to In Progress in Reth Tracker Jan 27, 2026

DaniPopes reviewed Feb 8, 2026

View reviewed changes

Comment thread crates/chain-state/src/execution_stats.rs Outdated

DaniPopes reviewed Feb 8, 2026

View reviewed changes

Comment thread crates/chain-state/src/execution_stats.rs

CPerezz added 4 commits February 9, 2026 19:55

fix: apply rustfmt and fix tracing feature propagation

98e2d3c

mattsse and others added 7 commits March 5, 2026 10:01

refactor: move SlowBlockInfo below ConsensusEngineEvent per type orde…

1db3175

…ring

refactor: make CachedStateMetrics.stats private, add getters

74f465d

refactor: replace forwarding helpers with single cache_stats() getter

fa23648

fix: restore account_cache_collisions reset

311de58

refactor: move CachedStatus above trait impls for CachedStateProvider

17749a9

refactor: remove unnecessary Box around ConsensusEngineEvent

8fdb5aa

mattsse force-pushed the feature/execution-metrics-standardization branch from a95a657 to 82dd69a Compare March 5, 2026 09:34

mattsse force-pushed the feature/execution-metrics-standardization branch from 067a938 to 41ab9d1 Compare March 5, 2026 10:26

mattsse and others added 6 commits March 5, 2026 11:45

refactor: restore original comments, rename slow block to detailed bl…

23a0ca0

…ock logging Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor(engine): accept PersistenceResult directly in on_persistence…

432e824

…_complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore: fix SlowBlockInfo total_duration doc, add changelog

dd25792

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feature/execution-met…

d583073

…rics-standardization

fix: add backticks for clippy::doc_markdown

a6fb304

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

shekhirin commented Mar 5, 2026

View reviewed changes

Comment thread crates/engine/tree/src/persistence.rs

mattsse and others added 4 commits March 5, 2026 13:20

fix: restore finalized/safe block clamping logic in persistence

fa96832

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor: move ValidationOutcome type alias to top of file

0937a32

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Revert "refactor: deduplicate cache hit/miss recording"

25c62b0

This reverts commit 2bbc78b.

shekhirin force-pushed the feature/execution-metrics-standardization branch from 25a187e to 25c62b0 Compare March 6, 2026 12:46

mattsse approved these changes Mar 6, 2026

View reviewed changes

mattsse enabled auto-merge March 6, 2026 13:38

mattsse added this pull request to the merge queue Mar 6, 2026

Merged via the queue into main with commit 99aea38 Mar 6, 2026
39 checks passed

mattsse deleted the feature/execution-metrics-standardization branch March 6, 2026 13:56

github-project-automation Bot moved this from In Progress to Done in Reth Tracker Mar 6, 2026

		let Self { builder, task_executor } = self;

		let engine_tree_config = builder.config.engine.tree_config();

Conversation

shekhirin commented Jan 26, 2026 • edited by mattsse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shekhirin commented Jan 26, 2026 •

edited by mattsse

Loading