perf: reduce cacheline ping pong in workers availability#23287

Closed

pepyakin wants to merge 2 commits into

mainfrom

pep/worker-availability-sheet

Member

pepyakin commented Mar 30, 2026 •

edited

Loading

Every time a core updates the availability counter it claims the cacheline invalidating it on other cores. Workers essentially ping pong this cacheline to each others. This showed up during perf c2c profiling.

The solution is to have each worker report its status on its own cacheline. When a non-worker core wants to know the status/counts, it will request the cachelines making them shared temporarily. This path should become a bit more expensive but AFAIU it's also a less frequent one.

At least that's the theory.

github-project-automation Bot added this to Reth Tracker

github-project-automation Bot moved this to Backlog in Reth Tracker

Contributor

github-actions Bot commented Mar 30, 2026 •

edited

Loading

⚠️ Changelog not found.

A changelog entry is required before merging. We've generated a suggested changelog based on your changes:

Preview

---
reth-trie-parallel: patch
reth-engine-tree: patch
---

Replaced atomic counter-based worker availability tracking with per-worker cacheline-aligned `AtomicBool` flags to reduce false sharing. Changed the public API from returning counts (`available_*_workers`) to returning a boolean (`has_multiple_idle_*_workers`).

Add changelog to commit this to your branch.

pepyakin changed the title ~~worker availability sheet~~ perf: reduce cacheline ping pong in workers availability

Member Author

pepyakin commented Mar 30, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 30, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	403.05ms	399.52ms	-0.88% ✅ (±0.67%)
StdDev	208.31ms	207.46ms
P50	439.19ms	441.51ms	+0.53% ⚪ (±3.58%)
P90	644.66ms	641.88ms	-0.43% ⚪ (±3.26%)
P99	933.44ms	940.00ms	+0.70% ⚪ (±0.77%)
Mgas/s	2168.20	2196.30	+1.30% ✅ (±0.71%)
Wall Clock	47.99s	47.57s	-0.87% ✅ (±0.66%)

59 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	487.63ms	491.60ms
P50	0.00ms	0.00ms
P95	2050.42ms	2077.89ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 30, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 30, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	420.21ms	414.31ms	-1.40% ✅ (±0.38%)
StdDev	220.18ms	217.48ms
P50	469.35ms	461.98ms	-1.57% ⚪ (±3.00%)
P90	679.52ms	672.07ms	-1.10% ⚪ (±1.56%)
P99	1026.97ms	1022.44ms	-0.44% ⚪ (±1.03%)
Mgas/s	2088.46	2112.95	+1.17% ✅ (±0.50%)
Wall Clock	50.04s	49.32s	-1.44% ✅ (±0.38%)

59 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	497.71ms	502.69ms
P50	0.00ms	0.00ms
P95	2106.12ms	2113.12ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

pepyakin marked this pull request as ready for review

March 30, 2026 17:46

pepyakin requested review from Rjected, gakonst, mattsse, mediocregopher, shekhirin and yongkangc as code owners

March 30, 2026 17:46

DaniPopes reviewed

View reviewed changes

crates/engine/tree/src/tree/payload_processor/sparse_trie.rs

    
                          self.proof_worker_handle.available_account_workers(),

                          self.proof_worker_handle.available_storage_workers(),

                          self.proof_worker_handle.has_multiple_idle_account_workers(),

                          self.proof_worker_handle.has_multiple_idle_storage_workers(),

Contributor

DaniPopes Mar 30, 2026

these are both called eagerly but may be discarded inside of the function, can also be optimized to compute lazily if you think that could help


          worker availability sheet

f81632a

pepyakin force-pushed the pep/worker-availability-sheet branch from a363ddc to 8157dbb Compare

March 31, 2026 05:03

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	397.10ms	398.61ms	+0.38% ❌ (±0.36%)
StdDev	221.97ms	221.83ms
P50	406.82ms	410.24ms	+0.84% ⚪ (±3.55%)
P90	662.53ms	666.47ms	+0.59% ⚪ (±2.09%)
P99	1029.47ms	1027.73ms	-0.17% ⚪ (±1.54%)
Mgas/s	2162.53	2149.27	-0.61% ❌ (±0.55%)
Wall Clock	51.30s	51.48s	+0.35% ⚪ (±0.35%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	495.61ms	494.49ms
P50	0.00ms	0.00ms
P95	2112.97ms	2100.83ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	395.42ms	397.98ms	+0.65% ❌ (±0.40%)
StdDev	220.33ms	221.70ms
P50	399.11ms	406.71ms	+1.90% ⚪ (±3.49%)
P90	658.45ms	662.59ms	+0.63% ⚪ (±2.43%)
P99	1016.80ms	1032.00ms	+1.49% ❌ (±1.05%)
Mgas/s	2167.64	2158.69	-0.41% ⚪ (±0.50%)
Wall Clock	51.08s	51.41s	+0.64% ❌ (±0.40%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	496.10ms	493.53ms
P50	0.00ms	0.00ms
P95	2098.33ms	2061.25ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=f0d07c38be40c173abab5879b49a20dc4126c427 feature=f81632a

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`f0d07c38be40c173abab5879b49a20dc4126c427`	`f81632a`	Change
Mean	379.59ms	384.04ms	+1.17% ❌ (±0.42%)
StdDev	209.48ms	212.71ms
P50	390.03ms	391.41ms	+0.35% ⚪ (±3.06%)
P90	634.67ms	644.92ms	+1.61% ⚪ (±3.44%)
P99	945.41ms	947.16ms	+0.19% ⚪ (±0.64%)
Mgas/s	2255.67	2237.17	-0.82% ❌ (±0.61%)
Wall Clock	49.03s	49.62s	+1.19% ❌ (±0.42%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`f0d07c38be40c173abab5879b49a20dc4126c427`	`f81632a`
Mean	435.42ms	436.42ms
P50	0.00ms	0.00ms
P95	1842.40ms	1829.32ms

Trie Cache Update Wait

Metric	`f0d07c38be40c173abab5879b49a20dc4126c427`	`f81632a`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`f0d07c38be40c173abab5879b49a20dc4126c427`	`f81632a`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

↑ to make it clear. I rebased and added a new commit. The 1st and 2nd recent runs are with the Dani's suggestion included, the 3rd is only the original change. As you can see it's showing an opposite result. Now, let's see what would happen if we re-run the original, pre-rebase, change.

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=930f2a6eb257854138bb9e291c7013586fcd0223 feature=a363ddc3a3f3ffcc08d5609f933918778f846f50

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`	Change
Mean	400.94ms	396.50ms	-1.11% ✅ (±0.40%)
StdDev	223.11ms	220.46ms
P50	412.82ms	405.75ms	-1.71% ⚪ (±3.42%)
P90	672.66ms	657.95ms	-2.19% ✅ (±1.36%)
P99	1025.32ms	1023.88ms	-0.14% ⚪ (±0.79%)
Mgas/s	2137.66	2159.46	+1.02% ✅ (±0.55%)
Wall Clock	51.78s	51.20s	-1.11% ✅ (±0.39%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	491.04ms	500.60ms
P50	0.00ms	0.00ms
P95	2095.01ms	2116.43ms

Trie Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=930f2a6eb257854138bb9e291c7013586fcd0223 feature=a363ddc3a3f3ffcc08d5609f933918778f846f50

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`	Change
Mean	388.52ms	386.63ms	-0.49% ⚪ (±0.83%)
StdDev	215.32ms	214.00ms
P50	396.38ms	392.80ms	-0.90% ⚪ (±4.13%)
P90	653.31ms	646.63ms	-1.02% ⚪ (±4.81%)
P99	937.88ms	956.99ms	+2.04% ⚪ (±5.21%)
Mgas/s	2214.40	2214.34	-0.00% ⚪ (±0.76%)
Wall Clock	50.19s	49.94s	-0.50% ⚪ (±0.82%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	481.44ms	482.16ms
P50	0.00ms	0.00ms
P95	2021.21ms	2024.90ms

Trie Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`a363ddc3a3f3ffcc08d5609f933918778f846f50`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

↑ I think this settles that derek reports are just noise. I can see improving the c2c metrics and can see that that place is no longer a hotspot but I don't think it's worth it

pepyakin closed this

github-project-automation Bot moved this from Backlog to Done in Reth Tracker

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=930f2a6eb257854138bb9e291c7013586fcd0223 feature=a363ddc3a3f3ffcc08d5609f933918778f846f50

Member

decofe commented Mar 31, 2026 •

edited

Loading

❌ Benchmark failed while unknown step. View logs

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=930f2a6eb257854138bb9e291c7013586fcd0223 feature=930f2a6eb257854138bb9e291c7013586fcd0223

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`930f2a6eb257854138bb9e291c7013586fcd0223`	Change
Mean	390.20ms	389.54ms	-0.17% ⚪ (±0.39%)
StdDev	213.27ms	213.93ms
P50	402.53ms	407.75ms	+1.30% ⚪ (±3.36%)
P90	648.86ms	651.82ms	+0.46% ⚪ (±2.86%)
P99	947.87ms	947.63ms	-0.03% ⚪ (±0.45%)
Mgas/s	2189.42	2193.08	+0.17% ⚪ (±0.63%)
Wall Clock	50.40s	50.32s	-0.16% ⚪ (±0.38%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`930f2a6eb257854138bb9e291c7013586fcd0223`
Mean	480.56ms	476.79ms
P50	0.00ms	0.00ms
P95	2033.20ms	2022.01ms

Trie Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`930f2a6eb257854138bb9e291c7013586fcd0223`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`930f2a6eb257854138bb9e291c7013586fcd0223`	`930f2a6eb257854138bb9e291c7013586fcd0223`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big baseline=930f2a6eb257854138bb9e291c7013586fcd0223 feature=a363ddc3a3f3ffcc08d5609f933918778f846f50

pepyakin reopened this

github-project-automation Bot moved this from Done to In Progress in Reth Tracker

pepyakin force-pushed the pep/worker-availability-sheet branch from 8157dbb to 1080ba2 Compare

March 31, 2026 10:58

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	385.10ms	386.60ms	+0.39% ⚪ (±0.40%)
StdDev	211.33ms	212.32ms
P50	397.22ms	394.56ms	-0.67% ⚪ (±3.88%)
P90	643.76ms	641.54ms	-0.34% ⚪ (±2.48%)
P99	941.11ms	959.10ms	+1.91% ❌ (±1.66%)
Mgas/s	2221.22	2211.07	-0.46% ⚪ (±0.58%)
Wall Clock	49.74s	49.93s	+0.39% ⚪ (±0.39%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	443.59ms	446.51ms
P50	0.00ms	0.00ms
P95	1851.74ms	1861.94ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

pepyakin force-pushed the pep/worker-availability-sheet branch from 1080ba2 to f81632a Compare

March 31, 2026 11:09

Member Author

pepyakin commented Mar 31, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 31, 2026 •

edited

Loading

✅ Benchmark complete! View job

Benchmark Results

Metric	`main`	`pep/worker-availability-sheet`	Change
Mean	406.37ms	410.44ms	+1.00% ❌ (±0.36%)
StdDev	224.97ms	226.69ms
P50	419.62ms	420.57ms	+0.22% ⚪ (±3.12%)
P90	678.62ms	685.16ms	+0.96% ⚪ (±2.21%)
P99	1031.67ms	1040.88ms	+0.89% ❌ (±0.80%)
Mgas/s	2109.93	2082.18	-1.31% ❌ (±0.49%)
Wall Clock	52.49s	53.02s	+1.01% ❌ (±0.36%)

64 big blocks

Wait Time Breakdown

Persistence Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	491.71ms	495.07ms
P50	0.00ms	0.00ms
P95	2096.50ms	2107.76ms

Trie Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Execution Cache Update Wait

Metric	`main`	`pep/worker-availability-sheet`
Mean	0.00ms	0.00ms
P50	0.00ms	0.00ms
P95	0.00ms	0.00ms

Grafana Dashboard

Charts

Latency, Throughput & Diff

Wait Time Breakdown

Gas vs Latency

Grafana Dashboard

View real-time metrics

mediocregopher approved these changes

View reviewed changes


          Merge remote-tracking branch 'origin/main' into pep/worker-availabili…

3d6f775

…ty-sheet

Member

mediocregopher commented Mar 31, 2026

derek bench otlp=false blocks=big

Member

decofe commented Mar 31, 2026

❌ Invalid bench command

Invalid value(s): blocks=big (must be a positive integer)

Usage: @decofe bench [blocks=N] [big-blocks] [warmup=N] [baseline=REF] [feature=REF] [samply] [no-slack] [cores=N] [reth_newPayload=true|false] [abba=true|false] [otlp=true|false] [wait-time=DURATION] [baseline-args="..."] [feature-args="..."]

Member

mediocregopher commented Mar 31, 2026

derek bench big-blocks

Member

decofe commented Mar 31, 2026 •

edited

Loading

cc @mediocregopher

🚀 Benchmark started! View job

⏳ Status: Running benchmarks...

Config: blocks: big, baseline: main, feature: pep/worker-availability-sheet

pepyakin closed this

github-project-automation Bot moved this from In Progress to Done in Reth Tracker

pepyakin mentioned this pull request

perf: reduce cacheline ping pong in workers availability #23321

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

DaniPopes DaniPopes left review comments

mediocregopher mediocregopher approved these changes

Rjected Awaiting requested review from Rjected Rjected is a code owner

shekhirin Awaiting requested review from shekhirin shekhirin is a code owner

yongkangc Awaiting requested review from yongkangc

mattsse Awaiting requested review from mattsse mattsse is a code owner

gakonst Awaiting requested review from gakonst gakonst is a code owner

Labels

None yet