Skip to content

[FRAME] Omni bencher run each benchmark at least 10 secs#10794

Merged
ggwpez merged 22 commits intomasterfrom
oty-bencher-stability
Feb 11, 2026
Merged

[FRAME] Omni bencher run each benchmark at least 10 secs#10794
ggwpez merged 22 commits intomasterfrom
oty-bencher-stability

Conversation

@ggwpez
Copy link
Copy Markdown
Member

@ggwpez ggwpez commented Jan 13, 2026

Changes:

  • Ensure all benchmarks run for at least 10 seconds. Configurable with --min-duration <s>
  • Turn off runtime logging in bench bot to reduce spam log output
  • Reduce DB repetition to 1 since PoV metering must be deterministic

Example of the System benchmark with the set_heap_pages benchmark that took less than 10 ms before:

2026-01-13T21:36:10.687286Z [ 22 % ] Starting benchmark: frame_system::set_heap_pages    
2026-01-13T21:36:10.688437Z [ 33 % ] Starting benchmark: frame_system::set_code    

Now takes 10 seconds:

2026-01-13T21:37:31.392981Z [ 22 % ] Starting benchmark: frame_system::set_heap_pages    
2026-01-13T21:37:32.271275Z [ 22 % ] Running  benchmark: frame_system::set_heap_pages (overtime)    
2026-01-13T21:37:37.272099Z [ 22 % ] Running  benchmark: frame_system::set_heap_pages (overtime)    
2026-01-13T21:37:41.393107Z [ 33 % ] Starting benchmark: frame_system::set_code    

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez marked this pull request as ready for review January 19, 2026 10:38
&mut Default::default(),
&executor,
"Benchmark_dispatch_benchmark",
&params(false, 1),
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reduced this to 1 since DB tracking should not need repetition.

@ggwpez
Copy link
Copy Markdown
Member Author

ggwpez commented Jan 19, 2026

/cmd bench --pallet pallet_staking_async --runtime asset-hub-westend

@github-actions
Copy link
Copy Markdown
Contributor

Command "bench --pallet pallet_staking_async --runtime asset-hub-westend" has started 🚀 See logs here

@paritytech-workflow-stopper
Copy link
Copy Markdown

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/21134312805
Failed job name: fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez requested review from a team as code owners January 19, 2026 12:58
@ggwpez
Copy link
Copy Markdown
Member Author

ggwpez commented Jan 19, 2026

/cmd bench --pallet pallet_staking_async --runtime asset-hub-westend

@github-actions
Copy link
Copy Markdown
Contributor

Command "bench --pallet pallet_staking_async --runtime asset-hub-westend" has started 🚀 See logs here

…t_staking_async --runtime asset-hub-westend'
@github-actions
Copy link
Copy Markdown
Contributor

Command "bench --pallet pallet_staking_async --runtime asset-hub-westend" has finished ✅ See logs here

Details

Subweight results:
File Extrinsic Old New Change [%]
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs chill 5.96ms 7.33ms +22.89
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs bond_extra 13.18ms 14.81ms +12.38
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_no_eras 22.65us 21.52us -5.01
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs set_staking_configs_all_set 8.89us 8.44us -5.05
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs restore_ledger 114.86us 108.99us -5.11
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs set_staking_configs_all_remove 8.12us 7.69us -5.30
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs withdraw_unbonded_update 181.11us 171.14us -5.50
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs bond 162.70us 153.71us -5.53
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs set_controller 77.23us 72.40us -6.26
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs update_payee 75.53us 70.38us -6.82
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs kick 2.36ms 2.16ms -8.26
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs set_payee 61.30us 56.16us -8.39
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs reap_stash 13.40ms 12.23ms -8.74
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs rc_on_offence 57.11ms 50.93ms -10.82
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_stakers_overview 399.79us 344.58us -13.81
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_validator_slash_in_era 417.46us 357.15us -14.45
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs process_offence_queue 12.92ms 10.99ms -14.98
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_apply_min_commission 67.31us 57.13us -15.12
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs rc_on_session_report 11.44ms 9.66ms -15.59
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs payout_stakers_alive_staked 37.56ms 31.53ms -16.04
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs deprecate_controller_batch 45.77ms 37.97ms -17.06
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs withdraw_unbonded_kill 14.83ms 12.28ms -17.16
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_claimed_rewards 419.64us 338.46us -19.35
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_stakers_paged 385.92us 309.37us -19.84
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs nominate 13.24ms 10.57ms -20.17
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_single_entry_cleanups 48.32us 37.76us -21.86
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_validator_prefs 426.54us 331.65us -22.25
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_reward_points 46.11us 35.81us -22.35
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs apply_slash 34.48ms 26.66ms -22.69
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_validator_reward 46.04us 35.56us -22.76
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs chill_other 12.29ms 9.35ms -23.92
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_unstake 16.18ms 11.55ms -28.61
Command output:

✅ Successful benchmarks of runtimes/pallets:
-- asset-hub-westend: ['pallet_staking_async']

Copy link
Copy Markdown
Member

@bkchr bkchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you want to achieve with this PR? Why are you not calling the actual benchmark in a tight loop?

Comment thread substrate/utils/frame/benchmarking-cli/src/pallet/command.rs Outdated
Comment thread substrate/utils/frame/benchmarking-cli/src/pallet/command.rs
Comment thread substrate/utils/frame/benchmarking-cli/src/pallet/command.rs Outdated
Comment thread substrate/utils/frame/benchmarking-cli/src/pallet/command.rs Outdated
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez
Copy link
Copy Markdown
Member Author

ggwpez commented Jan 26, 2026

/cmd bench --pallet pallet_staking_async --runtime asset-hub-westend

@ggwpez ggwpez added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Jan 26, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Command "bench --pallet pallet_staking_async --runtime asset-hub-westend" has started 🚀 See logs here

@ggwpez
Copy link
Copy Markdown
Member Author

ggwpez commented Jan 26, 2026

What do you want to achieve with this PR? Why are you not calling the actual benchmark in a tight loop?

We select the values for the components outside of the Runtime, here. The runtime then gets these component values via dispatch_benchmark.
So if we add a minimum duration in the runtime, then it would run too often (once per component value).
Its not ideal surely, but otherwise we have to change the Benchmarking interface and bump it to V3.

…t_staking_async --runtime asset-hub-westend'
@github-actions
Copy link
Copy Markdown
Contributor

Command "bench --pallet pallet_staking_async --runtime asset-hub-westend" has finished ✅ See logs here

Details

Subweight results:
File Extrinsic Old New Change [%]
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs chill 5.96ms 9.01ms +50.99
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs reap_stash 13.40ms 16.78ms +25.26
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs bond_extra 13.18ms 14.90ms +13.08
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_validator_slash_in_era 417.46us 463.82us +11.11
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_stakers_overview 399.79us 442.33us +10.64
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs rebond 10.70ms 11.72ms +9.52
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_stakers_paged 385.92us 414.41us +7.38
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_claimed_rewards 419.64us 446.36us +6.37
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs kick 2.36ms 2.49ms +5.82
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs rc_on_offence 57.11ms 59.99ms +5.04
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_new_era_always 22.01us 20.84us -5.35
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs apply_slash 34.48ms 32.43ms -5.95
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_unstake 16.18ms 15.07ms -6.82
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_no_eras 22.65us 20.98us -7.37
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_validator_reward 46.04us 42.51us -7.67
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs force_apply_min_commission 67.31us 62.02us -7.85
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_reward_points 46.11us 42.40us -8.06
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs bond 162.70us 149.34us -8.21
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs prune_era_single_entry_cleanups 48.32us 44.30us -8.33
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs chill_other 12.29ms 11.01ms -10.39
cumulus/parachains/runtimes/assets/asset-hub-westend/src/weights/pallet_staking_async.rs unbond 10.29ms 8.63ms -16.11
Command output:

✅ Successful benchmarks of runtimes/pallets:
-- asset-hub-westend: ['pallet_staking_async']

@ggwpez ggwpez added the A4-backport-stable2512 Pull request must be backported to the stable2512 release branch label Feb 4, 2026
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez added this pull request to the merge queue Feb 11, 2026
Merged via the queue into master with commit 822c6f6 Feb 11, 2026
241 of 244 checks passed
@ggwpez ggwpez deleted the oty-bencher-stability branch February 11, 2026 15:52
@paritytech-release-backport-bot
Copy link
Copy Markdown

Created backport PR for stable2512:

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-10794-to-stable2512
git worktree add --checkout .worktree/backport-10794-to-stable2512 backport-10794-to-stable2512
cd .worktree/backport-10794-to-stable2512
git reset --hard HEAD^
git cherry-pick -x 822c6f6f574e5d35e005d24757241300b458ed81
git push --force-with-lease

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A4-backport-stable2512 Pull request must be backported to the stable2512 release branch T1-FRAME This PR/Issue is related to core FRAME, the framework.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants