Skip to content

Benchmark storage access on block validation#8069

Merged
AndreiEres merged 91 commits intomasterfrom
AndreiEres/benchmark-storage-access-on-block-validation
May 22, 2025
Merged

Benchmark storage access on block validation#8069
AndreiEres merged 91 commits intomasterfrom
AndreiEres/benchmark-storage-access-on-block-validation

Conversation

@AndreiEres
Copy link
Copy Markdown
Contributor

@AndreiEres AndreiEres commented Mar 27, 2025 •
edited
Loading

Fixes #7987
Fixes #7868

@AndreiEres AndreiEres changed the base branch from master to alexggh/fixup_storage_read_write_benchmark12 March 27, 2025 14:52
@AndreiEres
Copy link
Copy Markdown
Contributor Author

AndreiEres commented Apr 1, 2025 •
edited
Loading

Results I received from a asset-hub-westend snapshot to compare weights from this pr and master. Ran on Apple Macbook M2 Pro.

cargo run -p polkadot-parachain-bin --features=runtime-benchmarks --profile production -- benchmark 
storage --skip-write --warmups 1 --state-version 1 --chain cumulus/polkadot-parachain/chain-specs/asset-hub-westend.json --detailed-log-output --base-path ~/code/snapshots/polkadot-
parachain

PR, average read: 11_495
Total: 1126520
Min: 9987, Max: 13019
Average: 11495, Median: 11496, Stddev: 928.21
Percentiles 99th, 95th, 75th: 13019, 12860, 12311
Value size summary:
Total: 5075
Min: 48, Max: 236
Average: 51, Median: 49, Stddev: 18.95
Percentiles 99th, 95th, 75th: 236, 56, 49

Master, average read 24_538
Total: 23945839198
Min: 4375, Max: 1291292
Average: 24538, Median: 23542, Stddev: 13477.33
Percentiles 99th, 95th, 75th: 38167, 30709, 25958
Value size summary:
Total: 51060368
Min: 0, Max: 1874775
Average: 52, Median: 76, Stddev: 1922.44
Percentiles 99th, 95th, 75th: 80, 80, 80

@AndreiEres AndreiEres force-pushed the AndreiEres/benchmark-storage-access-on-block-validation branch from 839b5c9 to 156dc9d Compare April 2, 2025 14:55
@AndreiEres AndreiEres mentioned this pull request Apr 2, 2025
Closed
@alexggh alexggh self-requested a review April 3, 2025 07:57
Base automatically changed from alexggh/fixup_storage_read_write_benchmark12 to master April 8, 2025 10:55
alexggh and others added 23 commits April 8, 2025 14:22
@alexggh @AndreiEres
Fixup read/write benchmark
3dd3249 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh @AndreiEres
Address review feedback
4540be0 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh @AndreiEres
Make clippy happy
bd3b0cb 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh @bkchr @AndreiEres
Update substrate/utils/frame/benchmarking-cli/src/storage/cmd.rs
5c3b0b3 
Co-authored-by: Bastian Köcher <git@kchr.de>
@alexggh @bkchr @AndreiEres
Update substrate/utils/frame/benchmarking-cli/src/storage/read.rs
6b59fe8 
Co-authored-by: Bastian Köcher <git@kchr.de>
@alexggh @bkchr @AndreiEres
Update substrate/utils/frame/benchmarking-cli/src/storage/read.rs
22e4cf4 
Co-authored-by: Bastian Köcher <git@kchr.de>
@alexggh @bkchr @AndreiEres
Update substrate/utils/frame/benchmarking-cli/src/storage/write.rs
e38d2ca 
Co-authored-by: Bastian Köcher <git@kchr.de>
@alexggh @AndreiEres
Apply review feedback
b33f629 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh @AndreiEres
Add prdoc
9ac3a52 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh @AndreiEres
Update pr_7867.prdoc
18caf04
@alexggh @AndreiEres
Fixup prdoc
022af2d 
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@AndreiEres
Add dummy test runtime
8699d5d
@AndreiEres
Run empty validation
dda8492
@AndreiEres
Add the working benchmark but without reading child keys
0c2e183
@AndreiEres
Add dry run correction
1d4f01e
@AndreiEres
Update params
247fb6d
@AndreiEres
Merge payload and modes
427ba86
@AndreiEres
Move common function
73ed34a
@AndreiEres
Add write bench
71a24d9
@AndreiEres
Add on_block_validation
acfb95b
@AndreiEres
Add different measurements
703675c
@AndreiEres
Update read bench to use on_block_validation
ce8120a
@AndreiEres
Remove todo
781e0d3
@AndreiEres AndreiEres enabled auto-merge May 22, 2025 12:08
@AndreiEres AndreiEres added this pull request to the merge queue May 22, 2025
Merged via the queue into master with commit 48d149b May 22, 2025
242 of 245 checks passed
@AndreiEres AndreiEres deleted the AndreiEres/benchmark-storage-access-on-block-validation branch May 22, 2025 17:41
github-merge-queue bot pushed a commit that referenced this pull request May 23, 2025
@alexggh @github-actions
Use hashbrown hashmap/hashset in validation context (#8606)
85f5dc7 
Discovered while profiling
#6131 (comment)
with the benchmark #8069
that when running in validation a big chunk of the time is spent
inserting and retrieving data from the BTreeMap/BTreeSet.

By switching to hashbrown HashMap/HashSet in validation TrieCache and
TrieRecorder and the memory-db
paritytech/trie#221 read costs improve with
around ~40% and write with about ~20%

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
ordian added a commit that referenced this pull request May 27, 2025
@ordian
Merge branch 'master' into ao-fix-issue-64
67a1bfe 
* master: (99 commits)
  Snowbridge: Remove asset location check for compatibility (#8473)
  add poke_deposit extrinsic to pallet-bounties (#8382)
  litep2p/peerset: Reject non-reserved peers in the reserved-only mode (#8650)
  Charge deposit based on key length (#8648)
  [pallet-revive] make subscription task panic on error (#8587)
  tx/metrics: Add metrics for the RPC v2 `transactionWatch_v1_submitAndWatch` (#8345)
  Bridges: Fix - Improve try-state for pallet-xcm-bridge-hub (#8615)
  Introduce CreateBare, deprecated CreateInherent (#7597)
  Use hashbrown hashmap/hashset in validation context (#8606)
  ci: rm gitlab config (#8622)
  🔪 flaky and Zombienet tests (#8600)
  cumulus: adjust unincluded segment size metric buckets (#8617)
  Benchmark storage access on block validation (#8069)
  Revert 7934 es/remove tj changes (#8611)
  collator-protocol: add more collation observability (#8230)
  `fatxpool`: add fallback for ready at light (#8533)
  txpool: fix tx removal from unlocks set (#8500)
  XCMP weight metering: account for the MQ page position (#8344)
  fix epmb solution duplicate issue + add remote mining apparatus to epm (#8585)
  Fix generated address returned by Substrate RPC runtime call (#8504)
  ...
github-merge-queue bot pushed a commit that referenced this pull request May 28, 2025
@alexggh
asset-hub-westend: use inmemorydb weights (#8359)
4b5a9ad 
Modify asset-hub-westend to use the read/write costs in validate block
on validator reference hardware, since these are
weights that guarantee the blocks will pass validation and make it on
the relay chain.

This were obtained with the benchmark created here:
#8069 and it contains all
the optimisation brought by
#8607.

Fixes: #7537

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
pgherveou pushed a commit that referenced this pull request Jun 11, 2025
@AndreiEres @alexggh
@bkchr @pgherveou
Benchmark storage access on block validation (#8069)
6f6efe1 
Fixes #7987
Fixes #7868

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
pgherveou pushed a commit that referenced this pull request Jun 11, 2025
@alexggh @github-actions @pgherveou
Use hashbrown hashmap/hashset in validation context (#8606)
ea758f0 
Discovered while profiling
#6131 (comment)
with the benchmark #8069
that when running in validation a big chunk of the time is spent
inserting and retrieving data from the BTreeMap/BTreeSet.

By switching to hashbrown HashMap/HashSet in validation TrieCache and
TrieRecorder and the memory-db
paritytech/trie#221 read costs improve with
around ~40% and write with about ~20%

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
pgherveou pushed a commit that referenced this pull request Jun 11, 2025
@alexggh @pgherveou
asset-hub-westend: use inmemorydb weights (#8359)
b75da6b 
Modify asset-hub-westend to use the read/write costs in validate block
on validator reference hardware, since these are
weights that guarantee the blocks will pass validation and make it on
the relay chain.

This were obtained with the benchmark created here:
#8069 and it contains all
the optimisation brought by
#8607.

Fixes: #7537

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
alexggh added a commit to alexggh/runtimes that referenced this pull request Sep 22, 2025
@alexggh
asset-hub-kusama: use inmemorydb weights
c3cfee0 
Modify asset-hub-kusama to use the read/write costs in validate block on
validator reference hardware, since these are weights that guarantee the
blocks will pass validation and make it on the relay chain.

This were obtained with the benchmark created here: paritytech/polkadot-sdk#8069.

Part of: paritytech/polkadot-sdk#6131
Fixes: polkadot-fellows#800

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh alexggh mentioned this pull request Sep 22, 2025
Merged
alvicsam pushed a commit that referenced this pull request Oct 17, 2025
@AndreiEres @alexggh
@bkchr @alvicsam
Benchmark storage access on block validation (#8069)
5cb73bc 
Fixes #7987
Fixes #7868

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
alvicsam pushed a commit that referenced this pull request Oct 17, 2025
@alexggh @github-actions @alvicsam
Use hashbrown hashmap/hashset in validation context (#8606)
e3f4cea 
Discovered while profiling
#6131 (comment)
with the benchmark #8069
that when running in validation a big chunk of the time is spent
inserting and retrieving data from the BTreeMap/BTreeSet.

By switching to hashbrown HashMap/HashSet in validation TrieCache and
TrieRecorder and the memory-db
paritytech/trie#221 read costs improve with
around ~40% and write with about ~20%

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
alvicsam pushed a commit that referenced this pull request Oct 17, 2025
@alexggh @alvicsam
asset-hub-westend: use inmemorydb weights (#8359)
9d93450 
Modify asset-hub-westend to use the read/write costs in validate block
on validator reference hardware, since these are
weights that guarantee the blocks will pass validation and make it on
the relay chain.

This were obtained with the benchmark created here:
#8069 and it contains all
the optimisation brought by
#8607.

Fixes: #7537

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@github-actions github-actions bot mentioned this pull request Oct 24, 2025
Closed
alexggh added a commit to polkadot-fellows/runtimes that referenced this pull request Dec 1, 2025
@alexggh
asset-hub-kusama: use inmemorydb weights (#918)
43ec4a2 
Modify asset-hub-kusama to use the read/write costs computed in validate
block on validator reference hardware, since these are weights that
guarantee the blocks will pass validation and make it on the relay
chain.

This were obtained with the benchmark created here:
paritytech/polkadot-sdk#8069.

Part of: #800

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh alexggh mentioned this pull request Dec 4, 2025
Merged
alexggh added a commit to polkadot-fellows/runtimes that referenced this pull request Dec 12, 2025
@alexggh
asset-hub-polkadot: use inmemorydb weights (#1025)
9e6c9aa 
Modify asset-hub-polkadot to use the read/write costs computed in
validate block on validator reference hardware, since these are weights
that guarantee the blocks will pass validation and make it on the relay
chain.

This were obtained with the benchmark created here:
paritytech/polkadot-sdk#8069.

Part of: #800

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bkchr bkchr bkchr approved these changes

@alexggh alexggh alexggh approved these changes

Assignees

No one assigned

Labels

T12-benchmarks This PR/Issue is related to benchmarking and weights.

Projects

[preview] release tracker
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve accuracy of parachain weight benchmarking Blocks run with high TrieCache hits seems to be 20% faster to produce than validate

4 participants

@AndreiEres @bkchr @alexggh @pgherveou