Add benchmarking for parachain runtime initializer pallet #3913

KiChjang · 2021-09-23T06:30:50Z

Part of #3850.

KiChjang · 2021-09-23T06:32:02Z

/benchmark runtime westend runtime_parachains::initializer

parity-benchapp · 2021-09-23T06:32:03Z

Benchmark Runtime Westend Pallet for branch "kckyeung/parachains-initializer-benchmarking" with command cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_initializer.rs

Results

error[E0207]: the type parameter `T` is not constrained by the impl trait, self type, or predicates
  --> runtime/parachains/src/initializer/weights.rs:35:6
   |
35 | impl<T: frame_system::Config> super::WeightInfo for () {
   |      ^ unconstrained type parameter

error: aborting due to previous error

For more information about this error, try `rustc --explain E0207`.
error: could not compile `polkadot-runtime-parachains`

To learn more, run the command again with --verbose.
error: build failed

ERROR: Unable to commit file ./runtime/westend/src/weights/runtime_parachains_initializer.rs

KiChjang · 2021-09-23T06:42:34Z

/benchmark runtime westend runtime_parachains::initializer

parity-benchapp · 2021-09-23T06:42:35Z

Benchmark Runtime Westend Pallet for branch "kckyeung/parachains-initializer-benchmarking" with command cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_initializer.rs

Results

Pallet: "runtime_parachains::initializer", Extrinsic: "force_approve", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Digest (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    4.257
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    4.257
              µs

Reads = 1
Writes = 1

…kckyeung/parachains-initializer-benchmarking

…k --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_initializer.rs

KiChjang · 2021-09-23T06:55:11Z

/benchmark runtime kusama runtime_parachains::initializer

parity-benchapp · 2021-09-23T07:03:31Z

Benchmark Runtime Kusama Pallet for branch "kckyeung/parachains-initializer-benchmarking" with command cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_initializer.rs

Results

Pallet: "runtime_parachains::initializer", Extrinsic: "force_approve", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Digest (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    4.297
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    4.297
              µs

Reads = 1
Writes = 1

…k --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_initializer.rs

emostov · 2021-09-24T01:38:08Z

runtime/westend/src/lib.rs

@@ -805,7 +805,7 @@ impl pallet_proxy::Config for Runtime {
 impl parachains_origin::Config for Runtime {}

 impl parachains_configuration::Config for Runtime {
-	type WeightInfo = parachains_configuration::weights::WeightInfo<Runtime>;
+	type WeightInfo = weights::runtime_parachains_configuration::WeightInfo<Runtime>;


I think #3914 should take of this

emostov · 2021-09-24T01:45:09Z

runtime/parachains/src/initializer/benchmarking.rs

+		);
+	}
+}
+


Essentially I think the only storage op here is an <Digest<T>>::append(item); - without looking at the code I am guessing it has to decode the vec, push the item, and then encode it (But maybe it is more optimized?). I am wondering if for something like this we should benchmark with a large vec to assume worst case?

My naive guess though is that practically we will never have enough logs for the append op to be that expensive + this is from a force origin so it doesn't look like a significant DoS vector.

I looked through the code, and while it doesn't decode the entire vec (it only decodes the first 4 bytes to get the encoded length), it does however copy the entire vec to a new one whenever the new encoded length is different from the previous (which is pretty much always).

I'll most likely need to add in a variable and see how the complexity varies with the digest vec length.

emostov

generally looks good to me. Maybe worth looking into if filling up the vec of digest logs would get a better worst case benchmark.

…lizer-benchmarking

KiChjang · 2021-09-25T04:38:51Z

/benchmark runtime westend runtime_parachains::initializer

parity-benchapp · 2021-09-25T04:38:53Z

Benchmark Runtime Westend Pallet for branch "kckyeung/parachains-initializer-benchmarking" with command cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_initializer.rs

Results

Pallet: "runtime_parachains::initializer", Extrinsic: "force_approve", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Digest (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    5.767
    + d    0.001
              µs

Reads = 1 + (0 * d)
Writes = 1 + (0 * d)

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    d   mean µs  sigma µs       %
    0     4.311     0.072    1.6%
 1310     7.196      0.22    3.0%
 2620     9.644     0.098    1.0%
 3930     12.65      0.09    0.7%
 5240     14.99     0.105    0.7%
 6550     16.38     0.116    0.7%
 7860     17.82     0.064    0.3%
 9170      20.2     0.126    0.6%
10480     22.56     0.475    2.1%
11790     23.07      0.07    0.3%
13100     25.74     0.143    0.5%
14410        27     0.149    0.5%
15720      24.1     0.201    0.8%
17030     31.14     0.131    0.4%
18340     33.81     0.163    0.4%
19650     35.15     0.166    0.4%
20960     37.67     0.265    0.7%
22270      40.1     0.137    0.3%
23580     41.58     0.104    0.2%
24890     44.89     0.545    1.2%
26200     46.82     0.249    0.5%
27510     48.37      0.16    0.3%
28820     50.37      0.55    1.0%
30130     53.03     0.113    0.2%
31440     53.47     0.182    0.3%
32750     55.61     0.635    1.1%
34060     57.03      0.22    0.3%
35370     60.47     0.415    0.6%
36680     63.62     0.808    1.2%
37990     65.49     0.902    1.3%
39300     66.66     0.481    0.7%
40610     68.27     0.289    0.4%
41920     71.79     0.734    1.0%
43230     73.22     0.415    0.5%
44540     74.86      0.17    0.2%
45850     76.35     0.778    1.0%
47160     78.69     0.767    0.9%
48470     88.04     6.141    6.9%
49780     86.31     1.057    1.2%
51090     88.52      0.72    0.8%
52400     90.34     1.531    1.6%
53710     90.04     1.607    1.7%
55020     89.61     0.981    1.0%
56330     90.23     0.616    0.6%
57640     73.68     0.317    0.4%
58950     75.72     0.222    0.2%
60260     99.38     0.597    0.6%
61570     99.74      0.57    0.5%
62880     102.7     2.895    2.8%
64190     96.18     9.046    9.4%
65500     104.8     0.554    0.5%

Quality and confidence:
param     error
d             0

Model:
Time ~=    6.607
    + d    0.001
              µs

Reads = 1 + (0 * d)
Writes = 1 + (0 * d)

…k --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_initializer.rs

KiChjang · 2021-09-25T05:03:34Z

/benchmark runtime kusama runtime_parachains::initializer

parity-benchapp · 2021-09-25T05:03:35Z

Benchmark Runtime Kusama Pallet for branch "kckyeung/parachains-initializer-benchmarking" with command cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_initializer.rs

Results

Pallet: "runtime_parachains::initializer", Extrinsic: "force_approve", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Digest (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    5.441
    + d    0.001
              µs

Reads = 1 + (0 * d)
Writes = 1 + (0 * d)

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    d   mean µs  sigma µs       %
    0     4.311     0.073    1.6%
 1310     7.376     0.122    1.6%
 2620     9.769     0.044    0.4%
 3930     12.47     0.102    0.8%
 5240     14.89     0.117    0.7%
 6550     16.15     0.125    0.7%
 7860     17.95     0.113    0.6%
 9170     20.35     0.078    0.3%
10480     22.29     0.085    0.3%
11790     23.88     0.145    0.6%
13100     26.29      0.16    0.6%
14410     28.19     0.129    0.4%
15720     30.06     0.105    0.3%
17030     31.63     0.182    0.5%
18340     34.01     0.192    0.5%
19650      35.8     0.178    0.4%
20960     38.46     0.147    0.3%
22270     39.89     0.068    0.1%
23580     41.92     0.221    0.5%
24890     46.18     1.054    2.2%
26200     47.69     0.493    1.0%
27510     48.59     0.068    0.1%
28820     49.83     0.143    0.2%
30130     53.21     0.159    0.2%
31440     54.74     0.142    0.2%
32750     56.06     0.226    0.4%
34060     58.22     0.102    0.1%
35370     60.06     0.193    0.3%
36680     63.26      0.16    0.2%
37990     65.29     0.128    0.1%
39300     67.04     0.576    0.8%
40610     68.43     0.153    0.2%
41920     72.04     0.142    0.1%
43230      73.3     0.373    0.5%
44540     75.43     0.182    0.2%
45850     77.55     0.371    0.4%
47160     78.43     0.259    0.3%
48470     84.68     0.278    0.3%
49780     84.78     1.042    1.2%
51090     86.37     0.279    0.3%
52400     88.97     0.577    0.6%
53710     90.51     0.268    0.2%
55020      92.1     0.843    0.9%
56330     96.07     3.941    4.1%
57640      94.5     0.224    0.2%
58950     96.49     0.487    0.5%
60260     103.5     0.687    0.6%
61570     104.3     0.525    0.5%
62880     105.2     0.529    0.5%
64190     109.4     0.949    0.8%
65500     109.4     3.791    3.4%

Quality and confidence:
param     error
d             0

Model:
Time ~=    5.156
    + d    0.002
              µs

Reads = 1 + (0 * d)
Writes = 1 + (0 * d)

…k --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_initializer.rs

KiChjang · 2021-09-25T07:23:54Z

bot merge

ghost · 2021-09-25T07:23:58Z

Merge aborted: Checks failed for 48d549e

KiChjang · 2021-09-28T04:55:15Z

bot merge

ghost · 2021-09-28T04:55:19Z

Trying merge.

* master: feat: measured oneshots (#3902) remove `AllSubsystems` and `AllSubsystemsGen` types (#3874) Companion for Substrate#9867 (#3938) Substrate Companion for #9552 (#3834) CI: run disputes tests (#3962) Bump parity-scale-codec from 2.3.0 to 2.3.1 (#3959) approval-voting: populate session cache in advance (#3954) Bump libc from 0.2.102 to 0.2.103 (#3950) fix master (#3955) Docker files chore (#3880) Bump nix from 0.19.1 to 0.20.0 (#3587) remove connected disconnected state, 3rd attempt (#3898) fix flaky chain-selection tests (#3948) Add benchmarking for parachain runtime initializer pallet (#3913) minor chore changes (#3944) disputes: reject single-sided disputes (#3903)

Add benchmarking for parachain runtime initializer pallet

b65c5e2

KiChjang requested review from shawntabrizi, kianenigma, gui1117 and emostov September 23, 2021 06:30

KiChjang mentioned this pull request Sep 23, 2021

Benchmark Parachains Runtime: Tracking Issue #3850

Closed

14 tasks

Fix default impl for initializer pallet WeightInfo

b425d22

Parity Bot added 2 commits September 23, 2021 06:42

Merge branch 'master' of https://github.com/paritytech/polkadot into …

c85f1cf

…kckyeung/parachains-initializer-benchmarking

Parity Bot and others added 2 commits September 23, 2021 07:12

Use real weights and use max_block as default weight

694dfa4

gui1117 approved these changes Sep 23, 2021

View reviewed changes

emostov reviewed Sep 24, 2021

View reviewed changes

emostov approved these changes Sep 24, 2021

View reviewed changes

KiChjang added 4 commits September 24, 2021 19:32

Add variable for digest vec length for initializer benchmark

bf724d8

Fix compilation errors

196ccf3

Merge remote-tracking branch 'origin' into kckyeung/parachains-initia…

e2cabdc

…lizer-benchmarking

Add WeightInfo to parachains_initializer config in polkadot runtime

ec78041

Parity Bot and others added 3 commits September 25, 2021 05:12

Appease spellchecker

acdb91a

Use kusama weights in polkadot runtime for parachain initializer pallet

48d549e

ghost merged commit 211a4fc into master Sep 28, 2021

ghost deleted the kckyeung/parachains-initializer-benchmarking branch September 28, 2021 04:55

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmarking for parachain runtime initializer pallet #3913

Add benchmarking for parachain runtime initializer pallet #3913

KiChjang commented Sep 23, 2021

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

emostov Sep 24, 2021

emostov Sep 24, 2021 •

edited

Loading

KiChjang Sep 24, 2021

emostov left a comment

KiChjang commented Sep 25, 2021

parity-benchapp bot commented Sep 25, 2021 •

edited

Loading

KiChjang commented Sep 25, 2021

parity-benchapp bot commented Sep 25, 2021 •

edited

Loading

KiChjang commented Sep 25, 2021

ghost commented Sep 25, 2021

KiChjang commented Sep 28, 2021

ghost commented Sep 28, 2021

Add benchmarking for parachain runtime initializer pallet #3913

Add benchmarking for parachain runtime initializer pallet #3913

Conversation

KiChjang commented Sep 23, 2021

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 • edited Loading

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 • edited Loading

KiChjang commented Sep 23, 2021

parity-benchapp bot commented Sep 23, 2021 • edited Loading

emostov Sep 24, 2021

Choose a reason for hiding this comment

emostov Sep 24, 2021 • edited Loading

Choose a reason for hiding this comment

KiChjang Sep 24, 2021

Choose a reason for hiding this comment

emostov left a comment

Choose a reason for hiding this comment

KiChjang commented Sep 25, 2021

parity-benchapp bot commented Sep 25, 2021 • edited Loading

KiChjang commented Sep 25, 2021

parity-benchapp bot commented Sep 25, 2021 • edited Loading

KiChjang commented Sep 25, 2021

ghost commented Sep 25, 2021

KiChjang commented Sep 28, 2021

ghost commented Sep 28, 2021

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

parity-benchapp bot commented Sep 23, 2021 •

edited

Loading

emostov Sep 24, 2021 •

edited

Loading

parity-benchapp bot commented Sep 25, 2021 •

edited

Loading

parity-benchapp bot commented Sep 25, 2021 •

edited

Loading