Add runtime-benchmarks for frame-benchmarking

[zjb0807]

Add runtime-benchmarks feature for frame-benchmarking to control baseline benchmarks.

polkadot companion: paritytech/polkadot#4387
cumulus companion: paritytech/cumulus#802

[bkchr]

What is the reason for this pr?

[zjb0807]

What is the reason for this pr?

benchmarks related code should be enabled in #![cfg(feature = "runtime-benchmarks")], otherwise some scenarios will cause pollution.(.e.g --no-default-features --target=wasm32-unknown-unknown)

https://github.com/open-web3-stack/open-runtime-module-library/runs/4281316679?check_suite_focus=true

[bkchr]

Why has your bencher the crate imported when the runtime-benchmarks feature is not enabled?

[zjb0807]

Why has your bencher the crate imported when the runtime-benchmarks feature is not enabled?

This is a module that imports some methods of frame_benchmarking, not runtime, so there is no runtime-benchmarks feature.
I think the benchmarks related code should be enabled with runtime-benchmarks feature.
@shawntabrizi

[shawntabrizi]

Looks like there is a lot more places this needs to be updated?

Just search for "frame-benchmarking"

And also Polkadot + Cumulus companion

[shawntabrizi]

/benchmark runtime pallet pallet_balances

[parity-benchapp]

Benchmark Runtime Pallet for branch "update-baseline" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results

Pallet: "pallet_balances", Extrinsic: "transfer", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    60.54
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    60.54
              µs

Reads = 1
Writes = 1

Pallet: "pallet_balances", Extrinsic: "transfer_keep_alive", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    46.11
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    46.11
              µs

Reads = 1
Writes = 1

Pallet: "pallet_balances", Extrinsic: "set_balance_creating", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    27.38
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    27.38
              µs

Reads = 1
Writes = 1

Pallet: "pallet_balances", Extrinsic: "set_balance_killing", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    31.86
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    31.86
              µs

Reads = 1
Writes = 1

Pallet: "pallet_balances", Extrinsic: "force_transfer", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:2 w:2)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    61.66
              µs

Reads = 2
Writes = 2

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    61.66
              µs

Reads = 2
Writes = 2

Pallet: "pallet_balances", Extrinsic: "transfer_all", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    57.22
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    57.22
              µs

Reads = 1
Writes = 1

Pallet: "pallet_balances", Extrinsic: "force_unreserve", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: System Account (r:1 w:1)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    25.56
              µs

Reads = 1
Writes = 1

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    25.56
              µs

Reads = 1
Writes = 1

ERROR: Unable to push ./frame/balances/src/weights.rs

[zjb0807]

Looks like there is a lot more places this needs to be updated?

Other places just import the frame-benchmarking without calling the benchmarks in frame-benchmarking, so it does not affect. I can add on if needed.

[zjb0807]

gitlab-check-dependent-cumulus error:

the package `parachain-template-runtime` depends on `frame-benchmarking`, with features: `runtime-benchmarks` but `frame-benchmarking` does not have these features.

gitlab-check-dependent-polkadot error:

the package `kusama-runtime` depends on `frame-benchmarking`, with features: `runtime-benchmarks` but `frame-benchmarking` does not have these features.

But the feature is added. And paritytech/polkadot#4387 works on my machine. Is there a problem with CI?

[zjb0807]

The Polkadot and Cumulus companion don't execute diener patch. How to fix the problem?

[kianenigma]

CI is broken for now. We either wait or merge later manually. Thanks!