Add benchmark machine

[ggwpez]

👉 Please put feedback about this command either in here or ping me on matrix @oliver.tale-yazdi:matrix.parity.io.

First version of a benchmark machine command using the sc-sysinfo crate (:pray: @koute).
@shawntabrizi wanted to have a MVP version handy.

Changes:

• Add benchmark machine command
• Integrate it into the substrate node and node-template
• Move new_rng to shared
• Add --no-hardware-benchmarks where applicable

Output:

$ substrate benchmark machine --dev

+----------+----------------+-------+------+
| Category | Function       | Score | Unit |
+----------+----------------+-------+------+
| CPU      | BLAKE2-256     | 1029  | MB/s |
+----------+----------------+-------+------+
| CPU      | SR25519 Verify | 666.6 | KB/s |
+----------+----------------+-------+------+
| Memory   | Copy           | 14668 | MB/s |
+----------+----------------+-------+------+
| Disk     | Seq Write      | 502   | MB/s |
+----------+----------------+-------+------+
| Disk     | Rnd Write      | 227   | MB/s |
+----------+----------------+-------+------+

Next steps:

• Add check that the build profile was production
• Extract more info from sc-sysinfo (eg. total time, reps, input len)
• Pass in thresholds such that every chain can have theirs own requirements
• Print results relative to thresholds
• Measure on ref-hardware to get polkadot thresholds

Polkadot companion: paritytech/polkadot#5326
Cumulus companion: paritytech/cumulus#1172

[koute]

It'd probably be better to modify sc-sysinfo to expose the benchmark_* functions separately and call them one by one here instead of relying the gather_hwbench.

The gather_hwbench is deliberately simple (only takes a path as an argument) and ignores error (if the disk benches fail it'll set their results to None and only print out a warning in the logs) since it's meant to be run at startup to gather the data for telemetry.

Here however it'd be good to 1) have direct access to the errors (which the benchmark_* functions do return), and 2) be able to configure the benchmarks to be able to override the limit of how long they can run and for how many iterations since this subcommand doesn't have the same timing constraints as the telemetry (so we can just add an extra argument to the benchmark_* functions which would allow you to override that).

[ggwpez]

Yea getting some more configurations in there would be good.

[gilescope]

In the spirit of pits of success, maybe add an extra line in the output if they're running in debug mode warning them that they should only trust the results if they are running in release mode?

[kianenigma]

nit: missing doc on public function. Does it need to be public?

[ggwpez]

Will add 😄
Currently it needs to be public since I call these functions directly. IMHO it would be better to have a gather function which calls all of them and returns a struct,
but thats probably next.

[koute]

IMHO it would be better to have a gather function which calls all of them and returns a struct,

Hmm... although the downside of that is that you won't be able to configure them separately (set a different ExecutionLimit for each benchmark), unless you add another struct which will have multiple ExecutionLimits in it, but then if you're going to do that you might as well have them as a separate functions I guess?

(And it might make sense to set different limits on each benchmark because e.g. the disk benchmarks are going to require more time than the memory benchmark to make sure their result isn't noisy, if you absolutely want to minimize their error.)

[kianenigma]

LGTM!

[kianenigma]

bot merge