You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently there is no automated or semi-automated way to run benchmarks in the repo and are done manually.
We need to be able to automate benchmarking on the reference machine before PRs are merged to the master.
Correct benchmarks are crucial to security of the network and forgetting to do this or missing important change can cause chain stall.
TLDR; Compare old and new benchmark results, if the new bench results differ by amount of reads/writes or some threshold % (let's say 1%), force update benchmarks.
We had a bench wizard before, can we revisit this? Why did it stop working?
The text was updated successfully, but these errors were encountered:
One good benefit I see in having them manually updated by us is that it gives us a chance to review the bench results locally before committing them, potentially spotting some issues early.
By having them automatically done we might easily overlook some results. Threshold can also help spot big changes (like an extrinsic has become 10-20% cheaper by accident or wrong benchhmark function), the question would be what is the right threshold as sometimes 10-20% change is also acceptable
I still like the idea, we just need to make it bulletproof.
By having them automatically done we might easily overlook some results.
To be clear - I don't want to push the changes automatically. I want to check if the benchmarks changed and sound alarm if they did. I would say that's the opposite of overlooking it.
We should then decide if we made mistake or if we should improve something or just commit re-benchmarked results.
Currently there is no automated or semi-automated way to run benchmarks in the repo and are done manually.
We need to be able to automate benchmarking on the reference machine before PRs are merged to the master.
Correct benchmarks are crucial to security of the network and forgetting to do this or missing important change can cause chain stall.
TLDR; Compare old and new benchmark results, if the new bench results differ by amount of reads/writes or some threshold % (let's say 1%), force update benchmarks.
We had a bench wizard before, can we revisit this? Why did it stop working?
The text was updated successfully, but these errors were encountered: