Add CI Job to Run Benchmark and Publish Results

[Brandon-Kimberly]

Is your feature request related to a problem? Please describe.

Yes. Currently the script used to benchmark must be run manually. This can lead to out-of-date or missing benchmark performance data.

Describe the solution you'd like

I would like a new job added to the CI pipeline that would run the benchmark script, then publish the results as an artifact. This job could run on ever push or every release, or at some other interval.

Describe alternatives you've considered

The alternative is to keep running the script manually. As mentioned above, this may result in benchmark data being out-of-sync with the actual code. Automatically running the benchmark script and publishing the results will ensure that the two are always synchronized.

[sidharthv96]

I'd like to work on this.

Deno is pushing benchmarks to a git repo in their CI.

@dyladan Will a similar approach be enough? If so, any pointers to which repo should be used to push the data would be helpful.

[dyladan]

Yes, I think a similar approach would be enough. I would push it to this repo.

[sidharthv96]

@dyladan Not sure if pushing to this repo will work from within CI, I'm not very familiar with CircleCI.
I think the data has to be maintained in a separate repo, because otherwise there will be merge conflicts right?

Currently benchmark data is exported as JSON and stored with historical data.
Kinda stuck on the pushing part though.

[dyladan]

It should be implemented as a github action and can either be triggered by a master push, or a release publish. For an example of a release publish triggered build you can look at what we are doing for the docs right now at .github/workflows/docs.yml

[pichlermarc]

done in #4144, benchmarking is now tracked by #4171