CI: add benchmark workflow #91
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is how the benchmark will look like: https://trym-b.github.io/warchaeology/benchmarks/ubuntu-22.04/ |
|
This is how a benchmark failure will look like: https://github.com/trym-b/warchaeology/actions/runs/8201485917 |
trym-b
force-pushed
the
ci/add-benchmark-workflow
branch
from
4d004de
to
7185470
Compare
maeb
reviewed
Mar 12, 2024
maeb
reviewed
Mar 12, 2024
maeb
reviewed
Mar 12, 2024
|
Ready for next round @maeb |
maeb
approved these changes
Mar 12, 2024
This commit adds a dummy benchmark to the cat command. Its only purpose is to enable the project to run any benchmarks at all, which is needed by the next commit.
# Motivation To avoid performance regressions, this commit adds a benchmark job to the CI workflow. The idea is that if a given PR decreases the performance sufficiently, (e.g. 150% or more), the CI will fail and the PR cannot be merged. The benchmark results should be visible at https://nlnwa.github.io/warchaeology/benchmarks/ubuntu-22.04 after this commit has been merged. # Changes Added a publish benchmark job in the `publish_benchmarks.yml` that also runs the benchmarks and publishes them to the github pages branch (currently `gh-pages`). # Quirks We need to re-run the benchmark results since getting artifacts from a separate workflow is not trivial. Ideally we should just run the benchmarks once. Publishing benchmarks must live in a separate workflow, since the permissions it needs to write to `gh-pages` should not be exposed to external users creating PRs targeting this repository. An alternative would be to not run workflows when receiving external PRs, but that would make the developer experience for contributors worse. # Future work After this PR has been merged, doing the actual comparison should be a priority. Currently the benchmarks are only published to the `gh-pages` branch, and not actually compared to anything. Add `Windows` to the benchmark matrix.
trym-b
force-pushed
the
ci/add-benchmark-workflow
branch
from
d052900
to
343fb0a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Best reviewed commit by commit