Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate criterion statistics and plots into CI #5354

Closed
konstin opened this issue Jun 25, 2023 · 1 comment
Closed

Integrate criterion statistics and plots into CI #5354

konstin opened this issue Jun 25, 2023 · 1 comment
Labels
performance Potential performance improvement

Comments

@konstin
Copy link
Member

konstin commented Jun 25, 2023

For each PR, our CI compares running criterion benchmarks on main vs. running them on the PR and posts a comment with the timing changes. Unfortunately, the github actions runner performance is rather noisy, so e.g. in this change that doesn't touch the linter at all we still see large seeming changes to linter performance.

criterion, the benchmarking framework we use, integrates statistical methods and e.g. computes p-values that tells you whether a change is statistically significant (vs. the just noise) and it plots each run to allow for manual inspection. It would be great if we could extend CI to show more information, e.g. whether criterion considers the difference significant. I don't know if that is possible with the way github handles artifacts, but exporting the criterion plots and linking them on the PR comment would also be helpful.

@konstin konstin added the performance Potential performance improvement label Jun 25, 2023
@charliermarsh
Copy link
Member

I think this is less relevant now that we're on CodSpeed, but can always revisit if we migrate away.

@charliermarsh charliermarsh closed this as not planned Won't fix, can't repro, duplicate, stale Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Potential performance improvement
Projects
None yet
Development

No branches or pull requests

2 participants