Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance regression testing in CI #470

Open
hidmic opened this issue Feb 10, 2025 · 0 comments
Open

Performance regression testing in CI #470

hidmic opened this issue Feb 10, 2025 · 0 comments
Labels
enhancement New feature or request infra Related to infrastructure and CI

Comments

@hidmic
Copy link
Collaborator

hidmic commented Feb 10, 2025

Feature description

We have been using benchmarks extensively to validate and steer. Alas, we have no way to include them in our CI pipelines. Nevermind code not having fully settled in its place (at least as of the time of writing), benchmarks take an inordinate amount of time to run. A developer cannot wait a day for CI check to come back. We need shorter, yet meaningful benchmarks and a suitable infrastructure for these to run repeatably on code submission.

Implementation considerations

LAMBKIN's black-box techniques may be reused on shorter datasets, but there may be other tools and approaches (e.g. SLAMBench) from which to draw ideas considering we would be benchmarking Beluga against a modified version of itself in the regression performance testing case.

@hidmic hidmic added enhancement New feature or request infra Related to infrastructure and CI labels Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request infra Related to infrastructure and CI
Projects
None yet
Development

No branches or pull requests

1 participant