Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BENCH: sync codspeed-benchmarks with BLAS-benchmarks #4763

Merged
merged 4 commits into from
Jun 30, 2024

Conversation

ev-br
Copy link
Contributor

@ev-br ev-br commented Jun 24, 2024

Sync the benchmark suite we run on codspeed with http://www.openmathlib.org/BLAS-Benchmarks/#summarylist?sort=0&dir=asc:

  • run s-/d-/c-/z- variants
  • run more matrix sizes

@ev-br
Copy link
Contributor Author

ev-br commented Jun 24, 2024

Expanding the benchmark suite of course takes longer. Especially given that codspeed runs things under callgrind, it is not entirely unexpected to be slow. I'm not sure what are their free usage limits,

Previously: the codspeed run was under 5 minutes (e.g. https://github.com/OpenMathLib/OpenBLAS/actions/runs/9632137639/job/26564891418)
This PR: an hour and countin (https://github.com/OpenMathLib/OpenBLAS/actions/runs/9644610499/job/26597037247?pr=4763)

I wonder if we should keep the a small set of system sizes and run s-/d-/c-/z- variants?

Copy link

codspeed-hq bot commented Jun 24, 2024

CodSpeed Performance Report

Merging #4763 will not alter performance

Comparing ev-br:sync-codspeed (28fb95d) with develop (1ba1b9c)

Summary

✅ 2 untouched benchmarks

🆕 60 new benchmarks
⁉️ 14 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark develop ev-br:sync-codspeed Change
🆕 test_daxpy[100-c] N/A 24.5 µs N/A
🆕 test_daxpy[100-d] N/A 23.6 µs N/A
🆕 test_daxpy[100-s] N/A 23.6 µs N/A
🆕 test_daxpy[100-z] N/A 25.3 µs N/A
🆕 test_daxpy[1000-c] N/A 32.1 µs N/A
🆕 test_daxpy[1000-d] N/A 31.7 µs N/A
🆕 test_daxpy[1000-s] N/A 27.2 µs N/A
🆕 test_daxpy[1000-z] N/A 40 µs N/A
⁉️ test_daxpy[1000] 30.9 µs N/A N/A
⁉️ test_daxpy[100] 25.2 µs N/A N/A
🆕 test_dgbmv[1-100-c] N/A 39.2 µs N/A
🆕 test_dgbmv[1-100-d] N/A 36.7 µs N/A
🆕 test_dgbmv[1-100-s] N/A 36.2 µs N/A
🆕 test_dgbmv[1-100-z] N/A 41 µs N/A
🆕 test_dgbmv[1-1000-c] N/A 98.5 µs N/A
🆕 test_dgbmv[1-1000-d] N/A 82.4 µs N/A
🆕 test_dgbmv[1-1000-s] N/A 73.6 µs N/A
🆕 test_dgbmv[1-1000-z] N/A 117.7 µs N/A
🆕 test_dgemv[100-c] N/A 148.3 µs N/A
🆕 test_dgemv[100-d] N/A 145.7 µs N/A
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@martin-frbg
Copy link
Collaborator

Yes I don't think it makes sense to expand the codspeed benchmarks to the same granularity and complexity as the standalone benchmark suite - apart from (potential direct monetary) cost, having a CI job run for hours can be a major nuisance when waiting to commit something. Small set of sizes and either of the two complex types in addition to the existing double precision real is probably good enough, and we are checking one architecture only anyway

@ev-br
Copy link
Contributor Author

ev-br commented Jun 24, 2024

Okay, down to 35 mins for the benchmarks themselves + some 15 mins build time with
https://github.com/OpenMathLib/OpenBLAS/actions/runs/9646547071/job/26603220881?pr=4763

@martin-frbg martin-frbg added this to the 0.3.28 milestone Jun 30, 2024
@martin-frbg martin-frbg merged commit 4052b31 into OpenMathLib:develop Jun 30, 2024
69 of 76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants