-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automated Themis Core benchmarks #580
Conversation
Let's add some reproducible, verifiable, and automatable benchmarks for Themis performance. We are using Criterion.rs -- a framework in Rust -- but since C is easily callable from Rust, it's not much an issue to use that framework for accurately benchmarking C code. And I don't have to reinvent the wheel and build a benchmarking harness myself as it is usually done in C/C++ world. Currently we cover Secure Cell in Seal mode with master key. More benches will come in later, this is only an example and a primer. We exercise both encryption and decryption code paths. The data does not really matter so we use whatever is easy to get. Message size comparison confirms that encryption and decryption have roughly the same cost and indeed have linear complexity.
Do this for every build, at least to check that they are still buildable. Also run some of them for a modest amount of time and attach benchmark reports to builds. It would be nice to compare base branch with PR branch to see if there are any changes in that. However, with current approach branch switching is kinda involved so we'll add that later.
Oh yeah, one more readme file has been lost during rebases.
Yes, I'm bad at doing git, sorry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice pr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmarks! Benchmarks!
``` | ||
|
||
Don’t forget to _reinstall_ Themis Core library every time you make a change in it and want to measure it. | ||
<!-- TODO: and to pester maintainers to support local builds --> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😬
Co-Authored-By: vixentael <[email protected]>
Exactly what it says on the tin.
Let's add some reproducible, verifiable, and automatable benchmarks for Themis performance. We are using Criterion.rs – a framework in Rust – but since C is easily callable from Rust, it's not much an issue to use that framework for accurately benchmarking C code. And I don't have to reinvent the wheel and build a benchmarking harness myself as it is usually done in C/C++ world.
Currently we cover Secure Cell in Seal mode with master key. More benches will come in later, this is only an example and a primer. We exercise both encryption and decryption code paths. The data does not really matter so we use whatever is easy to get. Message size comparison confirms that encryption and decryption have roughly the same cost and indeed have linear complexity.
Also, run benchmarks for every build, at least to check that they are still buildable. Run some of them for a modest amount of time and attach benchmark reports to builds.
It would be nice to compare base branch with PR branch to see if there are any changes in that. However, with current approach branch switching is kinda involved so we'll add that later.
You are also welcome to run them on your machines and provide feedback on UX.
Example of benchmark report.
Checklist