Automated Themis Core benchmarks

.circleci/config.yml

@@ -77,6 +77,66 @@ jobs:
       - run: make clean && make test CC=clang-8 WITH_TSAN=1
       - run: make clean && make test CC=clang-8 WITH_UBSAN=1
 
+  benchmark:
+    docker:
+      - image: cossacklabs/build:ubuntu-bionic
+    environment:
+      WITH_FATAL_WARNINGS: yes
+    steps:
+      - run:
+          name: Install native dependencies
+          command: |
+            sudo apt update
+            sudo apt install --yes \
+              gnuplot zip
+      - restore_cache:
+          keys:
+            - rust
+      - run:
+          name: Install Rust toolchain (stable)
+          command: |
+            # Instructions from https://rustup.rs
+            curl https://sh.rustup.rs -sSf | sh -s -- -y
+            cat ~/.cargo/env >> $BASH_ENV
+            source ~/.cargo/env
+            cargo --version
+            rustc --version
+      - checkout
+      - run:
+          name: Pull BoringSSL submodule
+          command: |
+            git reset --hard HEAD
+            git submodule sync
+            git submodule update --init
+      - run:
+          name: Themis Core - install
+          command: |
+            make
+            sudo make install
+      - run:
+          name: Themis Core - prepare benchmarks
+          command: |
+            cd benches/themis
+            cargo bench --no-run
+      # TODO: if building a pull request, compare base with updates
+      - run:
+          name: Themis Core - run benchmarks - Secure Cell (master key)
+          command: |
+            cd benches/themis
+            cargo bench -- 'Secure Cell .* master key/4 KB'
+      - run:
+          name: Pack benchmark report
+          command: |
+            cd benches/themis/target
+            zip -r ../report.zip criterion
+      - store_artifacts:
+          path: benches/themis/report.zip
+      - save_cache:
+          key: rust
+          paths:
+            - ~/.cargo
+            - ~/.rustup
+
   x86_64:
     docker:
       - image: cossacklabs/android-build:2019.01
@@ -466,6 +526,7 @@ workflows:
   tests:
     jobs:
       - analyze
+      - benchmark
       - android
       - x86_64
       - jsthemis
@@ -486,6 +547,7 @@ workflows:
                 - stable
     jobs:
       - analyze
+      - benchmark
       - android
       - x86_64
       - jsthemis

CHANGELOG.md

@@ -66,6 +66,10 @@ _Code:_
 
   - New class `SymmetricKey` can be used to generate symmetric keys for Secure Cell ([#561](https://github.com/cossacklabs/themis/pull/561)).
 
+_Infrastructure:_
+
+- Automated benchmarking harness is now tracking Themis performance. See [`benches`](https://github.com/cossacklabs/themis/tree/master/benches/) ([#580](https://github.com/cossacklabs/themis/pull/580)).
+
 ## [0.12.0](https://github.com/cossacklabs/themis/releases/tag/0.12.0), September 27th 2019
 
 **TL;DR:**

Cargo.toml

@@ -1,5 +1,6 @@
 [workspace]
 members = [
+    "benches/themis",
     "src/wrappers/themis/rust",
     "src/wrappers/themis/rust/libthemis-sys",
     "src/wrappers/themis/rust/libthemis-src",

benches/README.md

@@ -0,0 +1,11 @@
+# ⏱ Themis benchmarks
+
+Here you can find benchmarks used to measure overhead and performance of Themis and its wrappers.
+
+## 👾 Themis Core
+
+These are the main benchmarks for Themis.
+They establish a baseline for perfomance:
+high-level language wrappers generally add overhead to Core.
+
+See [`themis`](themis) directory for details.

benches/themis/.gitignore

@@ -0,0 +1,2 @@
+# Criterion output
+target/

benches/themis/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+name = "themis-core-bench"
+version = "0.0.0"
+edition = "2018"
+publish = false
+
+[dependencies]
+themis = { version = "0.12", path = "../../src/wrappers/themis/rust" }
+libthemis-sys = { version = "0.12", path = "../../src/wrappers/themis/rust/libthemis-sys" }
+
+[dev-dependencies]
+criterion = "0.3"
+
+[[bench]]
+name = "secure_cell_seal_master_key"
+harness = false

benches/themis/README.md

@@ -0,0 +1,134 @@
+# ⏱ Themis Core benchmarks
+
+## Quickstart
+
+Themis Core benchmarks are written using [**Criterion.rs**](https://bheisler.github.io/criterion.rs/book/criterion_rs.html) statistical benchmarking tool.
+
+You will need Rust toolchain installed to run benchmarks.
+[Visit rustup.rs](https://rustup.rs/) to install Rust.
+
+ 1. 📦 **Install Themis Core**
+
+    Benchmarks use Themis library from the system by default.
+    <!-- TODO: use local builds if available -->
+
+    Normally, this should be enough to install Themis:
+
+    ```bash
+    make install
+    ```
+
+    If it doesn’t work (or this is your first time building Themis)
+    then you might need to [review the documentation](https://docs.cossacklabs.com/pages/documentation-themis/#building-and-installing).
+
+    If it still doesn’t work, please [file an issue](https://github.com/cossacklabs/themis/issues/new?labels=bug,installation,core&template=bug_report.md&title=).
+
+ 2. ⚙️ **Change directory from repository root**
+
+    ```bash
+    cd benches/themis
+    ```
+
+    It’s not required but you would have to type less.
+
+ 3. ⏳ **Build dependencies**
+
+    ```bash
+    cargo bench --no-run
+    ```
+
+    Criterion.rs has quite a few dependencies so be patient,
+    you need to do this only once.
+
+ 4. 🚀 **Run some benchmarks**
+
+    ```bash
+    cargo bench -- "Secure Cell .* Seal, master key/64 KB"
+    ```
+
+    [See FAQ](#faq) for more information on how and what you can run.
+
+ 5. 📊 **Analyze result report**
+
+    ```bash
+    open target/criterion/report/index.html
+    ```
+
+    Done! 🎉
+
+## Coverage
+
+### Secure Cell
+
+|                   | Master keys   | Passphrases   |
+| ----------------- | ------------- | ------------- |
+| Seal              | ✅ complete   | 🛠 WIP        |
+| Token Protect     | 💭 soon       | 🛠 WIP        |
+| Context Imprint   | 💭 soon       | ➖ N/A        |
+
+### Secure Message
+
+|                   | ECDSA         | RSA           |
+| ----------------- | ------------- | ------------- |
+| Encrypt / Decrypt | 💭 soon       | 💭 soon       |
+| Sign / Verify     | 💭 soon       | 💭 soon       |
+
+### Secure Session
+
+💭 soon
+
+### Secure Comparator
+
+💭 soon
+
+<!--
+
+## Benchmark results
+
+TODO: describe current benchmark results here
+
+-->
+
+## FAQ
+
+First of all, it’s a good idea to familiarize yourself with
+[Criterion.rs User Guide](https://bheisler.github.io/criterion.rs/book/criterion_rs.html),
+especially sections on
+[command-line options](https://bheisler.github.io/criterion.rs/book/user_guide/command_line_options.html),
+[output format](https://bheisler.github.io/criterion.rs/book/user_guide/command_line_output.html),
+and [interpreting results](https://bheisler.github.io/criterion.rs/book/analysis.html).
+
+**Q:** What benchmarks are available?
+
+```bash
+cargo bench -- --list
+```
+
+**Q:** How do I run one of them?
+
+```bash
+cargo bench -- 'one of them'    # filter by regular expression
+```
+
+**Q:** How do I see if my optimizations have an effect?
+
+```bash
+git checkout feature
+cargo bench -- --save-baseline feature-unoptimized
+
+git checkout optimizations
+# Work on performance
+# ...
+
+# Compare against the baseline version
+cargo bench -- --baseline feature-unoptimized
+```
+
+Don’t forget to _reinstall_ Themis Core library every time you make a change in it and want to measure it.
+<!-- TODO: and to pester maintainers to support local builds -->
+
+**Q:** Benchmarking takes ages, what can I do?
+
+```bash
+cargo bench -- --sample-size 20     # cannot be lower than 10
+```