Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Experimental Tango support #25

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/bench.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Benchmarks

permissions:
contents: read

on:
pull_request:

jobs:
bench:
runs-on: ubuntu-22.04
steps:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think several of these could probably be improved upon by drawing from other helpful GitHub actions. For example, installing Rust and installing crate-binaries have pretty good (and more optimized) variants. The other workflow files may be able to provide some inspiration.

Copy link
Contributor Author

@bazhenov bazhenov Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • migrated towards dtolnay/rust-toolchain
  • using lock files when building benchmarks
  • add support of taiki-e/install-action to cargo-export

- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
ref: main
path: baseline-branch

- name: Install Toolchain
uses: dtolnay/rust-toolchain@nightly

- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@v2
with:
tool: cargo-export

- name: cargo generate-lockfile
if: hashFiles('Cargo.lock') == ''
run: cargo generate-lockfile
- name: Building Benchmarks
run: |
cargo export target/benchmarks -- bench --bench=search_ord --locked
# !!!SHOULD BE FIXED AFTER MERGE TO MAIN!!!
# At the moment we can only test branch agains itself, because there is no
# tango benchmarks on the main.
# cd baseline-branch
# cargo export target/benchmarks -- bench --bench=search_ord --locked

- name: Run Benchmarks
run: |
set -eo pipefail

target/benchmarks/search_ord --color=never compare target/benchmarks/search_ord \
-ot 1 --fail-threshold 10 | tee target/benchmark.txt
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is -ot here?
Also, what is the unit for --fail-threshold?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • -t – duration of each test (in seconds)
  • -o – filter outliers. Because tango is measuring difference directly it can detect and remove severe observations that are manifest itself symmetrically (in both algorithms), therefore they are considered independent of the algorithms performance.
  • --fail-threshold – exit with non zero exit code if candidate is slower than baseline on given amount of percents. Tango also do a z-test and fail only if difference is statistically significant (planning to move to bootstrap later on)

More on cli-arguments


- uses: actions/upload-artifact@v3
with:
name: benchmark.txt
path: target/benchmark.txt
Loading
Loading