diff --git a/salvo/ENVOY_DEVELOP_WORKFLOW.md b/salvo/ENVOY_DEVELOP_WORKFLOW.md new file mode 100644 index 00000000..a0a28ae1 --- /dev/null +++ b/salvo/ENVOY_DEVELOP_WORKFLOW.md @@ -0,0 +1,233 @@ +# Measure Envoy's Performance Change with an A/B Testing + +## Objective + +This documentation will describe the user journey and detailed workflow to use Salvo to measure +Envoy's performance change with an A/B testing. + +## User Journey + +You are implementing a new feature in Envoy and reach a point where you need to decide between two +possible implementation approaches. You wonder what are the performance implications of each choice +and decide to run the benchmarking framework to find out. You run two executions in A/B testing +mode, both comparing an official Envoy version to an Envoy version with your change using one of +the approaches. Once the tests complete, you examine the results and choose an implementation +approach with the better performance characteristics. + +## Workflow + +This section will describe the whole workflow to use Salvo to Measure Performance Change with an +A/B Testing. And we will use [Binary Benchmark](./README.md#binary-benchmark) as an example. + +1. The requirements to build Envoy and Nighthawk in local machine + + In oder to build Envoy and Nighthawk in local machine, + follow [this documentation](https://www.envoyproxy.io/docs/envoy/latest/start/building.html#linux-mac-target-requirements) + to ensure local machine meets the requirements. + +2. Download Salvo + + The next step is to download Salvo project from github, you can use `git clone` command: + + ```bash + git clone https://github.com/envoyproxy/envoy-perf.git + cd envoy-perf/salvo + ``` + +3. Install dependencies + + Before using Salvo, you need to install its dependencies. The + [install_deps.sh](./install_deps.sh) + script can be used to install any dependencies required by Salvo. + + ```bash + sudo ./install_deps.sh + ``` + + You can install fortio CLI command to gain the visualized benchmarking report, the installation + method: https://github.com/fortio/fortio#installation + +4. Build Salvo + + The next step is to build Salvo, Salvo can be built by + [Bazel](https://docs.bazel.build/versions/main/install-ubuntu.html): + + ```bash + bazel build //... + ``` + +5. Set up commits for evaluation + + This step will introduce the workflow of setting up commits for evaluation if you haven't + committed your code changes. If you have set up commits somewhere Salvo can fetch, plase ingore + this step. + + 5.1. Create a local fork of Envoy repository. We will take the forked repo: + https://github.com/gyohuangxin/envoy as the example. + + 5.2. Clone the forked repo and create a new branch: + + ```bash + git clone https://github.com/gyohuangxin/envoy + cd envoy && git checkout -b benchmark_testing + ``` + + 5.3. Set up commits for your two approaches: + + ```bash + # Apply code changes for approach one. + git add . && git commit -m "This is approach one." + + # Apply code changes for approach two. + git add . && git commit -m "This is approach two." + ``` + + 5.4. Push the two commits to the remote forked repo: + + ```bash + git push origin benchmark_testing + ``` + +6. Binary Benchmark Configuration + + Currently, you have committed two Envoy code changes to the forked repo: + "https://github.com/gyohuangxin/envoy" and the commit hashes are + "c2e1c0d0da61f3fec25f48f8bd63e4a76db8a2a9" and "e95717bcd0672910826de4a9bb2f402f6e057b2c". + **These commits are fake and just for the sake of examples.** + + [TODO]: <> (These commits are fake now, we should convert them to real commits) + + Then you want to measure their performance change to Envoy v1.21.0 as the baseline and choose + the commit which has the better performance. You can compose the configuration of Salvo in YAML. + For example, you can create an configuration YAML file named `binary_benchmark.yaml` and store + it in anywhere Salvo can read from. + + ```yaml + remote: false + binaryBenchmark: true + environment: + outputDir: /home/ubuntu/nighthawk_output + testVersion: IPV_V4ONLY + images: + nighthawkBenchmarkImage: envoyproxy/nighthawk-benchmark-dev:latest + source: + - identity: SRCID_ENVOY + commit_hash: v1.21.0 + additional_hashes: [c2e1c0d0da61f3fec25f48f8bd63e4a76db8a2a9, e95717bcd0672910826de4a9bb2f402f6e057b2c] + source_url: https://github.com/gyohuangxin/envoy.git + bazelOptions: + - parameter: --jobs 16 + - parameter: --define tcmalloc=gperftools + - identity: SRCID_NIGHTHAWK + source_url: https://github.com/envoyproxy/nighthawk.git + bazelOptions: + - parameter: --jobs 16 + - parameter: --define tcmalloc=gperftools + ``` + + The `environment.outputDir` is directory where benchmark results placed. And the + `environment.testVersion` specify the ip address family to use, choose from "IPV_V4ONLY", + "IPV_V6ONLY" and "ALL". + + The `images.nighthawkBenchmarkImage` specifies the name of the docker image containing the + benchmark framework and default test cases. If you want to provide your own test cases, you + can create a directory and put test files like this + [one](https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/test/test_discovery.py) + into this directory, then add the directory path to `environment.testDir` field. The next + section will introduce the details of test cases. + + Salvo will search in the repo from `source.source_url` for each hash in the + `source.additional_hashes` list, and choose the baseline version v1.21.0 from + `source.commit_hash`. `source.commit_hash` field can be filled in a commit hash or a commit + tag, which Salvo can judge automatically and search for. + + Then Salvo will build each Envoy commit and the baseline version. You can add bazel options + in `source.bazelOptions`, and they will be added as parameters to bazel command when building + Envoy. + + Finally you can also determine which Nighthawk version to use like Envoy's configuration, in + this section we will build the latest Nighthawk version. You can find more details about + configuration fields from [APIs definition](./api). + +7. Details of test cases + + [TODO]: <> (add the link of test_architecure.md here to introduce the architecure of Salvo) + + The Benchmark Framework is developed via PyTest Framework. Therefore, the test cases file is + defined as a python file contains the test cases. The + [default test cases file](https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/test/test_discovery.py) + provides an example to configure Nighthawk client/server, outputs and test cases. + + There are two main parts in the test case file. + + - `_run_benchmark` function: + + At first, it defined a function named [`_run_benchmark`] + (https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/test/test_discovery.py#L20) + to run the specific PyTest fixture, which will define the behavior of Envoy to be tested and + [Nighthawk test server](https://github.com/envoyproxy/nighthawk/blob/main/source/server/README.md), + you can find fixture definitions from these two files: + + https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/envoy_proxy.py + + https://github.com/envoyproxy/nighthawk/blob/main/test/integration/integration_test_fixtures.py + + Then, the `_run_benchmark` function provides some configurable parameters used by Nighthawk + client to send test requests, such as `rps`, `duration`, `request_body_size` and so on. + More details about Nighthawk's arguments: + https://github.com/envoyproxy/nighthawk/blob/main/api/client/options.proto + + By default, Nighthawk will use closed-loop mode, because closed-loop testing can be useful to + get a sense of the maximum requests per second that can be squeezed out of a system + (irrespective of the high latency incurred), as well as for comparing numbers with other load + generators that use this methodology. More details about closed-loop mode: + https://github.com/envoyproxy/nighthawk/blob/main/docs/root/terminology.md#closed-loop + + Finally, `_run_benchmark` function defines the format of result outputs after the benchmark + is done. + + - Test cases functions: + + After `_run_benchmark` function defined, you can write the specific test cases to use it. + Take [`test_http_h1_small_request_small_reply_via`](https://github.com/envoyproxy/nighthawk/blob/8d6947b2d4b41edf0ea95a811d129c7a81629b86/benchmarks/test/test_discovery.py#L87) as an + example, it will setup a Nighthawk client as the frontend, the Envoy Proxy to be tested as + the load balancer, and a Nighthawk test server as the backend. The configurations of Envoy + and test server can be assigned with Python decorators. Then you can invoke `_run_benchmark` + function with specific fixture and parameters. + + More test cases examples: + https://github.com/envoyproxy/nighthawk/blob/8d6947b2d4b41edf0ea95a811d129c7a81629b86/benchmarks/test/test_discovery.py#L92-#L119 + +8. Run Salvo + + To run Salvo, use the following command: + + ```bash + bazel-bin/salvo --job /binary_benchmark.yaml + ``` + +9. The Benchmark Result + + Salvo creates a symlink in the local directory to the location of the output artifacts for each + Envoy version tested. + + ``` + ~ envoy-perf/salvo# ls -l |grep source_url + + lrwxrwxrwx 1 root root 80 Feb 7 02:17 source_url__c2e1c0d0da61f3fec25f48f8bd63e4a76db8a2a9 -> /home/ubuntu/nighthawk_output/source_url__c2e1c0d0da61f3fec25f48f8bd63e4a76db8a2a9 + lrwxrwxrwx 1 root root 80 Feb 7 02:17 source_url__e95717bcd0672910826de4a9bb2f402f6e057b2c -> /home/ubuntu/nighthawk_output/source_url__e95717bcd0672910826de4a9bb2f402f6e057b2c + lrwxrwxrwx 1 root root 47 Feb 2 16:32 source_url__v1.21.0 -> /home/ubuntu/nighthawk_output/source_url__v1.21.0 + ``` + + In these directories, you can find benchmark results of different test cases, and the + `nighthawk-human.txt` file provides the human-readable benchmark results from Nighthawk. + + The benchmark results can be visualized via + [Fortio's report-only UI](https://github.com/fortio/fortio#report-only-ui). + The following is an example. + + ``` + fortio report --data-dir /home/ubuntu/nighthawk_output/source_url__c2e1c0d0da61f3fec25f48f8bd63e4a76db8a2a9/2022-02-21-05-16-54/http_h1_small_request_small_reply_via_IpVersion.IPV4-nighthawk_test_integration_configurations_nighthawk_http_origin.yaml-nighthawk_benchmarks_configurations_envoy_proxy.yaml + ``` + + ![fortio_report](./images/fortio_report.png) \ No newline at end of file diff --git a/salvo/README.md b/salvo/README.md index c8122e63..919d3b4a 100644 --- a/salvo/README.md +++ b/salvo/README.md @@ -6,7 +6,11 @@ This is a framework that abstracts executing multiple benchmarks of the Envoy Pr ## Goals of Salvo -Salvo allows Envoy developers to perform A/B testing to monitor performance change of Envoy. Salvo provides the local excution mode allowing developers to run a benchark on their own machine and also provides the remote excution mode to run a benchmark on a remote machine, such as a remote CI system. +Salvo allows Envoy developers to perform A/B testing to monitor performance change of Envoy. Salvo provides the local execution mode allowing developers to run a benchmark on their own machine and also provides the remote execution mode to run a benchmark on a remote machine, such as a remote CI system. + +## About this section + +This section will give a brief overview of Salvo's building, usage and results. If you want to dive into using Salvo to measure Envoy's performance change with an A/B testing, please read the detailed workflow documentation: [ENVOY_DEVELOP_WORKFLOW.md](./ENVOY_DEVELOP_WORKFLOW.md) ## Dependencies @@ -22,7 +26,7 @@ bazel build //... ## Benchmark Test Cases for Salvo -Benchmark test cases for Salvo are defined as Python files with test cases written in pytest framework, here is an example: https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/test/test_discovery.py. Users can provide thier own Python files of test cases into Salvo. +Benchmark test cases for Salvo are defined as Python files with test cases written in pytest framework, here is an example: https://github.com/envoyproxy/nighthawk/blob/main/benchmarks/test/test_discovery.py. Users can provide their own Python files of test cases into Salvo. ## Control Documents @@ -72,7 +76,7 @@ images: envoyImage: "envoyproxy/envoy:v1.21.0" ``` -`remote`: Whether to enable remote excution mode. +`remote`: Whether to enable remote execution mode. `dockerizedBenchmark`: It will run fully dockerized benchmarks. @@ -84,7 +88,7 @@ images: `environment.envoyPath`: Envoy is called 'Envoy' in the Envoy Docker image. -`images.reuseNhImages`: Whether to reuse Nighthawk image if it exsists on the machine. +`images.reuseNhImages`: Whether to reuse Nighthawk image if it exists on the machine. `images.nighthawkBenchmarkImage`: The image of Nighthawk benchmarking tests. diff --git a/salvo/images/fortio_report.png b/salvo/images/fortio_report.png new file mode 100644 index 00000000..1b2e7238 Binary files /dev/null and b/salvo/images/fortio_report.png differ