Simplify benchmark comparison between yew master and changed project

.gitignore

@@ -77,4 +77,8 @@ glimmer-*/tmp
 Cargo.lock
 
 # emacs
-*~
+*~
+
+**/bundled-dist/*.js
+**/bundled-dist/*.wasm
+**/bundled-dist/*.html

README.md

@@ -37,7 +37,9 @@ There are 3 framework entries in this repository:
 * `keyed/yew`
 * `keyed/yew-baseline` _(Used to determine if recent changes improved performance)_
 
-### 1. Prerequisites
+### Prerequisites
+
+#### Install Node
 
 Have *node.js (>=10.0)* installed. We recommend using `nvm` to manage node versions The benchmark has been tested with node v12.16.2.
 Please make sure that the following command work before trying to build:
@@ -51,136 +53,23 @@ v12.16.2
 v12.16.2
 ```
 
-### Install `chromedriver`
+#### Install `chromedriver`
 
 https://chromedriver.chromium.org/downloads
 
 
-### Setup local server
-
-You can use any server which properly handles the MIME type of wasm files. We recommend the `https` crate:
+#### Install local server
 
 ```
 cargo install https
 ```
 
-Start server in the root directory
-```
-http -p 8080
-```
-
-Verify that the server works:
-Try to open [http://localhost:8080/index.html](http://localhost:8080/index.html). If you see something like that you're on the right track:
-![Index.html](images/index.png?raw=true "Index.html")
-
-Now open a new terminal window and keep `http` running in background.
-
-### 2. Building and running a single framework
-
-We now try to build a framework. Go to the `wasm-bindgen` reference implementation
-```
-cd frameworks/keyed/wasm-bindgen
-```
-and install the dependencies
-```
-npm install
-```
-and build the framework
-```
-npm run build-prod
-```
-There should be no build errors and we can open the framework in the browser:
-[http://localhost:8080/frameworks/keyed/wasm-bindgen/index.html](http://localhost:8080/frameworks/keyed/wasm-bindgen/index.html)
-
-## 3. Running a single framework with the automated benchmark driver
-
-The benchmark uses an automated benchmark driver using `chromedriver` to measure the duration for each operation using chrome's timeline. Here are the steps to run for a single framework:
+Make sure port `8080` is not taken.
 
 ### Run benchmarks
 
-First, navigate to the `webdriver-ts/` directory.
-
-and install the dependencies
-```
-npm install
-```
-and build the benchmark driver
-```
-npm run build-prod
-```
-now run the benchmark driver for the "wasmbindgen-keyed" framework:
-```
-npm run bench -- --headless keyed/wasm-bindgen
-```
-Just lean back and wait for chrome to run the benchmarks. 
-Keep an eye on the console output to see if benchmark checks are timing out. If so, visit [http://localhost:8080/frameworks/keyed/wasm-bindgen/index.html](http://localhost:8080/frameworks/keyed/wasm-bindgen/index.html) and check for console errors. You could also try without using `--headless` to debug.
-
-The results for that run will be saved in the `webdriver-ts/results` directory. We can take a look at the results of a single result:
-```
-cat results/wasm-bindgen-v0.2.47-keyed_01_run1k.json
-
-{
-  "framework": "wasm-bindgen-v0.2.47-keyed",
-  "keyed": true,
-  "benchmark": "01_run1k",
-  "type": "cpu",
-  "min": 101.123,
-  "max": 114.547,
-  "mean": 107.69300000000001,
-  "median": 105.821,
-  "geometricMean": 107.5587774247631,
-  "standardDeviation": 6.028027040417119,
-  "values": [
-    114.547,
-    113.509,
-    103.465,
-    105.821,
-    101.123
-  ]
-}
-```
-As you can see the mean duration for create 1000 rows was 107 msecs.
-
-You can also check whether the implementation appears to be compliant to the rules:
-```
-npm run check keyed/wasm-bindgen
-```
-If it finds anything it'll report an ERROR.
-
-## 4. Building the result table
-
-Install libraries
-```
-(cd webdriver-ts-results && npm install)
-```
-
-In the webdriver-ts directory issue the following command:
-```
-npm run results
-```
-Now a result table should have been created which can be opened on [http://localhost:8080/webdriver-ts-results/table.html](http://localhost:8080/webdriver-ts-results/table.html).
-
-## 5. Building and running the benchmarks for all frameworks
-
-From the base directory, you can build all frameworks by running
 ```
-npm install
-npm run build-prod
+./benchmark_change.sh $CHANGED_YEW_PROJECT
 ```
-
-You can now run the benchmark for all frameworks by:
-
-1. Navigate to the `webdriver-ts/` directory
-2. Run benches: `npm run bench -- --headless --count <COUNT>`
-3. Process results: `npm run results`
-
-After that, you can check all results in [http://localhost:8080/webdriver-ts/table.html](http://localhost:8080/webdriver-ts/table.html).
-
-## Tips and tricks
-
-* You can run multiple frameworks by passing their directory names (cd to webdriver-ts):
-`npm run bench keyed/yew-baseline keyed/yew` or if you want to pass more options it becomes: 
-`npm run bench -- --count 3 keyed/yew`.
-* You can run all of the frameworks you've installed using `npm run bench -- --installed`
-* To achieve good precision you should run each framework often enough. I recommend at least 10 times, more is better. The result table contains the mean and the standard deviation. You can seen the effect on the latter pretty well if you increase the count.
-
+where `$CHANGED_YEW_PROJECT` is the path to the root of the yew project you have
+your changes in.

benchmark_change.sh

@@ -0,0 +1,51 @@
+#! /bin/bash
+# Usage: ./bechmark_change.sh CHANGE_BRANCH
+
+http -p 8080 &
+
+(
+	set -euo pipefail
+
+	git checkout HEAD -- frameworks/keyed/yew/Cargo.toml
+	sed -i "s,PATH,$1/packages/yew," frameworks/keyed/yew/Cargo.toml
+
+	npm ci
+
+	(
+		set -euo pipefail
+		cd frameworks/keyed/yew
+		rm -f Cargo.lock
+		npm ci
+		npm run build-prod
+	)
+	(
+		set -euo pipefail
+		cd frameworks/keyed/yew-baseline
+		rm -f Cargo.lock
+		npm ci
+		npm run build-prod
+	)
+	(
+		set -euo pipefail
+		cd webdriver-ts-results
+		npm ci
+	)
+	(
+		set -euo pipefail
+		cd webdriver-ts
+		rm -rf results
+		npm ci
+		npm run build-prod
+		npm run bench -- --headless keyed/yew keyed/yew-baseline
+		npm run results
+	)
+
+	if [[ $OSTYPE == 'darwin'* ]]; then
+		open http://localhost:8080/webdriver-ts-results/table.html &
+	else
+		xdg-open http://localhost:8080/webdriver-ts-results/table.html &
+	fi
+	sleep 1
+)
+
+kill %%