ci: add benchmark job to workflow

# Motivation

To avoid performance regressions, this commit adds
a benchmark job to the CI workflow.

The idea is that if a given PR decreases the
performance sufficiently, (e.g. 150% or more), the
CI will fail and the PR cannot be merged.

The benchmark results should be visible at
https://nlnwa.github.io/warchaeology/benchmarks/ubuntu-22.04
after this commit has been merged.

# Changes

Added a publish benchmark job in the
`publish_benchmarks.yml` that also runs the
benchmarks and publishes them to the github pages
branch (currently `gh-pages`).

# Quirks

We need to re-run the benchmark results since
getting artifacts from a separate workflow is not
trivial. Ideally we should just run the benchmarks
once.

Publishing benchmarks must live in a separate
workflow, since the permissions it needs to write
to `gh-pages` should not be exposed to external
users creating PRs targeting this repository.

An alternative would be to not run workflows when
receiving external PRs, but that would make the
developer experience for contributors worse.

# Future work

After this PR has been merged, doing the actual
comparison should be a priority. Currently the
benchmarks are only published to the `gh-pages`
branch, and not actually compared to anything.

Add `Windows` to the benchmark matrix.

.github/workflows/main.yml

@@ -35,6 +35,40 @@ jobs:
       - name: Test
         run: go test ./...
 
+  benchmark:
+    name: Benchmark
+    strategy:
+      matrix:
+        os: [ubuntu-22.04]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v4
+        with:
+          go-version: "1.21"
+      - name: Run benchmark
+        run: go test ./... -bench="Bench" | tee benchmark-result.txt
+      - uses: actions/upload-artifact@v4
+        with:
+          name: ${{ matrix.os }}-benchmark-results
+          path: benchmark-result.txt
+      # TODO: immediately after merging this, the following code should be
+      # replaced by downloading the previous benchmark results instead
+      - name: make dummy dir
+        run: mkdir --parents ./previous/benchmarks/${{ matrix.os }}/
+      - name: create dummy file
+        run: echo ' ' > ./previous/benchmarks/${{ matrix.os }}/results.json
+      - name: Compare benchmark results
+        uses: benchmark-action/github-action-benchmark@v1
+        with:
+          name: Benchmarks
+          tool: 'go'
+          output-file-path: benchmark-result.txt
+          external-data-json-path: ./previous/benchmarks/${{ matrix.os }}/results.json
+          auto-push: false
+          fail-on-alert: true
+          alert-threshold: "150%"
+
   build:
     runs-on: ubuntu-latest
     name: Build

.github/workflows/publish_benchmarks.yml

@@ -0,0 +1,41 @@
+# Do not run this workflow on pull request since this workflow has permission to modify contents.
+name: Publish benchmarks
+
+on:
+  push:
+    branches:
+      - main
+
+permissions:
+  # deployments permission to deploy GitHub pages website
+  deployments: write
+  # contents permission to update benchmark contents in gh-pages branch
+  contents: write
+
+jobs:
+  publish-benchmarks:
+    name: Publish benchmark results
+    strategy:
+      matrix:
+        os: [ubuntu-22.04]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Code Checkout
+        uses: actions/checkout@v4
+      - name: Setup Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: "1.21"
+      - name: Run benchmarks
+        run: go test ./... -bench="Bench" | tee benchmark-result.txt
+      - name: Publish benchmark results
+        uses: benchmark-action/github-action-benchmark@v1
+        with:
+          name: Go Benchmark
+          tool: 'go'
+          output-file-path: benchmark-result.txt
+          benchmark-data-dir-path: benchmarks/${{ matrix.os }}
+          auto-push: true
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          fail-on-alert: true
+          alert-threshold: "150%"