Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate the new benchmarks repo and update #4310

Merged
merged 21 commits into from
Mar 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ on:
branches:
- '**'
paths:
- 'src/**.js'
- '**/src/**.js'
push:
branches:
- main
- restructure
- v11
paths:
- 'src/**.js'
- '**/src/**.js'

jobs:
build_test:
Expand Down Expand Up @@ -53,57 +53,57 @@ jobs:
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: todo
benchmark: todo/todo
timeout: 10

bench_text_update:
name: Bench text_update
name: Bench text-update
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: text_update
benchmark: text-update/text-update
timeout: 10

bench_many_updates:
name: Bench many_updates
name: Bench many-updates
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: many_updates
benchmark: many-updates/many-updates
timeout: 10

bench_02_replace1k:
name: Bench 02_replace1k
bench_replace1k:
name: Bench replace1k
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: 02_replace1k
benchmark: table-app/replace1k

bench_03_update10th1k_x16:
bench_update10th1k:
name: Bench 03_update10th1k_x16
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: 03_update10th1k_x16
benchmark: table-app/update10th1k

bench_07_create10k:
name: Bench 07_create10k
bench_create10k:
name: Bench create10k
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: 07_create10k
benchmark: table-app/create10k

bench_hydrate1k:
name: Bench hydrate1k
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: hydrate1k
benchmark: table-app/hydrate1k

bench_filter_list:
name: Bench filter_list
name: Bench filter-list
uses: ./.github/workflows/run-bench.yml
needs: prepare
with:
benchmark: filter_list
benchmark: filter-list/filter-list
timeout: 10
80 changes: 67 additions & 13 deletions .github/workflows/run-bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,36 +32,90 @@ jobs:
timeout-minutes: ${{ inputs.timeout }}
steps:
- uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: actions/setup-node@v4
with:
node-version-file: 'package.json'
cache: 'npm'
cache-dependency-path: '**/package-lock.json'

# Setup pnpm
- name: Install pnpm
uses: pnpm/action-setup@v3
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_OUTPUT
- uses: actions/cache@v4
name: Setup pnpm cache
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-

# Install benchmark dependencies
- uses: actions/download-artifact@v3
with:
name: bench-environment
- name: Move tarballs from env to correct location
run: |
ls -al
mv preact-local.tgz benchmarks/dependencies/preact/local-pinned/preact-local-pinned.tgz
ls -al benchmarks/dependencies/preact/local-pinned
mv preact-main.tgz benchmarks/dependencies/preact/main/preact-main.tgz
ls -al benchmarks/dependencies/preact/main
- name: Install deps
working-directory: benches
run: npm ci
working-directory: benchmarks
# Set the CHROMEDRIVER_FILEPATH so the chromedriver npm package uses the
# correct binary when its installed
run: |
export CHROMEDRIVER_FILEPATH=$(which chromedriver)
pnpm install
# Install local dependencies with --no-frozen-lockfile to ensure local tarballs
# are installed regardless of if they match the integrity hash stored in the lockfile
pnpm install --no-frozen-lockfile --filter ./dependencies

# Run benchmark
- name: Run benchmark
working-directory: benches
working-directory: benchmarks
run: |
export CHROMEDRIVER_FILEPATH=$(which chromedriver)
npm run bench ${{ inputs.benchmark }}.html -- --trace=${{ inputs.trace }}
pnpm run bench apps/${{ inputs.benchmark }}.html -d preact@local-pinned -d preact@main --trace=${{ inputs.trace }}

# Prepare output
- name: Anaylze logs if present
working-directory: benchmarks
run: '[ -d out/logs ] && pnpm run analyze ${{ inputs.benchmark }} || echo "No logs to analyze"'
- name: Tar logs if present
working-directory: benchmarks
run: |
if [ -d out/logs ]; then
LOGS_FILE=out/${{ inputs.benchmark }}_logs.tgz
mkdir -p $(dirname $LOGS_FILE)
tar -zcvf $LOGS_FILE out/logs
else
echo "No logs found"
fi

# Upload results and logs
- name: Calculate log artifact name
id: log-artifact-name
run: |
NAME=$(echo "${{ inputs.benchmark }}" | sed -r 's/[\/]+/_/g')
echo "artifact_name=logs_$NAME" >> $GITHUB_OUTPUT
- name: Upload results
uses: actions/upload-artifact@v3
with:
name: results
path: benches/results/${{ inputs.benchmark }}.json
- name: Anaylze logs if present
working-directory: benches
run: '[ -d logs ] && npm run analyze ${{ inputs.benchmark }} || echo "No logs to analyze"'
- name: Tar logs if present
working-directory: benches
run: '[ -d logs ] && tar -zcvf ${{ inputs.benchmark}}_logs.tgz logs || echo "No logs found"'
path: benchmarks/out/results/${{ inputs.benchmark }}.json
- name: Upload logs
uses: actions/upload-artifact@v3
with:
name: ${{ inputs.benchmark}}_logs
path: benches/${{ inputs.benchmark}}_logs.tgz
name: ${{ steps.log-artifact-name.outputs.artifact_name }}
path: benchmarks/out/${{ inputs.benchmark }}_logs.tgz
if-no-files-found: ignore
16 changes: 8 additions & 8 deletions .github/workflows/single-bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ on:
description: 'Which benchmark to run'
type: choice
options:
- 02_replace1k
- 03_update10th1k_x16
- 07_create10k
- filter_list
- hydrate1k
- many_updates
- text_update
- todo
- table-app/replace1k
- table-app/update10th1k
- table-app/create10k
- table-app/hydrate1k
- filter_list/filter-list
- many-updates/many-updates
- text-update/text-update
- todo/todo
required: true
base:
description: 'The branch name, tag, or commit sha of the version of preact to benchmark against.'
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "benchmarks"]
path = benchmarks
url = https://github.com/preactjs/benchmarks
24 changes: 24 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,30 @@ it.only('should test something', () => {
- Check the JSDoc block right above the function definition to understand what it does. It contains a short description of each function argument and what it does.
- Check the callsites of a function to understand how it's used. Modern editors/IDEs allow you to quickly find those, or use the plain old search feature instead.

## Benchmarks

We have a benchmark suite that we use to measure the performance of Preact. Our benchmark suite lives in our [preactjs/benchmarks repository](https://github.com/preactjs/benchmarks), but is included here as Git submodule. To run the benchmarks, first ensure [PNPM](https://pnpm.io/installation) is installed on your system and initialize and setup the submodule (it uses `pnpm` as a package manager):

```bash
pnpm -v # Make sure pnpm is installed
git submodule update --init --recursive
cd benchmarks
pnpm i
```

Then you can run the benchmarks:

```bash
# In the benchmarks folder
pnpm run bench
```

Checkout the README in the benchmarks folder for more information on running benchmarks.

> **Note:** When switching branches, git submodules are not automatically updated to the commit of the new branch - it stays at the commit of the previous branch. This can be a feature! It allows you to work in different branches with the latest versions of the benchmarks - especially if you have made changes to the benchmarks.
>
> However if you want to switch branches and also update the benchmarks to the latest commit of the new branch, you can run `git submodule update --recursive` after switching branches, or run `git checkout --recurse-submodules` when checking out a new branch.

## FAQ

### Why does the JSDoc use TypeScript syntax to specify types?
Expand Down
1 change: 1 addition & 0 deletions benchmarks
Submodule benchmarks added at 0e3f96
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,8 @@
"test/fixtures",
"test/ts/",
"*.ts",
"dist"
"dist",
"benchmarks"
],
"prettier": {
"singleQuote": true,
Expand Down
Loading