Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

try to fix benchmarks #4304

Merged
merged 9 commits into from
Mar 18, 2024
Merged

try to fix benchmarks #4304

merged 9 commits into from
Mar 18, 2024

Conversation

JoviDeCroock
Copy link
Member

@JoviDeCroock JoviDeCroock commented Mar 17, 2024 •
edited
Loading

Looks like bumping our Actions causes a few issues in our bench setup. Not sure what's wrong here, I chose to revert the commits to get us back into a working state. Upgrading all actions and adding overwrite: true brings us into a working state but then it fails during benchmark runs, like this

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 17, 2024 •
edited
Loading

📊 Tachometer Benchmark Results

Summary

duration

  • 02_replace1k: unsure 🔍 -2% - +0% (-1.33ms - +0.38ms)
    preact-local vs preact-main
  • 03_update10th1k_x16: slower ❌ 0% - 6% (0.02ms - 1.83ms)
    preact-local vs preact-main
  • 07_create10k: unsure 🔍 -1% - +0% (-9.80ms - +3.91ms)
    preact-local vs preact-main
  • filter_list: unsure 🔍 -0% - +0% (-0.08ms - +0.06ms)
    preact-local vs preact-main
  • hydrate1k: unsure 🔍 -2% - +0% (-1.40ms - +0.07ms)
    preact-local vs preact-main
  • many_updates: unsure 🔍 -2% - +4% (-0.38ms - +0.65ms)
    preact-local vs preact-main
  • text_update: unsure 🔍 -4% - +1% (-0.09ms - +0.02ms)
    preact-local vs preact-main
  • todo: unsure 🔍 -3% - +3% (-0.69ms - +0.68ms)
    preact-local vs preact-main

usedJSHeapSize

  • 02_replace1k: unsure 🔍 -0% - +0% (-0.00ms - +0.00ms)
    preact-local vs preact-main
  • 03_update10th1k_x16: unsure 🔍 -0% - +0% (-0.00ms - +0.01ms)
    preact-local vs preact-main
  • 07_create10k: unsure 🔍 -0% - +0% (-0.00ms - +0.00ms)
    preact-local vs preact-main
  • filter_list: unsure 🔍 -0% - +0% (-0.00ms - +0.01ms)
    preact-local vs preact-main
  • hydrate1k: unsure 🔍 -0% - +0% (-0.01ms - +0.00ms)
    preact-local vs preact-main
  • many_updates: unsure 🔍 -0% - +0% (-0.00ms - +0.00ms)
    preact-local vs preact-main
  • text_update: unsure 🔍 -0% - +0% (-0.00ms - +0.00ms)
    preact-local vs preact-main
  • todo: unsure 🔍 -0% - +0% (-0.00ms - +0.00ms)
    preact-local vs preact-main

Results

02_replace1k

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main81.63ms - 82.79ms-unsure 🔍
-0% - +2%
-0.38ms - +1.33ms		unsure 🔍
-1% - +1%
-0.92ms - +0.60ms
preact-local81.11ms - 82.37msunsure 🔍
-2% - +0%
-1.33ms - +0.38ms		-unsure 🔍
-2% - +0%
-1.43ms - +0.17ms
preact-hooks81.87ms - 82.87msunsure 🔍
-1% - +1%
-0.60ms - +0.92ms		unsure 🔍
-0% - +2%
-0.17ms - +1.43ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main3.40ms - 3.40ms-unsure 🔍
-0% - +0%
-0.00ms - +0.00ms		faster ✔
1% - 1%
0.02ms - 0.03ms
preact-local3.40ms - 3.40msunsure 🔍
-0% - +0%
-0.00ms - +0.00ms		-faster ✔
1% - 1%
0.02ms - 0.03ms
preact-hooks3.43ms - 3.43msslower ❌
1% - 1%
0.02ms - 0.03ms		slower ❌
1% - 1%
0.02ms - 0.03ms		-

run-warmup-0

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main30.40ms - 31.28ms-unsure 🔍
-2% - +2%
-0.65ms - +0.53ms		faster ✔
1% - 5%
0.32ms - 1.52ms
preact-local30.51ms - 31.28msunsure 🔍
-2% - +2%
-0.53ms - +0.65ms		-faster ✔
1% - 4%
0.30ms - 1.42ms
preact-hooks31.36ms - 32.16msslower ❌
1% - 5%
0.32ms - 1.52ms		slower ❌
1% - 5%
0.30ms - 1.42ms		-

run-warmup-1

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main42.14ms - 43.57ms-unsure 🔍
-3% - +1%
-1.46ms - +0.59ms		unsure 🔍
-4% - +0%
-1.76ms - +0.04ms
preact-local42.55ms - 44.03msunsure 🔍
-1% - +3%
-0.59ms - +1.46ms		-unsure 🔍
-3% - +1%
-1.35ms - +0.50ms
preact-hooks43.16ms - 44.26msunsure 🔍
-0% - +4%
-0.04ms - +1.76ms		unsure 🔍
-1% - +3%
-0.50ms - +1.35ms		-

run-warmup-2

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main29.04ms - 29.69ms-unsure 🔍
-2% - +3%
-0.50ms - +0.74ms		unsure 🔍
-2% - +2%
-0.49ms - +0.46ms
preact-local28.72ms - 29.78msunsure 🔍
-3% - +2%
-0.74ms - +0.50ms		-unsure 🔍
-3% - +2%
-0.77ms - +0.50ms
preact-hooks29.03ms - 29.73msunsure 🔍
-2% - +2%
-0.46ms - +0.49ms		unsure 🔍
-2% - +3%
-0.50ms - +0.77ms		-

run-warmup-3

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main26.70ms - 27.72ms-unsure 🔍
-3% - +2%
-0.83ms - +0.60ms		unsure 🔍
-4% - +1%
-1.09ms - +0.42ms
preact-local26.83ms - 27.83msunsure 🔍
-2% - +3%
-0.60ms - +0.83ms		-unsure 🔍
-4% - +2%
-0.97ms - +0.53ms
preact-hooks26.99ms - 28.11msunsure 🔍
-2% - +4%
-0.42ms - +1.09ms		unsure 🔍
-2% - +4%
-0.53ms - +0.97ms		-

run-warmup-4

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main29.95ms - 31.76ms-unsure 🔍
-5% - +3%
-1.70ms - +1.00ms		faster ✔
3% - 9%
0.89ms - 2.91ms
preact-local30.19ms - 32.21msunsure 🔍
-3% - +6%
-1.00ms - +1.70ms		-faster ✔
1% - 8%
0.44ms - 2.66ms
preact-hooks32.29ms - 33.21msslower ❌
3% - 10%
0.89ms - 2.91ms		slower ❌
1% - 9%
0.44ms - 2.66ms		-

run-final

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main23.70ms - 24.35ms-unsure 🔍
-2% - +2%
-0.51ms - +0.48ms		faster ✔
1% - 4%
0.13ms - 1.08ms
preact-local23.67ms - 24.41msunsure 🔍
-2% - +2%
-0.48ms - +0.51ms		-faster ✔
0% - 4%
0.09ms - 1.09ms
preact-hooks24.29ms - 24.97msslower ❌
1% - 5%
0.13ms - 1.08ms		slower ❌
0% - 5%
0.09ms - 1.09ms		-
03_update10th1k_x16

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main28.43ms - 29.61ms-faster ✔
0% - 6%
0.02ms - 1.83ms		faster ✔
1% - 7%
0.25ms - 2.25ms
preact-local29.26ms - 30.64msslower ❌
0% - 6%
0.02ms - 1.83ms		-unsure 🔍
-5% - +2%
-1.39ms - +0.73ms
preact-hooks29.47ms - 31.08msslower ❌
1% - 8%
0.25ms - 2.25ms		unsure 🔍
-2% - +5%
-0.73ms - +1.39ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main3.35ms - 3.36ms-unsure 🔍
-0% - +0%
-0.01ms - +0.00ms		faster ✔
1% - 1%
0.02ms - 0.03ms
preact-local3.36ms - 3.36msunsure 🔍
-0% - +0%
-0.00ms - +0.01ms		-faster ✔
1% - 1%
0.02ms - 0.03ms
preact-hooks3.38ms - 3.39msslower ❌
1% - 1%
0.02ms - 0.03ms		slower ❌
1% - 1%
0.02ms - 0.03ms		-
07_create10k

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main893.25ms - 905.41ms-unsure 🔍
-0% - +1%
-3.91ms - +9.80ms		unsure 🔍
-2% - +1%
-17.97ms - +7.72ms
preact-local893.23ms - 899.54msunsure 🔍
-1% - +0%
-9.80ms - +3.91ms		-unsure 🔍
-2% - +0%
-19.82ms - +3.68ms
preact-hooks893.14ms - 915.77msunsure 🔍
-1% - +2%
-7.72ms - +17.97ms		unsure 🔍
-0% - +2%
-3.68ms - +19.82ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main26.40ms - 26.40ms-unsure 🔍
-0% - +0%
-0.00ms - +0.00ms		unsure 🔍
-0% - -0%
-0.02ms - -0.02ms
preact-local26.40ms - 26.40msunsure 🔍
-0% - +0%
-0.00ms - +0.00ms		-unsure 🔍
-0% - -0%
-0.02ms - -0.02ms
preact-hooks26.42ms - 26.42msunsure 🔍
+0% - +0%
+0.02ms - +0.02ms		unsure 🔍
+0% - +0%
+0.02ms - +0.02ms		-
filter_list

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main16.52ms - 16.65ms-unsure 🔍
-0% - +0%
-0.06ms - +0.08ms		unsure 🔍
-0% - +1%
-0.04ms - +0.09ms
preact-local16.56ms - 16.60msunsure 🔍
-0% - +0%
-0.08ms - +0.06ms		-unsure 🔍
-0% - +0%
-0.01ms - +0.04ms
preact-hooks16.54ms - 16.58msunsure 🔍
-1% - +0%
-0.09ms - +0.04ms		unsure 🔍
-0% - +0%
-0.04ms - +0.01ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main1.42ms - 1.42ms-unsure 🔍
-0% - +0%
-0.01ms - +0.00ms		faster ✔
2% - 2%
0.02ms - 0.03ms
preact-local1.42ms - 1.43msunsure 🔍
-0% - +0%
-0.00ms - +0.01ms		-faster ✔
1% - 2%
0.02ms - 0.03ms
preact-hooks1.45ms - 1.45msslower ❌
2% - 2%
0.02ms - 0.03ms		slower ❌
1% - 2%
0.02ms - 0.03ms		-
hydrate1k

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main79.49ms - 80.67ms-unsure 🔍
-0% - +2%
-0.07ms - +1.40ms		unsure 🔍
-3% - +1%
-2.52ms - +0.85ms
preact-local78.99ms - 79.85msunsure 🔍
-2% - +0%
-1.40ms - +0.07ms		-unsure 🔍
-4% - +0%
-3.13ms - +0.13ms
preact-hooks79.34ms - 82.49msunsure 🔍
-1% - +3%
-0.85ms - +2.52ms		unsure 🔍
-0% - +4%
-0.13ms - +3.13ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main6.10ms - 6.11ms-unsure 🔍
-0% - +0%
-0.00ms - +0.01ms		unsure 🔍
-0% - -0%
-0.02ms - -0.01ms
preact-local6.10ms - 6.11msunsure 🔍
-0% - +0%
-0.01ms - +0.00ms		-unsure 🔍
-0% - -0%
-0.02ms - -0.02ms
preact-hooks6.12ms - 6.13msunsure 🔍
+0% - +0%
+0.01ms - +0.02ms		unsure 🔍
+0% - +0%
+0.02ms - +0.02ms		-
many_updates

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main16.63ms - 16.95ms-unsure 🔍
-4% - +2%
-0.65ms - +0.38ms		unsure 🔍
-1% - +2%
-0.21ms - +0.27ms
preact-local16.44ms - 17.42msunsure 🔍
-2% - +4%
-0.38ms - +0.65ms		-unsure 🔍
-2% - +4%
-0.36ms - +0.69ms
preact-hooks16.59ms - 16.94msunsure 🔍
-2% - +1%
-0.27ms - +0.21ms		unsure 🔍
-4% - +2%
-0.69ms - +0.36ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main4.52ms - 4.52ms-unsure 🔍
-0% - +0%
-0.00ms - +0.00ms		unsure 🔍
-0% - -0%
-0.02ms - -0.02ms
preact-local4.52ms - 4.52msunsure 🔍
-0% - +0%
-0.00ms - +0.00ms		-unsure 🔍
-0% - -0%
-0.02ms - -0.02ms
preact-hooks4.54ms - 4.54msunsure 🔍
+0% - +0%
+0.02ms - +0.02ms		slower ❌
0% - 1%
0.02ms - 0.02ms		-
text_update

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main2.48ms - 2.55ms-unsure 🔍
-1% - +4%
-0.02ms - +0.09ms		faster ✔
4% - 8%
0.10ms - 0.21ms
preact-local2.44ms - 2.52msunsure 🔍
-4% - +1%
-0.09ms - +0.02ms		-faster ✔
5% - 9%
0.13ms - 0.25ms
preact-hooks2.63ms - 2.71msslower ❌
4% - 8%
0.10ms - 0.21ms		slower ❌
5% - 10%
0.13ms - 0.25ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main0.80ms - 0.80ms-unsure 🔍
-0% - +0%
-0.00ms - +0.00ms		faster ✔
3% - 3%
0.02ms - 0.02ms
preact-local0.80ms - 0.80msunsure 🔍
-0% - +0%
-0.00ms - +0.00ms		-faster ✔
3% - 3%
0.02ms - 0.02ms
preact-hooks0.83ms - 0.83msslower ❌
3% - 3%
0.02ms - 0.02ms		slower ❌
3% - 3%
0.02ms - 0.02ms		-
todo

duration

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main26.59ms - 27.62ms-unsure 🔍
-3% - +3%
-0.68ms - +0.69ms		faster ✔
0% - 4%
0.01ms - 1.05ms
preact-local26.65ms - 27.55msunsure 🔍
-3% - +3%
-0.69ms - +0.68ms		-faster ✔
0% - 4%
0.08ms - 1.00ms
preact-hooks27.54ms - 27.73msunsure 🔍
-0% - +4%
+0.01ms - +1.05ms		slower ❌
0% - 4%
0.08ms - 1.00ms		-

usedJSHeapSize

VersionAvg timevs preact-mainvs preact-localvs preact-hooks
preact-main0.88ms - 0.88ms-unsure 🔍
-0% - +0%
-0.00ms - +0.00ms		faster ✔
3% - 3%
0.03ms - 0.03ms
preact-local0.88ms - 0.88msunsure 🔍
-0% - +0%
-0.00ms - +0.00ms		-faster ✔
3% - 3%
0.03ms - 0.03ms
preact-hooks0.90ms - 0.91msslower ❌
3% - 3%
0.03ms - 0.03ms		slower ❌
3% - 3%
0.03ms - 0.03ms		-

tachometer-reporter-action v2 for Benchmarks

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 17, 2024 •
edited
Loading

Size Change: 0 B

Total Size: 60 kB

ℹ️ View Unchanged
Filename Size Change
compat/dist/compat.js 4 kB 0 B
compat/dist/compat.module.js 3.92 kB 0 B
compat/dist/compat.umd.js 4.06 kB 0 B
debug/dist/debug.js 3.52 kB 0 B
debug/dist/debug.module.js 3.52 kB 0 B
debug/dist/debug.umd.js 3.6 kB 0 B
devtools/dist/devtools.js 232 B 0 B
devtools/dist/devtools.module.js 241 B 0 B
devtools/dist/devtools.umd.js 316 B 0 B
dist/preact.js 4.53 kB 0 B
dist/preact.min.js 4.57 kB 0 B
dist/preact.min.module.js 4.56 kB 0 B
dist/preact.min.umd.js 4.6 kB 0 B
dist/preact.module.js 4.55 kB 0 B
dist/preact.umd.js 4.61 kB 0 B
hooks/dist/hooks.js 1.55 kB 0 B
hooks/dist/hooks.module.js 1.59 kB 0 B
hooks/dist/hooks.umd.js 1.63 kB 0 B
jsx-runtime/dist/jsxRuntime.js 963 B 0 B
jsx-runtime/dist/jsxRuntime.module.js 938 B 0 B
jsx-runtime/dist/jsxRuntime.umd.js 1.04 kB 0 B
test-utils/dist/testUtils.js 453 B 0 B
test-utils/dist/testUtils.module.js 454 B 0 B
test-utils/dist/testUtils.umd.js 536 B 0 B

compressed-size-action

@coveralls
Copy link

coveralls commented Mar 17, 2024 •
edited
Loading

Coverage Status

coverage: 99.467%. remained the same
when pulling 171fd19 on fix-benchmarks
into a2c12f5 on main.

@JoviDeCroock JoviDeCroock merged commit b2ff056 into main Mar 18, 2024
2 checks passed
@JoviDeCroock JoviDeCroock deleted the fix-benchmarks branch March 18, 2024 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marvinhagemeister marvinhagemeister marvinhagemeister approved these changes

@andrewiggins andrewiggins Awaiting requested review from andrewiggins

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JoviDeCroock @coveralls @marvinhagemeister