Vitest switch by JoviDeCroock · Pull Request #4687 · preactjs/preact

JoviDeCroock · 2025-02-18T06:45:41Z

Requirements before merge

Follow-ups

Get rid of sinon
Get rid of sinon-chai

github-actions · 2025-02-18T06:46:04Z

📊 Tachometer Benchmark Results

Summary

duration

create10k: unsure 🔍 -0% - +0% (-4.02ms - +4.10ms)
^{preact-local vs preact-main}
filter-list: unsure 🔍 -0% - +0% (-0.04ms - +0.03ms)
^{preact-local vs preact-main}
hydrate1k: unsure 🔍 -2% - +2% (-1.42ms - +1.31ms)
^{preact-local vs preact-main}
many-updates: unsure 🔍 -1% - +0% (-0.09ms - +0.05ms)
^{preact-local vs preact-main}
replace1k: unsure 🔍 -2% - +1% (-1.26ms - +0.77ms)
^{preact-local vs preact-main}
text-update: unsure 🔍 -1% - +5% (-0.03ms - +0.11ms)
^{preact-local vs preact-main}
todo: unsure 🔍 -1% - +1% (-0.39ms - +0.39ms)
^{preact-local vs preact-main}
update10th1k: unsure 🔍 -1% - +5% (-0.15ms - +1.42ms)
^{preact-local vs preact-main}

usedJSHeapSize

create10k: unsure 🔍 -0% - +0% (-0.01ms - +0.02ms)
^{preact-local vs preact-main}
filter-list: unsure 🔍 -0% - +4% (-0.00ms - +0.05ms)
^{preact-local vs preact-main}
hydrate1k: unsure 🔍 -1% - +7% (-0.10ms - +0.69ms)
^{preact-local vs preact-main}
many-updates: unsure 🔍 -1% - +1% (-0.02ms - +0.02ms)
^{preact-local vs preact-main}
replace1k: unsure 🔍 -1% - +0% (-0.02ms - +0.01ms)
^{preact-local vs preact-main}
text-update: unsure 🔍 +0% - +0% (+0.00ms - +0.00ms)
^{preact-local vs preact-main}
todo: unsure 🔍 -1% - +3% (-0.01ms - +0.04ms)
^{preact-local vs preact-main}
update10th1k: unsure 🔍 -0% - +0% (-0.01ms - +0.01ms)
^{preact-local vs preact-main}

Results

create10k

Browser: chrome-headless
Sample size: 50
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	922.50ms - 928.80ms	-	unsure 🔍 -0% - +0% -4.02ms - +4.10ms
preact-main	923.05ms - 928.17ms	unsure 🔍 -0% - +0% -4.10ms - +4.02ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	19.18ms - 19.20ms	-	unsure 🔍 -0% - +0% -0.01ms - +0.02ms
preact-main	19.18ms - 19.19ms	unsure 🔍 -0% - +0% -0.02ms - +0.01ms	-

filter-list

Browser: chrome-headless
Sample size: 50
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	16.55ms - 16.61ms	-	unsure 🔍 -0% - +0% -0.04ms - +0.03ms
preact-main	16.56ms - 16.60ms	unsure 🔍 -0% - +0% -0.03ms - +0.04ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	1.56ms - 1.61ms	-	unsure 🔍 -0% - +4% -0.00ms - +0.05ms
preact-main	1.54ms - 1.58ms	unsure 🔍 -3% - +0% -0.05ms - +0.00ms	-

hydrate1k

Browser: chrome-headless
Sample size: 90
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	78.15ms - 80.56ms	-	unsure 🔍 -2% - +2% -1.42ms - +1.31ms
preact-main	78.78ms - 80.05ms	unsure 🔍 -2% - +2% -1.31ms - +1.42ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	9.54ms - 10.11ms	-	unsure 🔍 -1% - +7% -0.10ms - +0.69ms
preact-main	9.26ms - 9.80ms	unsure 🔍 -7% - +1% -0.69ms - +0.10ms	-

many-updates

Browser: chrome-headless
Sample size: 50
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	16.47ms - 16.60ms	-	unsure 🔍 -1% - +0% -0.09ms - +0.05ms
preact-main	16.52ms - 16.59ms	unsure 🔍 -0% - +1% -0.05ms - +0.09ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	3.77ms - 3.79ms	-	unsure 🔍 -1% - +1% -0.02ms - +0.02ms
preact-main	3.77ms - 3.79ms	unsure 🔍 -1% - +1% -0.02ms - +0.02ms	-

replace1k

Browser: chrome-headless
Sample size: 100
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	65.14ms - 66.23ms	-	unsure 🔍 -2% - +1% -1.26ms - +0.77ms
preact-main	65.07ms - 66.79ms	unsure 🔍 -1% - +2% -0.77ms - +1.26ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	2.98ms - 2.99ms	-	unsure 🔍 -1% - +0% -0.02ms - +0.01ms
preact-main	2.98ms - 3.00ms	unsure 🔍 -0% - +1% -0.01ms - +0.02ms	-

run-warmup-0

Version	Avg time	vs preact-local	vs preact-main
preact-local	29.82ms - 30.62ms	-	unsure 🔍 -1% - +2% -0.25ms - +0.69ms
preact-main	29.75ms - 30.24ms	unsure 🔍 -2% - +1% -0.69ms - +0.25ms	-

run-warmup-1

Version	Avg time	vs preact-local	vs preact-main
preact-local	33.68ms - 34.65ms	-	unsure 🔍 -4% - +0% -1.31ms - +0.15ms
preact-main	34.19ms - 35.29ms	unsure 🔍 -0% - +4% -0.15ms - +1.31ms	-

run-warmup-2

Version	Avg time	vs preact-local	vs preact-main
preact-local	25.25ms - 25.81ms	-	unsure 🔍 -1% - +2% -0.17ms - +0.46ms
preact-main	25.23ms - 25.54ms	unsure 🔍 -2% - +1% -0.46ms - +0.17ms	-

run-warmup-3

Version	Avg time	vs preact-local	vs preact-main
preact-local	26.35ms - 26.96ms	-	unsure 🔍 -2% - +1% -0.58ms - +0.31ms
preact-main	26.47ms - 27.11ms	unsure 🔍 -1% - +2% -0.31ms - +0.58ms	-

run-warmup-4

Version	Avg time	vs preact-local	vs preact-main
preact-local	27.45ms - 28.12ms	-	unsure 🔍 -0% - +3% -0.07ms - +0.79ms
preact-main	27.15ms - 27.70ms	unsure 🔍 -3% - +0% -0.79ms - +0.07ms	-

run-final

Version	Avg time	vs preact-local	vs preact-main
preact-local	20.02ms - 20.39ms	-	unsure 🔍 -1% - +2% -0.26ms - +0.39ms
preact-main	19.88ms - 20.40ms	unsure 🔍 -2% - +1% -0.39ms - +0.26ms	-

text-update

Browser: chrome-headless
Sample size: 230
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	2.04ms - 2.15ms	-	unsure 🔍 -1% - +5% -0.03ms - +0.11ms
preact-main	2.02ms - 2.10ms	unsure 🔍 -5% - +1% -0.11ms - +0.03ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	1.12ms - 1.12ms	-	unsure 🔍 +0% - +0% +0.00ms - +0.00ms
preact-main	1.12ms - 1.12ms	unsure 🔍 -0% - -0% -0.00ms - -0.00ms	-

todo

Browser: chrome-headless
Sample size: 50
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	34.74ms - 35.19ms	-	unsure 🔍 -1% - +1% -0.39ms - +0.39ms
preact-main	34.65ms - 35.28ms	unsure 🔍 -1% - +1% -0.39ms - +0.39ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	1.24ms - 1.27ms	-	unsure 🔍 -1% - +3% -0.01ms - +0.04ms
preact-main	1.23ms - 1.26ms	unsure 🔍 -3% - +1% -0.04ms - +0.01ms	-

update10th1k

Browser: chrome-headless
Sample size: 120
Built by: CI #4554
Commit: 5dc412d

duration

Version	Avg time	vs preact-local	vs preact-main
preact-local	30.21ms - 31.32ms	-	unsure 🔍 -1% - +5% -0.15ms - +1.42ms
preact-main	29.57ms - 30.68ms	unsure 🔍 -5% - +0% -1.42ms - +0.15ms	-

usedJSHeapSize

Version	Avg time	vs preact-local	vs preact-main
preact-local	2.95ms - 2.96ms	-	unsure 🔍 -0% - +0% -0.01ms - +0.01ms
preact-main	2.94ms - 2.96ms	unsure 🔍 -0% - +0% -0.01ms - +0.01ms	-

_{tachometer-reporter-action v2 for CI}

github-actions · 2025-02-22T09:38:37Z

Size Change: -59 B (-0.08%)

Total Size: 78.4 kB

Filename	Size	Change
`debug/dist/debug.js`	3.82 kB	-14 B (-0.37%)
`debug/dist/debug.mjs`	3.82 kB	-15 B (-0.39%)
`debug/dist/debug.module.js`	3.82 kB	-15 B (-0.39%)
`debug/dist/debug.umd.js`	3.9 kB	-15 B (-0.38%)

ℹ️ View Unchanged

Filename	Size
`compat/dist/compat.js`	4.12 kB
`compat/dist/compat.mjs`	4.05 kB
`compat/dist/compat.module.js`	4.05 kB
`compat/dist/compat.umd.js`	4.19 kB
`devtools/dist/devtools.js`	260 B
`devtools/dist/devtools.mjs`	274 B
`devtools/dist/devtools.module.js`	274 B
`devtools/dist/devtools.umd.js`	346 B
`dist/preact.js`	4.74 kB
`dist/preact.min.js`	4.74 kB
`dist/preact.min.module.js`	4.76 kB
`dist/preact.min.umd.js`	4.77 kB
`dist/preact.mjs`	4.75 kB
`dist/preact.module.js`	4.75 kB
`dist/preact.umd.js`	4.77 kB
`hooks/dist/hooks.js`	1.54 kB
`hooks/dist/hooks.mjs`	1.57 kB
`hooks/dist/hooks.module.js`	1.57 kB
`hooks/dist/hooks.umd.js`	1.61 kB
`jsx-runtime/dist/jsxRuntime.js`	978 B
`jsx-runtime/dist/jsxRuntime.mjs`	952 B
`jsx-runtime/dist/jsxRuntime.module.js`	952 B
`jsx-runtime/dist/jsxRuntime.umd.js`	1.05 kB
`test-utils/dist/testUtils.js`	473 B
`test-utils/dist/testUtils.mjs`	477 B
`test-utils/dist/testUtils.module.js`	477 B
`test-utils/dist/testUtils.umd.js`	555 B

_{compressed-size-action}

vitest.config.mjs

test/browser/components.test.js

coveralls · 2025-02-23T08:46:11Z

coverage: 99.523% (-0.09%) from 99.61%
when pulling a2dcfc6 on switch-to-vitest
into bb804cd on main.

rschristian

Lot of work went into this, great job!

However, I'm not in love with merging this as-is, entirely due to the state of Vitest, not any of the work here.

It seems slower across the board
It's an extra 100MB+ of node_modules & an 800MB in ~/.cache/ms-playright (tried using local Chrome install but I think that'd require different config/setup, it doesn't seem to function properly out-of-the-box)
It's a hell of a lot more dependencies
We've had to switch to some Vitest-specific APIs. While the "lock-in" doesn't look to be too rough, we are switching from runner agnostic (or meant to be runner agnostic) tools to something that's specific & will need to be changed if we need to migrate again in the future
We've encountered a fair number of Vitest bugs in the process of this PR

I suppose most importantly, I don't see a clear benefit to switching at the moment? It's something newer and that has some value, and we do shave a bit of config, but browser testing seems pretty well established, as are our tests, so I don't see any obvious possibilities this opens up. Karma, while deprecated, is super stable & doesn't seem to be holding us back at all.

I know Vitest's maintainers have been hard at work improving things, and with a bit of luck some of this will be addressed soon-ish, but at this particular moment it seems like a rather large regression instead of a clear improvement.

That being said, I don't want to hold this back at all; if people want this then let's go forward with it.

package.json

JoviDeCroock · 2025-02-24T10:39:49Z

It seems slower across the board and It's an extra 100MB+ of node_modules & an 800MB in ~/.cache/ms-playright (tried using local Chrome install but I think that'd require different config/setup, it doesn't seem to function properly out-of-the-box)

I agree, merging this feels like a regression, all though I am convinced that we should eventually move to a new tool because of karma being deprecated and it blocking upgrading our dependencies. I am going to explore the landscape a bit more, maybe there's a middle ground here.

Ideally webdriver would be as fast as playwright but... I am unsure where this is headed in terms of performance.

We've had to switch to some Vitest-specific APIs. While the "lock-in" doesn't look to be to rough, we are switching from runner agnostic (or meant to be runner agnostic) tools to something that's specific & will need to be changed if we need to migrate again in the future

This is more because sinon in more recent versions is very broken, when we use sinon.spyOn it will break the prototype of the underlying object 😅

We've encountered a fair number of Vitest bugs in the process of this PR

This was part of my intention, I wanted to uncover bugs so we can help the vitest people with testing browser mode as I am really looking for a community project to step into the shoes of karma.

The --skipLibCheck flags can probably be removed now that it was added to js-lint.json, right?

Well, sadly, the setting doesn't work 😅

I agree with you, let's hold off on merging this, I will surface this with the vitest people so there's information sharing.

marvinhagemeister · 2025-02-24T11:11:57Z

+1 on waiting on some vitest improvements before merging this. I don't mind the switch away from sinon as it hasn't been well maintained either.

rschristian · 2025-02-24T11:12:32Z

This is more because sinon in more recent versions is very broken, when we use sinon.spyOn it will break the prototype of the underlying object 😅

Oh fair enough, wasn't quite sure if it was sinon or vitest that was the problem there.

Well, sadly, the setting doesn't work 😅

Huh. I thought the flag would act the exact same there. Admittedly I can't even run test:ts:core at the moment, some Node error... test:ts:compat is a-ok though. Fair enough on that too then.

.github/workflows/build-test.yml

rschristian

Looks great!

marvinhagemeister

LGTM, great work!

This reverts commit 0e1893c, reversing changes made to bb804cd.

JoviDeCroock force-pushed the switch-to-vitest branch from 80efc74 to ea827e9 Compare February 18, 2025 07:00

rschristian reviewed Feb 22, 2025

View reviewed changes

vitest.config.mjs Show resolved Hide resolved

rschristian reviewed Feb 22, 2025

View reviewed changes

vitest.config.mjs Show resolved Hide resolved

JoviDeCroock commented Feb 22, 2025

View reviewed changes

test/browser/components.test.js Outdated Show resolved Hide resolved

JoviDeCroock force-pushed the switch-to-vitest branch 2 times, most recently from da6d806 to 2bfe8b0 Compare February 23, 2025 08:43

JoviDeCroock force-pushed the switch-to-vitest branch from 948a4e4 to 091a3ab Compare February 23, 2025 09:26

JoviDeCroock marked this pull request as ready for review February 24, 2025 09:18

JoviDeCroock force-pushed the switch-to-vitest branch 2 times, most recently from 6f8b8ef to 0a4c379 Compare February 24, 2025 09:21

rschristian approved these changes Feb 24, 2025

View reviewed changes

package.json Outdated Show resolved Hide resolved

JoviDeCroock commented Feb 24, 2025

View reviewed changes

.github/workflows/build-test.yml Show resolved Hide resolved

JoviDeCroock force-pushed the switch-to-vitest branch 5 times, most recently from 97ebbf3 to 5dc412d Compare February 25, 2025 10:26

JoviDeCroock force-pushed the switch-to-vitest branch from 5dc412d to 2bf8e23 Compare March 7, 2025 12:40

JoviDeCroock and others added 6 commits April 25, 2025 07:53

Vitest switch

62a9b19

Make render work

0db9822

Phewie

bcd6310

fix: dedupe preact (#4702)

edf8b79

Getting close

ee7fbe2

All crashes are gone

d90a8a3

rschristian and others added 14 commits April 25, 2025 07:55

test: Copy karma's filter over

e152bf0

Try a cache

1c1ab32

Fix test regex

3ed298b

Try a run with isolate false

fb71e9f

Bump node and use playwright

b38f5f2

Restore CI

473c65b

Update .github/workflows/build-test.yml

70e6f1f

Switch to webdirver

ced6160

Package json

7eba152

Bump vitest

7651074

Add pending

8d64419

Skip more files

f8ddf37

Expand

87eed69

Bump some stuff

e6a4080

JoviDeCroock force-pushed the switch-to-vitest branch from 41d9a27 to e6a4080 Compare April 25, 2025 05:58

bump vitest

e7cfb64

JoviDeCroock force-pushed the switch-to-vitest branch 4 times, most recently from 5fed9f3 to a2dcfc6 Compare April 25, 2025 07:01

JoviDeCroock added 2 commits April 25, 2025 09:02

Enable lint

a2dcfc6

Try some stuff

94db9b4

rschristian approved these changes Apr 27, 2025

View reviewed changes

marvinhagemeister approved these changes Apr 28, 2025

View reviewed changes

JoviDeCroock merged commit 0e1893c into main Apr 28, 2025
4 checks passed

JoviDeCroock deleted the switch-to-vitest branch April 28, 2025 08:12

JoviDeCroock mentioned this pull request May 5, 2025

10.26.6 #4755

Merged

JoviDeCroock added a commit that referenced this pull request May 9, 2025

Revert "Vitest switch (#4687)"

2e2a867

This reverts commit 0e1893c, reversing changes made to bb804cd.

rschristian mentioned this pull request Jun 7, 2025

chore: Bump terser #4791

Closed

rschristian mentioned this pull request Dec 27, 2025

types: Correct outdated imports in jsx-runtime #4981

Merged

Uh oh!

Conversation

JoviDeCroock commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Requirements before merge

Follow-ups

Uh oh!

github-actions bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Tachometer Benchmark Results

Summary

duration

usedJSHeapSize

Results

duration

usedJSHeapSize

duration

usedJSHeapSize

duration

usedJSHeapSize

duration

usedJSHeapSize

duration

usedJSHeapSize

run-warmup-0

run-warmup-1

run-warmup-2

run-warmup-3

run-warmup-4

run-final

duration

usedJSHeapSize

duration

usedJSHeapSize

duration

usedJSHeapSize

Uh oh!

github-actions bot commented Feb 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rschristian left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoviDeCroock commented Feb 24, 2025

Uh oh!

marvinhagemeister commented Feb 24, 2025

Uh oh!

rschristian commented Feb 24, 2025

Uh oh!

Uh oh!

rschristian left a comment

Choose a reason for hiding this comment

Uh oh!

marvinhagemeister left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JoviDeCroock commented Feb 18, 2025 •

edited

Loading

github-actions bot commented Feb 18, 2025 •

edited

Loading

github-actions bot commented Feb 22, 2025 •

edited

Loading

coveralls commented Feb 23, 2025 •

edited

Loading

rschristian left a comment •

edited

Loading