Replace optionalDependencies with script for installing swc #33496

ijjk · 2022-01-20T13:54:29Z

This continues the work in #32850 to reduce install size/time for the next-swc binary. Since optionalDependencies don't allow us to accurately download the specific binary we need per-platform since platforms can match arch and platform but still need a different binary e.g. musl vs glibc for linux this adds a custom install script which will directly download the correct binary.

Initial testing seems to reduce install time by 2 -3 seconds on mac OS and Windows even without the local cache for the next-swc binary being leveraged. We can expand on the install script in the future to download the WASM variant of next-swc if a platform specific binary is not available yet.

before (yarn cache was cleared before run)

yarn add next@/Users/ijjk/dev/vercel/next.js/packages/next/next-v12.0.9-canary.3-optional.tgz
yarn add v1.22.17
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning " > [email protected]" has unmet peer dependency "react@^17.0.2 || ^18.0.0-0".
warning " > [email protected]" has unmet peer dependency "react-dom@^17.0.2 || ^18.0.0-0".
warning "next > [email protected]" has unmet peer dependency "react@>= 16.8.0 || 17.x.x || 18.x.x".
warning "next > [email protected]" has unmet peer dependency "react@^16.8.0 || ^17.0.0".
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 11 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ @next/[email protected]
├─ @next/[email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
└─ [email protected]
✨  Done in 6.96s.

after (yarn cache was cleared before run)

yarn add next@/Users/ijjk/dev/vercel/next.js/packages/next/next-v12.0.9-canary.3-install.tgz
yarn add v1.22.17
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning " > [email protected]" has unmet peer dependency "react@^17.0.2 || ^18.0.0-0".
warning " > [email protected]" has unmet peer dependency "react-dom@^17.0.2 || ^18.0.0-0".
warning "next > [email protected]" has unmet peer dependency "react@>= 16.8.0 || 17.x.x || 18.x.x".
warning "next > [email protected]" has unmet peer dependency "react@^16.8.0 || ^17.0.0".
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 10 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ @next/[email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
└─ [email protected]
✨  Done in 3.69s.

Brooooooklyn · 2022-01-20T15:08:27Z

Yarn berry has started working on the libc field in the package.json yarnpkg/berry#3981

ijjk · 2022-01-20T16:02:03Z

@Brooooooklyn I'm curious if optionalDependencies is the right solution here even if the musl/glib issue is fixed in all relevant package managers as it still requires resolving all of the package data for each optionalDependency from the registry even though we can know earlier if the package won't be needed for specific platforms.

styfle · 2022-01-20T16:02:52Z

packages/next/install-swc.js

+
+if (!activePackage) {
+  // TODO: should this be a hard error even though it fails install?
+  // should we fallback to wasm in this case?


Falling back to wasm makes sense to me.

Thats the advantage of this custom install script 👍

Yeah, falling back to wasm sounds good to me.

packages/next/install-swc.js

merceyz · 2022-01-20T16:17:05Z

packages/next/install-swc.js

+
+  try {
+    await fetch(
+      `https://registry.npmjs.org/${activePackage.packageName}/-/${tarFileName}`


Some setups don't have access to the npm registry so this will be problematic
Ref evanw/esbuild#1621

Yeah this is problematic because package config like .npmrc or .yarnrc is no longer respected evanw/esbuild#286

Perhaps we need to double down on getting this feature into npm now that yarn just landed it in yarnpkg/berry#3981

We can definitely handle loading the registry from the package config. I think it's gonna be a while before this is sorted out/performant in relevant package managers so the custom install script will still be needed for a bit.

Brooooooklyn · 2022-01-20T16:21:14Z

I'm curious if optionalDependencies is the right solution here even if the musl/glib issue is fixed in all relevant package managers as it still requires resolving all of the package data for each optionalDependency from the registry even though we can know earlier if the package won't be needed for specific platforms.

Yes, resolving extra meta info from the registry is a massive problem in the optionalDependencies strategy.

We are discussing some other solutions for native packages npm/rfcs#438 (comment)

Co-authored-by: Steven <[email protected]>

padmaia · 2022-01-20T19:02:30Z

packages/next/build/swc/index.js

+      break
+    } catch (e) {}
+  }
+
  if (!bindings) {


Do we need this block any more?

I don't think we should for most cases although we could leave it in case users want to manually install the package and skip our custom install script 🤔

ijjk · 2022-01-20T19:04:51Z

Stats from current PR

Default Build (Decrease detected ✓)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
buildDuration	18.5s	18.8s	⚠️ +294ms
buildDurationCached	4.1s	4.2s	⚠️ +40ms
nodeModulesSize	355 MB	355 MB	⚠️ +102 kB

Page Load Tests Overall decrease ⚠️

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.995	3.996	0
/ avg req/sec	625.77	625.62	⚠️ -0.15
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	2.067	2.189	⚠️ +0.12
/error-in-render avg req/sec	1209.71	1141.99	⚠️ -67.72

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
450.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42.2 kB	42.2 kB	✓
main-HASH.js gzip	27.2 kB	27.2 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	71 kB	71 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
_app-HASH.js gzip	1.37 kB	1.37 kB	✓
_error-HASH.js gzip	194 B	194 B	✓
amp-HASH.js gzip	312 B	312 B	✓
css-HASH.js gzip	326 B	326 B	✓
dynamic-HASH.js gzip	2.37 kB	2.37 kB	✓
head-HASH.js gzip	350 B	350 B	✓
hooks-HASH.js gzip	919 B	919 B	✓
image-HASH.js gzip	4.87 kB	4.87 kB	✓
index-HASH.js gzip	263 B	263 B	✓
link-HASH.js gzip	2.13 kB	2.13 kB	✓
routerDirect..HASH.js gzip	321 B	321 B	✓
script-HASH.js gzip	383 B	383 B	✓
withRouter-HASH.js gzip	318 B	318 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.2 kB	14.2 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
_buildManifest.js gzip	459 B	459 B	✓
Overall change	459 B	459 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
index.html gzip	531 B	531 B	✓
link.html gzip	545 B	545 B	✓
withRouter.html gzip	526 B	526 B	✓
Overall change	1.6 kB	1.6 kB	✓

Default Build with SWC (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
buildDuration	23.3s	23.5s	⚠️ +195ms
buildDurationCached	4.2s	4.1s	-34ms
nodeModulesSize	355 MB	355 MB	⚠️ +102 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
/ failed reqs	0	0	✓
/ total time (seconds)	4	4.061	⚠️ +0.06
/ avg req/sec	624.94	615.55	⚠️ -9.39
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	2.159	2.117	-0.04
/error-in-render avg req/sec	1157.86	1180.64	+22.78

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
450.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42.3 kB	42.3 kB	✓
main-HASH.js gzip	27.3 kB	27.3 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	71.3 kB	71.3 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
_app-HASH.js gzip	1.35 kB	1.35 kB	✓
_error-HASH.js gzip	180 B	180 B	✓
amp-HASH.js gzip	305 B	305 B	✓
css-HASH.js gzip	321 B	321 B	✓
dynamic-HASH.js gzip	2.36 kB	2.36 kB	✓
head-HASH.js gzip	342 B	342 B	✓
hooks-HASH.js gzip	911 B	911 B	✓
image-HASH.js gzip	4.88 kB	4.88 kB	✓
index-HASH.js gzip	256 B	256 B	✓
link-HASH.js gzip	2.19 kB	2.19 kB	✓
routerDirect..HASH.js gzip	314 B	314 B	✓
script-HASH.js gzip	375 B	375 B	✓
withRouter-HASH.js gzip	309 B	309 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.2 kB	14.2 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
_buildManifest.js gzip	459 B	459 B	✓
Overall change	459 B	459 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js update/swc-downloading	Change
index.html gzip	532 B	532 B	✓
link.html gzip	545 B	545 B	✓
withRouter.html gzip	526 B	526 B	✓
Overall change	1.6 kB	1.6 kB	✓

Commit: 54bd12a

styfle · 2022-01-26T03:14:10Z

packages/next/install-swc.js

+  // should we fallback to wasm in this case?
+  console.error(
+    `Error: unsupported next-swc platform: ` +
+      `${process.platform} ${process.arch} (glibc: ${isGlibc})\n` +


Suggested change

`${process.platform} ${process.arch} (glibc: ${isGlibc})\n` +

`${process.platform} ${process.arch} (glibcVersionRuntime: ${glibcVersionRuntime})\n` +

Follow-up to #36527 this adds falling back to the wasm swc build when loading the native bindings fails so that we don't block the build on the native dependency being available. This continues off of #33496 but does not add a postinstall script yet and only downloads the fallback when the native dependency fails to load.

ijjk

Going to close this for now as we have wasm fallback handling now and taking separate steps to refine this installation process.

Replace optionalDependencies with script for installing swc

1061b2a

ijjk added created-by: Next.js team PRs by the Next.js team. type: next labels Jan 20, 2022

This comment has been minimized.

Sign in to view

ijjk marked this pull request as ready for review January 20, 2022 15:58

ijjk requested review from huozhi, padmaia, shuding and timneutkens as code owners January 20, 2022 15:58

ijjk requested a review from styfle January 20, 2022 15:58

styfle reviewed Jan 20, 2022

View reviewed changes

packages/next/install-swc.js Outdated Show resolved Hide resolved

merceyz reviewed Jan 20, 2022

View reviewed changes

Apply suggestions from code review

54bd12a

Co-authored-by: Steven <[email protected]>

padmaia reviewed Jan 20, 2022

View reviewed changes

styfle reviewed Jan 26, 2022

View reviewed changes

ijjk mentioned this pull request May 2, 2022

Add falling back to wasm swc build on native load failure #36612

Merged

ijjk commented Jul 26, 2022

View reviewed changes

ijjk closed this Jul 26, 2022

github-actions bot locked as resolved and limited conversation to collaborators Aug 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace optionalDependencies with script for installing swc #33496

Replace optionalDependencies with script for installing swc #33496

ijjk commented Jan 20, 2022 •

edited

Loading

This comment has been minimized.

Brooooooklyn commented Jan 20, 2022

ijjk commented Jan 20, 2022

styfle Jan 20, 2022

padmaia Jan 20, 2022

merceyz Jan 20, 2022 •

edited

Loading

styfle Jan 21, 2022 •

edited

Loading

ijjk Jan 21, 2022

Brooooooklyn commented Jan 20, 2022

padmaia Jan 20, 2022

ijjk Jan 20, 2022

ijjk commented Jan 20, 2022

styfle Jan 26, 2022

ijjk left a comment •

edited

Loading

	`${process.platform} ${process.arch} (glibc: ${isGlibc})\n` +
	`${process.platform} ${process.arch} (glibcVersionRuntime: ${glibcVersionRuntime})\n` +

Replace optionalDependencies with script for installing swc #33496

Replace optionalDependencies with script for installing swc #33496

Conversation

ijjk commented Jan 20, 2022 • edited Loading

This comment has been minimized.

Brooooooklyn commented Jan 20, 2022

ijjk commented Jan 20, 2022

styfle Jan 20, 2022

Choose a reason for hiding this comment

padmaia Jan 20, 2022

Choose a reason for hiding this comment

merceyz Jan 20, 2022 • edited Loading

Choose a reason for hiding this comment

styfle Jan 21, 2022 • edited Loading

Choose a reason for hiding this comment

ijjk Jan 21, 2022

Choose a reason for hiding this comment

Brooooooklyn commented Jan 20, 2022

padmaia Jan 20, 2022

Choose a reason for hiding this comment

ijjk Jan 20, 2022

Choose a reason for hiding this comment

ijjk commented Jan 20, 2022

Stats from current PR

styfle Jan 26, 2022

Choose a reason for hiding this comment

ijjk left a comment • edited Loading

Choose a reason for hiding this comment

ijjk commented Jan 20, 2022 •

edited

Loading

merceyz Jan 20, 2022 •

edited

Loading

styfle Jan 21, 2022 •

edited

Loading

ijjk left a comment •

edited

Loading