feat(next): turbopack build support, fix incorrect handling of external packages #14475
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
📦 esbuild Bundle Analysis for payloadThis analysis was generated by esbuild-bundle-analyzer. 🤖 |
AlessioGr
commented
Nov 5, 2025
| ...(process.env.NODE_ENV === 'development' && options.devBundleServerPackages === false | ||
| ? [ | ||
| 'payload', | ||
| '@payloadcms/db-mongodb', |
Member
Author
There was a problem hiding this comment.
These are always externalized now, not just during dev, thus the removal
AlessioGr
commented
Nov 5, 2025
| { module: /node_modules\/mongodb\/lib\/bson\.js/ }, | ||
| { file: /node_modules\/mongodb\/lib\/bson\.js/ }, | ||
| ], | ||
| externals: [...(incomingWebpackConfig?.externals || []), 'require-in-the-middle'], |
Member
Author
There was a problem hiding this comment.
the dependencies in webpack.externals are now always externalized server-side - see serverExternalDependencies.
On the client, those dependencies are (and should) never imported anyways, thus no point in manually declaring them here
AlessioGr
commented
Nov 5, 2025
| 'libsql', | ||
| 'require-in-the-middle', | ||
| ], | ||
| ignoreWarnings: [ |
Member
Author
There was a problem hiding this comment.
This is probably a relict from the old next-poc branch days. Doubt we'll run into issues now that this is properly externalized. Works fine when testing it. If we do run into issues, we can add it back with a comment.
AlessioGr
commented
Nov 5, 2025
| }, | ||
| fallback: { | ||
| ...(incomingWebpackConfig?.resolve?.fallback || {}), | ||
| '@aws-sdk/credential-providers': false, |
Member
Author
There was a problem hiding this comment.
This is a relict as well. With our clean server-client separation, those dependencies shouldn't sneak into the client bundle anyways.
Member
There was a problem hiding this comment.
webpack also is used for the server-side bundle in nextjs
jmikrut
approved these changes
Nov 6, 2025
zubricks
pushed a commit
that referenced
this pull request
Nov 10, 2025
…al packages (#14475) Fixes #14338 and vercel/next.js#85110 (comment). This PR should fix support for using turbopack to build your Next.js project (which is now the default in Next.js 16, unless you pass `next build --webpack`). Previously, we were directly externalizing dependencies like `libsql` as they cause an error when being bundled. However, as those dependencies were not installed at the top-level by the user, the resulting `require('libsql')` calls could fail in production. Turbopack threw the following warning: ```bash Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.\nTry to install it into the project directory by running ```` However, we ignored this warning, as we cannot force users to manually install dependencies of our dependencies. In Next.js 16, when using Turbopack build, this is no longer a warning but an error, which is why attempting to use turbopack build right now will not work. This PR properly fixes this externalization issue, by externalizing the `entry points` of those problematic packages (our db adapters). This means we no longer need to hide the warning.
Contributor
|
🚀 This is included in version v3.64.0 |
AlessioGr
added a commit
that referenced
this pull request
Nov 20, 2025
…on (#14696) ## Background The following PRs attempted to add support for Turbopack build: - #14475 - #14473 ## The Fundamental Problem Payload's database adapters (e.g., `@payloadcms/db-postgres`) depend on packages with native dependencies that cannot be bundled (e.g., `drizzle-kit`, which imports `esbuild`). We need to externalize these packages. **Why we can't externalize them directly:** With pnpm, externalizing a package like `drizzle-kit` generates `require('drizzle-kit')` calls in the bundle. However, `drizzle-kit` is not in the user's `package.json` (it's a transitive dependency installed by `db-postgres`). pnpm's strict dependency isolation prevents importing dependencies of dependencies, causing runtime failures. **The attempted workaround:** Instead of externalizing `drizzle-kit`, we tried externalizing the entry-point package `@payloadcms/db-postgres` (which users DO install) via `serverExternalPackages`. This works in development, but creates severe issues in production builds. ## Why the Workaround Failed When you externalize `@payloadcms/db-postgres`: 1. **Everything it imports becomes external**, including `payload` itself 2. This creates **two copies of `payload`**: - One bundled (from user's direct imports) - One external in node_modules (from db-postgres's imports) 3. **Bundle size explodes** due to: - Disabled tree-shaking for externalized packages - Duplicate package installations - Loss of code-splitting optimizations **Another example of duplication:** ``` @payloadcms/richtext-lexical (bundled) → qs-esm (bundled) payload (external) → qs-esm (external) Result: Two copies of qs-esm in production ``` This issue was reported on our discord [here](https://discord.com/channels/967097582721572934/1422639568808841329/1440689060015374437). ## The Solution (This PR) **Short term:** Disable Turbopack build support until Next.js provides a proper solution. ### Why Webpack Works Webpack has `webpack.externals`, which can externalize **any** package regardless of whether it's in the user's `package.json`: - We externalize `drizzle-kit` directly via `webpack.externals` - Webpack generates `require('drizzle-kit')` calls in the bundle - At runtime, Node.js resolves these just fine - we're not yet sure why that is - We avoid externalizing entry-point packages, preventing the duplication problem ### Why Turbopack Build Doesn't Work Turbopack only has `serverExternalPackages` (similar to webpack.externals but with restrictions): - **The constraint**: Packages must be resolvable from the project root (i.e., in the user's `package.json`) - If a package isn't directly installed by the user, Next.js **ignores the externalization rule** and tries to bundle it anyway - This forces us to externalize entry-point packages (`db-postgres`), which causes the duplication and bundle size problems described above ### Why Turbopack Dev Works Turbopack dev has the same `serverExternalPackages` constraint, BUT: - **In dev, we can afford the trade-off** of externalizing entry-point packages because: - Bundle size doesn't matter in development - Faster compilation speed is more important - We're not shipping to production - The duplication problem still exists, but it's acceptable for the dev experience **Changes made:** 1. **Throw error for Turbopack builds** - Prevent production builds with Turbopack until Next.js fixes the underlying issue 2. **Restore webpack.externals** - Use webpack-specific externals for problematic transitive dependencies (`drizzle-kit`, `sharp`, `libsql`, etc.) that aren't in user's package.json 3. **Simplify serverExternalPackages** - Only externalize packages resolvable from project root (`graphql`, `@sentry/nextjs`) 4. **Clean up unnecessary config** - Remove webpack configurations that are no longer justifiable. Any configuration we have left now comes with a comment block explaining why we need it 5. **enable devBundleServerPackages optimization by default** - there have not been any reported issues since this was introduced, and this setting is now **necessary** for turbopack support during dev ## Future In order to properly support Turbopack Build, Next.js will have to implement one of these solutions: - **Option 1**: Implement webpack.externals-like functionality for Turbopack (no package.json constraint) - **Option 2**: Remove the need for declaring all externals as direct dependencies in the application We're tracking Next.js's progress on this issue.
zubricks
pushed a commit
that referenced
this pull request
Jan 6, 2026
…on (#14696) ## Background The following PRs attempted to add support for Turbopack build: - #14475 - #14473 ## The Fundamental Problem Payload's database adapters (e.g., `@payloadcms/db-postgres`) depend on packages with native dependencies that cannot be bundled (e.g., `drizzle-kit`, which imports `esbuild`). We need to externalize these packages. **Why we can't externalize them directly:** With pnpm, externalizing a package like `drizzle-kit` generates `require('drizzle-kit')` calls in the bundle. However, `drizzle-kit` is not in the user's `package.json` (it's a transitive dependency installed by `db-postgres`). pnpm's strict dependency isolation prevents importing dependencies of dependencies, causing runtime failures. **The attempted workaround:** Instead of externalizing `drizzle-kit`, we tried externalizing the entry-point package `@payloadcms/db-postgres` (which users DO install) via `serverExternalPackages`. This works in development, but creates severe issues in production builds. ## Why the Workaround Failed When you externalize `@payloadcms/db-postgres`: 1. **Everything it imports becomes external**, including `payload` itself 2. This creates **two copies of `payload`**: - One bundled (from user's direct imports) - One external in node_modules (from db-postgres's imports) 3. **Bundle size explodes** due to: - Disabled tree-shaking for externalized packages - Duplicate package installations - Loss of code-splitting optimizations **Another example of duplication:** ``` @payloadcms/richtext-lexical (bundled) → qs-esm (bundled) payload (external) → qs-esm (external) Result: Two copies of qs-esm in production ``` This issue was reported on our discord [here](https://discord.com/channels/967097582721572934/1422639568808841329/1440689060015374437). ## The Solution (This PR) **Short term:** Disable Turbopack build support until Next.js provides a proper solution. ### Why Webpack Works Webpack has `webpack.externals`, which can externalize **any** package regardless of whether it's in the user's `package.json`: - We externalize `drizzle-kit` directly via `webpack.externals` - Webpack generates `require('drizzle-kit')` calls in the bundle - At runtime, Node.js resolves these just fine - we're not yet sure why that is - We avoid externalizing entry-point packages, preventing the duplication problem ### Why Turbopack Build Doesn't Work Turbopack only has `serverExternalPackages` (similar to webpack.externals but with restrictions): - **The constraint**: Packages must be resolvable from the project root (i.e., in the user's `package.json`) - If a package isn't directly installed by the user, Next.js **ignores the externalization rule** and tries to bundle it anyway - This forces us to externalize entry-point packages (`db-postgres`), which causes the duplication and bundle size problems described above ### Why Turbopack Dev Works Turbopack dev has the same `serverExternalPackages` constraint, BUT: - **In dev, we can afford the trade-off** of externalizing entry-point packages because: - Bundle size doesn't matter in development - Faster compilation speed is more important - We're not shipping to production - The duplication problem still exists, but it's acceptable for the dev experience **Changes made:** 1. **Throw error for Turbopack builds** - Prevent production builds with Turbopack until Next.js fixes the underlying issue 2. **Restore webpack.externals** - Use webpack-specific externals for problematic transitive dependencies (`drizzle-kit`, `sharp`, `libsql`, etc.) that aren't in user's package.json 3. **Simplify serverExternalPackages** - Only externalize packages resolvable from project root (`graphql`, `@sentry/nextjs`) 4. **Clean up unnecessary config** - Remove webpack configurations that are no longer justifiable. Any configuration we have left now comes with a comment block explaining why we need it 5. **enable devBundleServerPackages optimization by default** - there have not been any reported issues since this was introduced, and this setting is now **necessary** for turbopack support during dev ## Future In order to properly support Turbopack Build, Next.js will have to implement one of these solutions: - **Option 1**: Implement webpack.externals-like functionality for Turbopack (no package.json constraint) - **Option 2**: Remove the need for declaring all externals as direct dependencies in the application We're tracking Next.js's progress on this issue.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #14338 and vercel/next.js#85110 (comment).
This PR should fix support for using turbopack to build your Next.js project (which is now the default in Next.js 16, unless you pass
next build --webpack).Previously, we were directly externalizing dependencies like
libsqlas they cause an error when being bundled. However, as those dependencies were not installed at the top-level by the user, the resultingrequire('libsql')calls could fail in production.Turbopack threw the following warning:
However, we ignored this warning, as we cannot force users to manually install dependencies of our dependencies.
In Next.js 16, when using Turbopack build, this is no longer a warning but an error, which is why attempting to use turbopack build right now will not work.
This PR properly fixes this externalization issue, by externalizing the
entry pointsof those problematic packages (our db adapters). This means we no longer need to hide the warning.