esbuild + ESM + NodeJS + Typescript troubles

[bschick]

Trying to get the combo in the title of this ticket working and have not found the correct incantation. I found a few posts saying it won't work, so decided to report a bug / feature request / post to understand if this should work.

Setup

Simple server app that will run in nodejs v20.x in an AWS lambda function
Would like to use ESM style modules
Using just '@simplewebauthn/server' package and native node
Ubuntu 22.04 in a VM on MacBook Pro
I have tried various config options, this was the latest attempt:

package.json

{
  "name": "server",
  "type": "module",
  "main": "index.ts",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^20.11.16",
    "esbuild": "^0.20.0",
    "typescript": "^5.3.3"
  },
  "dependencies": {
    "@simplewebauthn/server": "^9.0.1"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "es2022",
    "lib": ["es2022"],
    "moduleDetection": "force",
    "module": "es6",
    "rootDir": "src",
    "moduleResolution": "Bundler",
    "noEmit": true,
    "resolveJsonModule": true,
    "allowJs": true,
    "outDir": "build",
    "isolatedModules": true,
    "verbatimModuleSyntax": true,
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

index.ts

"src/index.ts" 25L, 645B                                                                                15,19         All
import { generateAuthenticationOptions } from '@simplewebauthn/server';
import {readFileSync} from "fs"

export async function handler() {
  // test local module import
  const options = await generateAuthenticationOptions({
     rpID: "server",
     userVerification: 'preferred',
     challenge: new Uint8Array([2,89,201,0,2,45,5,7,223,50,1,90,194,44,54, 3, 32,123,220]),
   });

  console.log(options);

  // testing native NodeJS import
  fs.readFileSync('index.js', 'utf8');
};

await handler();

Results

If I build with the following:
./node_modules/.bin/esbuild --bundle --format=esm --target=es2022 --outdir=build src/index.ts

I get the error below, which is what I'd expect. If I delete the nodejs native fs usage, the project builds and runs without error (but I need to be able import nodejs modules). Also, with fs removed if I edit the output index.js file there are no require statements.

✘ [ERROR] Could not resolve "fs"

src/index.ts:2:27:
2 │ import {readFileSync} from "fs"
╵ ~~~~

The package "fs" wasn't found on the file system but is built into node. Are you trying to bundle
for node? You can use "--platform=node" to do that, which will remove this error.

As described in esbuild docs, I add the --platform=node option as follows:
./node_modules/.bin/esbuild --bundle --format=esm --platform=node --target=es2022 --outdir=build src/index.ts

The project builds but when I run it with

nodejs build/index.js

I get an error like the following with a stack trace. When I edit the output index.js file I find many require statements.

Error: Dynamic require of "stream" is not supported

Is there a way to use esbuild for a typescript project with ESM modules and nodejs?

[hyrious]

First of all let's explain the error:

Error: Dynamic require of "stream" is not supported

It is thrown by esbuild's require wrapper, meaning that there's no global require function found in the runtime. It is true since you're bundling to ESM and running it in ESM, where there's no global require.

 var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof 
 Proxy !== "undefined" ? new Proxy(x, {
   get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
 }) : x)(function(x) {
   if (typeof require !== "undefined")
     return require.apply(this, arguments);
+  throw Error('Dynamic require of "' + x + '" is not supported');
 });

So which code causes this error? You can find the call-chain:

->: depends on / calls
(ESM | CJS): the package's format

@simplewebauthn/server (ESM) ->
  cross-fetch (CJS) ->
    node-fetch (CJS): require('stream')
      since 'stream' is a Node.js builtin module,
      esbuild --platform=node will externalize it (i.e. leave a require call here)

You can find more details in #1921.

There are some solutions but none of them is the silver bullet.

• Since it complains no global require, you can provide one: How to fix "Dynamic require of "os" is not supported" #1921 (comment)
• You can replace the require call with import statements, like what @rollup/plugin-commonjs did: How to fix "Dynamic require of "os" is not supported" #1921 (comment) (I wrote a similar plugin for it). This way is not always safe since dynamic require can optional be called (if (1) require(a) else require(b)) but import statements always take effect before other codes.
• You can externalize all these dependencies via --packages=external and install node modules in your final runtime, so that Node.js will be happy to run CJS codes in CJS context and ESM codes in ESM context. AWS Lambda maybe not provide a way to do this.

[bschick]

Appreciate the quick response. I'm new to the JS and Node, so the plethora of module formats and versions is a bit baffling. Looking at the actual node_module/node-fetch module causing the problem, it has both .js and .mjs files. The .mjs uses import Stream from 'stream'; as you'd expect.

Could esbuild pull in the mjs file even though the parent used require? That would bundle the import instead of the require. I guess that could lead to other problems if the .js and .mjs files had different exports or something (or the original problem would just move elsewhere assuming other modules required built-in node modules).

Its also unclear to me why esbuild without --platfrom=node but keeping --format=esm causes an error when the top level file imports from fs but it does not complain when a dependent file requires stream. Wouldn't they both just get bundled (copying in the native node modules)?

[hyrious]

Could esbuild pull in the mjs file even though the parent used require?

Yep. If you looking at the node-fetch's package.json, the module field points to the mjs file AND there's no exports field that fully controls the resolving process. So you can use --main-fields=module,main to tell esbuild to prefer the file pointed by module over main. By default esbuild would prefer main when --platform=node.

However that won't make your example above work because node-fetch depends on whatwg-url which is a CJS only package, the require('punycode') in its code will throw similar errors.

Its also unclear to me why…but it does not complain when a dependent file requires stream.

This was because esbuild recursively resolves source files and it will shutdown when it hit any errors. Your src/index.ts is the first file to be processed and import 'fs' in that file throws an error, so esbuild just stops here.

[bschick]

This was because esbuild recursively resolves source files and it will shutdown when it hit any errors. Your src/index.ts is the first file to be processed and import 'fs' in that file throws an error, so esbuild just stops here.

But if I delete all use of fs from src/index.ts the build works and the output index.js runs without problems.

[hyrious]

But if I delete all use of fs from src/index.ts the build works and the output index.js runs without problems.

Sorry I didn't understand the question. You're asking why this command works:

esbuild --bundle --format=esm src/index.ts

When there's no --platform=node, the default platform is browser which means that esbuild would try to produce a bundle that can run in the browser (which doesn't mean that it cannot run in Node.js, as long as the features it used is compatible in these environments). The resolver would prefer browser specific fields in package.json (browser > module > main). The result is cross-fetch points to dist/browser-ponyfill.js which does not depend on any Node.js builtin modules. Thus you won't trigger the error Dynamic require of "stream" is not supported.

[bschick]

Thanks for all the detail. I hope this helps other newbies