Compile to ECMAScript Modules (ESM) instead of CommonJS

[webmaster128]

Right now, CosmJS produces CommonJS builds. This works well for all versions of Node.js as well as bundlers like Webpack in order to bundle for the web.

As mentioned in https://vuejsdevelopers.com/2019/02/04/vue-es-module-browser-build/ ("What benefit is there of using Vue as an ES module?"), there is little obvious benefit for switching from CommonJS to ESM since the consumers are either node.js apps or bundlers. However, there is hope that ESM helps optimizing bundle size through tree shaking because ESM can better do static analysis on what is used and what is not (https://webpack.js.org/guides/tree-shaking/). In #904 you see that right now web apps include a lot of unused code that is only needed for key storage.

Other libraries use bundlers to bundle ES modules for the web. This is not desired here as we don't have a single entry point and do not want to decide for that user what should be included in such a bundle.

Do we need CommonJS+ESM hybrid builds?

• Node.js 12 does not support ESM natively without feature flags. But this version reaches EOL at 2022-04-30.
• Webpack support ESM inputs since version 2 ("The webpack 2 release came with built-in support for ES2015 modules (alias harmony modules)")

Does this reduce bundle size

• Test in a next.js project with the analyze plugin

Testing

• Jasmine supports ESM build testing (see Support ES modules with .js extension and package type module jasmine/jasmine-npm#170)
• JSON test vectors cannot be used easily in tests. const bip39Vectors = require("./src/testdata/bip39.json"); works and TS transpiles import bip39Vectors from "./testdata/bip39.json"; to const bip39_json_1 = __importDefault(require("./testdata/bip39.json")). But an untranspiled import bip39Vectors from "./testdata/bip39.json"; output is not working. See https://nodejs.org/api/esm.html#no-json-module-loading and https://nodejs.org/api/esm.html#json-modules. A workaround for tests is:

  import { createRequire } from "module";
  const require = createRequire(import.meta.url);
  const bip39Vectors = require("../src/testdata/bip39.json");

  but this does not work in Webpack/Karma.
• File extensions must me used. import { EnglishMnemonic } from "./englishmnemonic.js"; must be used for a local file instead of import { EnglishMnemonic } from "./englishmnemonic";. TS will not do this for us.

[dynst]

Working ESM support is partially blocked on cosmos/cosmjs-types#93 which might involve changes to how cosmjs-types exposes its API - either imports need to specify the .js / .mjs / .cjs filename extension to use anything from that package, or they can only import something listed in its "exports" field (it currently doesn't have one). on a further look, adding .js to every import works fine as long as there's no "exports" field in a legacy package.

https://nodejs.org/api/esm.html#import-specifiers

• JSON test vectors cannot be used easily in tests.

Didn't Node.js v17.1 and v16.14 add support for import data from './data.json' assert { type: 'json' }; and await import('./data.json', { assert: { type: 'json' } }).

• File extensions must me used. import { EnglishMnemonic } from "./englishmnemonic.js"; must be used for a local file instead of import { EnglishMnemonic } from "./englishmnemonic";. TS will not do this for us.

Should the project even stay on the tsc+webpack combination in 2025? Would migrating to Rollup instead be worthwhile, it can handle omitted file extensions when transpiling.

Plus, if a consumer project is running ESM, named imports from a CJS package like cosmjs can be finicky if it was generated by webpack; they added a specific setting for output.library.type just to make webpack-generated modules statically-analyzable, which is the minimum to make it work.

https://webpack.js.org/configuration/output/#type-commonjs-static

And files like:

• karma.conf.js
• jasmine-testrunner.js
• typedoc.js
• webpack.{web,node,demo}.config.js

will need renaming.

[webmaster128]

Didn't Node.js v17.1 and v16.14 add support for import data from './data.json' assert { type: 'json' }; and await import('./data.json', { assert: { type: 'json' } }).

It seems like we have support for JSON imports in modern browsers and Node 20: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import/with#browser_compatibility

[dynst]

Great. Then for the file extension problem, can you just switch to using tsc --noEmit for type-checking followed by using a bundler for generating the build products instead? If it's all bundled into one file, no local file import statements exist so there's no 'file extensions must be used' problem like in #1737.

https://www.npmjs.com/package/esbuild
https://www.npmjs.com/package/rollup
https://www.npmjs.com/package/vite

[webmaster128]

Isn't that what webpack is supposed to do? To be honest I lost track of all those fancy new tools doing more or less the same.

sing a bundler for generating the build products instead

You mean the build and packaging step just produces a single .js file instead of many? Isn't that much harder to navigate in case of debugging when runtime is pointing you to a place in the .js?

[webmaster128]

Also: can't tsc output a single file instead of many?

[dynst]

Webpack is the legacy choice these days, its modern ESM support is very new, immature, and gated behind the experiments boolean.

https://webpack.js.org/configuration/output/#type-modern-module

I think Vite is the choice people gravitate towards now. It uses rollup and esbuild.

https://vite.dev/guide/build.html#library-mode
https://vite.dev/config/build-options.html#build-rollupoptions

It looks like Rollup is capable of producing separate files (chunks), but unlike tsc, it'll helpfully rewrite imports to add omitted file extensions like .js or omitted file names in the case of directory imports, adding /index.js for you.

https://rollupjs.org/configuration-options/#output-preservemodules

Also: can't tsc output a single file instead of many?

True, didn't think of that. Now that I check, a single file may not even be needed given Rollup's import-rewriting features though.

The changes to tsconfig.json in #1737 will be needed either way though!

[webmaster128]

After reading through https://medium.com/@lucksp_22012/3-options-to-compile-typescript-to-js-rollup-tsc-babel-3319977a6946 I think this is really the question of which tool compiles .ts to .js. Webpack is doing something different. I do not yet see super strong reasons to use rollup instead of tsc for this library. We'd probably need two tsc runs to build both CommonJS and ES6 outputs.

[dynst]

Are these not good reasons tsc isn't ideal?

microsoft/TypeScript#49462
microsoft/TypeScript#42151

Rewriting imports would be tedious, and you'd have to make sure never to add a new import that omitted the .js extension by accident ever again. But then even if that's done, to make CJS and ESM coexist, you need to give them different file extensions like .cjs and .mjs but tsc just doesn't support custom file extensions for its output. At all. It's very barebones.

But then Node 20 is the minimum version now and supports require(esm) from CJS code, so is there actually a technical reason to keep CJS output at this point?