Support ES modules with .js extension and package type `module`

[JannesMeyer]

I tried to use the new ES module support from version 3.6.2 with some *.js files that are generated via TypeScript.

The 3.6.2 release notes only mention *.mjs files, but I can't see any information on how to use ES modules with "type": "module" in package.json.

Unfortunately, TypeScript cannot emit *.mjs files yet. There is an open issue on the project, but it seems to be far from resolved: microsoft/TypeScript#18442

As an alternative I am using "type": "module", which is supported by node. However, it doesn't seem to be supported by Jasmine yet.

Node version: 15.0.1
Jasmine version: 3.6.2

[JannesMeyer]

If anyone else stumbles on this issue, here is the workaround that I am using in the meantime until official support arrives:

import glob from 'glob';
import Jasmine from 'jasmine';

glob('./dist/spec/**/*.js', async (err, files) => {
  if (err) { throw err; }
  let jasmine = new Jasmine();
  await Promise.all(files.map(f => import(f)));
  jasmine.execute();
});

I save this as jasmine.js and run it with node jasmine.js.

The only problem with it is that I'm losing the ability to pass CLI arguments and IDE integration suffers a bit. Anyway, Jasmine is amazing for making it so easy to run the tests from arbitrary sources!

[sgravrock]

Loading files as ES modules when type is set to module is on my to-do list. In the meantime, I'd be happy to review a pull requests that adds that functionality.

[JannesMeyer]

I would be happy to work on this, even though I am not familiar with the Jasmine codebase yet.

I suppose the first step is to decide which detection logic to use. As far as I am aware, there is no specific Node API to detect the module type of a file.

Option 1:
Load the package.json file that is relevant to the specDir and read the type property from the object, if it's missing default to CommonJS.

I think it would be easiest to use something like path.join(this.projectBaseDir, 'package.json'). However, this could possibly come with some edge cases. I don't know how likely this is going to occur in the real world, but it's possible for a project to have multiple package.json files.

In theory, each sub-directory could have its own package.json and Node seems to respect the settings of the 'closest' package.json.

Option 2:
Call require(...) inside of a try { } catch { } block. After doing some quick testing, I noticed that the error code is always 'ERR_REQUIRE_ESM' when the module is an ES module. In that case we could fall back to an import(...) call. It would look somewhat like this (pseudo code):

let file = './path/to/some-es-module.js';
try {
  require(file);
} catch (e) {
  if (e.code === 'ERR_REQUIRE_ESM') {
    import(file);
  } else {
    throw e;
  }
}

I found another instance where you already rely on an Error's code value, so hopefully it wouldn't be considered too fragile:

jasmine-npm/lib/jasmine.js

Line 113 in 5fc7e6a

if(configFilePath || e.code != 'MODULE_NOT_FOUND') { throw e; }

Not sure if there is any other way to detect the file type that I am not aware of. I'm still looking at the documentation trying to find alternatives.

[JannesMeyer]

After thinking about this for a little bit I would probably favour the second option, because it allows us to have exactly the same detection algorithm as Node without having to worry about re-implementing it (or tracking possible future changes)

[jehon]

Looking into documentation, I wonder if "import" could be used t import both cjs and esm... Reading it here: https://nodejs.org/api/esm.html#esm_interoperability_with_commonjs

This is the test I did:

main.mjs:
import('./test1.cjs').then(data => console.log("cjs", data));
test1.cjs:

module.exports.default = 1;
module.exports.b = 2;

output
cjs [Module] { b: 2, default: { default: 1, b: 2 } }

But this would be a breaking change, since it requires the main.mjs to be an esm module.

[JannesMeyer]

That's a good point. Node.js seems to allow importing CommonJS modules with import().

Does it even require main.mjs to be an ES module? The import() function is available in CommonJS modules as well. The major drawback would be that Node.js 10 doesn't support it out of the box (you still had to enable --experimental-modules back then).

In conclusion, using that strategy would definitely be a breaking change because it would drop support for Node.js 10. You can see Jasmine's officially supported Node versions here:

jasmine-npm/package.json

Line 35 in 5fc7e6a

"node": "^10 || ^12 || ^14"

Btw, not sure if you noticed it, but I had already created pull request #173 which uses require() and then falls back to import() (it's linked right above your comment). This is my first pull request for this project and I am hoping to get some feedback from the maintainers.

[frank-dspeed]

i need to point out !!!! that no detection of the cases is needed

Node Top Level Import allows Import of CJS and ESM

import 'my.cjs'

so you can always go for Top Level Import no dynamic import needed. And also other big news Node Now got Top Level Await also so you could do await import() but as sayed before general top level import for everything is fine already

[JannesMeyer]

That’s correct, but in this case we are talking about a situation where:

1. Maintaining compatibility with Node 10 is desired
2. The filename is in a variable

[sgravrock]

I spent some time both experimenting with Node's actual behavior and digging into the spec a bit. Since ES modules and CommonJS modules have different semantics, I wanted to answer two questions:

1. Will switching from require('./foo') to import('./foo.js') change the behavior of the code in foo.js?
2. How reliable is the answer to the first question? Can we count on it to not change, or might it be an accident of how Node currently implements dynamic import?

For the first question, I used a few different approaches to try to figure out what kind of module the file was being loaded as. For each approach I tried loading the module via require, dynamic import, and static import. In all cases the module’s file extension was .js and I did not have ”type”: “module” set in package.json. The results were the same for Node 10 with --experimental-modules, Node 12, and Node 14:

Loading method	exports supported?	Defaults to strict mode?	module.parent defined?
require	no	no	yes
dynamic import	no	no	no
static import	no	no	no

That's not a complete exploration of all the possible differences. But it's enough to show that loading a CommonJS module via import() can change its behavior. It's possible that there are more important differences than whether or not module.parent is defined.

For the second question, I can't find anything in either the Node documentation or the dynamic import proposal that discusses using either static or dynamic import to load CommonJS modules. It could be that the Node team thought carefully about what should happen when a CommonJS module is imported and made a decision that they plan to stick with, or it could be that the current behavior is an accidental result of how things are implemented. In any case, I don’t see anything in the draft proposal that prevents future versions of Node from either defaulting to ES module semantics or disallowing import() of CommonJS modules altogether. (If anyone has a resource that says otherwise, please post a link. I certainly could have missed something.)

So where does that leave us?

• TypeScript users can’t use ES modules today.
• Having to use .mjs to get ES module semantics is likely to be more and more unpopular as time goes on and more users switch to pure ES module setups.
• We can’t use try/catch to decide how to load a module, because a module might load without throwing but behave differently when loaded the wrong way.
• Defaulting to import() is a breaking change, so we can’t do it in 3.x.
• We’ll still need to support require() for the foreseeable future, both for users who need full CommonJS semantics and in case the behavior of importing a CommonJS module changes.

We could check the package type as described in the title of this issue. That’s slightly fiddly (we’d need to make sure we check the correct package.json in cases where the imported module is in a different package). But it should be pretty robust, and it should just work without users having to do any configuration.

Thoughts?

[frank-dspeed]

@sgravrock even if this is the wrong topic for that but we in nodejs got import { createRequire } from 'module'

and can this way init a require = createRequrire(import.,meta.url);

we are also able to change the behavior of all loaders via hooks and we are able to write a own require shim via dynamic import like

async function require(path) {
	let _module = window.module;
	window.module = {};
	await import(path);
	let exports = module.exports;
	window.module = _module; // restore global
	return exports;
}

(async () => { // top-level await cannot come soon enough…

let x = await require("https://cdn.jsdelivr.net/gh/x/lib/index.js");

})();

hope that helps you a bit while this is true not everything we have but we can not write books in so short time

[jehon]

@sgravrock : here is the doc to the dynamic import in node JS: https://nodejs.org/dist/latest-v14.x/docs/api/esm.html#esm_import_expressions

[niyarlatotep]

@sgravrock sorry for a stupid question but cant' we just simply read package.json from current working dir?