module: support ES modules without file extension within module scope

[LiviaMedeiros]

This makes interpreting extensionless local files symmetrical in modules and commonjs scopes: they are treated the same as .js.

Earlier his feature was removed in favor of eventually supporting extensionless Wasm. Turns out, extensionless Wasm support can be added without making extensionless ES modules not work.

semver-minor PRs that contain new features and should be released in the next minor version. with --experimental-extensionless-modules flag unless there are objections.

Either semver-major PRs that contain breaking changes and should be released in the next major version. follow-up that unflags this or flagless alternative that can be landed as is: #49629

Potential follow-up that adds extensionless Wasm support: #49540 (d78a9da, to be precise).

Refs: #49431

[nodejs-github-bot]

Review requested:

• @nodejs/loaders

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53783/

[GeoffreyBooth]

Hi, thanks for this @LiviaMedeiros. A few things that I think are needed:

• If this is unflagged, it’s might be semver-major; I’m not sure. Someone could have code that tries to run an extensionless file, and if it catches an exception then it pipes it through a transpiler to run. This is a bit more dangerous than “it errored before, it doesn’t now” because there are various ways via other tools that it could have make it work before. My preference would be to flag this, but rather than create a new flag specifically for this we should include it in the larger new flag we’re discussing in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431.

• This needs to support extensionless Wasm entries, at least in the module scope. I don’t think we can just assume that that can come later, because we need to make sure that we have an approach for supporting extensionless Wasm that finds consensus (or passes a TSC vote). It seems like we have only one option at the moment, the magic bytes detection, so maybe try implementing that and see if you can get it to work? You might be able to steal tests and some other code from esm: add package type wasm #31388 (but please don’t take the “new type field” stuff).

[ljharb]

type module controls .js, but there’s ways to use ESM without type module - it shouldn’t be overloaded to also control extensionless, and there needs to be a way to use extensionless ESM without type module imo.

[LiviaMedeiros]

Please elaborate if you mean a need to reword this or suggest changes to the implementation.
The type controls how any JS code is interpreted. The exceptions are files with explicitly typed extensions (.cjs, .mjs, maybe future formats) and with unknown extensions (these should throw to ensure that it breakage is minimized if we add future formats).

[ljharb]

I’m suggesting an additional package.json key to control extensionless. “type” only controls how .js is interpreted, not “any” js code.

[LiviaMedeiros]

Someone could have code that tries to run an extensionless file, and if it catches an exception then it pipes it through a transpiler to run. This is a bit more dangerous than “it errored before, it doesn’t now” because there are various ways via other tools that it could have make it work before.

Changing to semver-major PRs that contain breaking changes and should be released in the next major version. . I guess if there will be need in this feature for earlier releases, we can backport it with flag.

This needs to support extensionless Wasm entries, at least in the module scope.

No, this must come in separate PR. I'll make a minimal follow-up implementation for extensionless module/Wasm.

[GeoffreyBooth]

No, this must come in separate PR. I’ll make a minimal follow-up implementation for extensionless module/Wasm.

I don’t think we can land this without some consensus that the other can land. If you want to make two PRs and they both get approvals and then a third PR combines the two and can land, or some other process, that’s fine; but unless we know that landing this won’t foreclose the possibility of extensionless Wasm in the future, or we have consensus that we’re okay with potentially not supporting extensionless Wasm if no detection algorithm can be found, then I don’t think we can land this on its own.

[GeoffreyBooth]

This is referring to import, not entry points, and shouldn’t be removed. I think the docs for this PR would more or less match the reverse of the changes in #31415.

[GeoffreyBooth]

I think we’re aiming to change how extensionless files are handled as entry points, not in general from any import.

[LiviaMedeiros]

How come? Both ESM with other protocols and CJS have no problems with extensionless imports.

[aduh95]

It would be very weird to have a file parsed as ESM only some of the time, I much prefer the current approach.

[GeoffreyBooth]

In the original implementation before it was limited, we supported extensionless per "type" only for entry points, not for any import, per #31415 (comment). We’ve never supported extensionless for import. I could search around for why there was the original limitation to make this only for entry points, but I’d bet there was a good reason. There’s also not really a use case for import of extensionless files.

[aduh95]

We’ve never supported extensionless for import

In the comment you just linked, you say:

support for extensionless files in import was added only last month, in #31021.

[GeoffreyBooth]

Okay so we didn’t never support it, but it wasn’t part of the original implementation. It was briefly added and then reverted. Probably because of this: #31021 (review)

One of the major features of the ES module resolver is being able to support new file extensions in future, which is reliant on the fact that we always throw for unknown file extensions.

Giving special treatment to isMain was how we did this while ensuring compatibility with existing bins.

Happy to flesh this out further, but I don’t think we should lose that extension property for the ESM resolver.

[GeoffreyBooth]

I dug through a lot of old issues and consulted @guybedford to try to rediscover why we limited extensionless support to entry points in the initial implementation, and why we removed even that in #31415. Basically, we wanted to preserve the ability for future extensionless Wasm entry points. And I think we still do; that goal hasn’t changed, and #49540 is a viable way to achieve that goal. I think years ago we weren’t aware of or really thinking about the “disambiguate based on magic bytes” solution, and so we just locked down the design space in the hope that we’d figure something out eventually. And I think the magic bytes approach is that something, and so it can open up the ability for both extensionless ESM entry points and extensionless Wasm entry points.

As for import of extensionless files, there would be the similar question of how to disambiguate an import of an extensionless JavaScript file from an extensionless Wasm file. The simplest solution is probably to just take same approach that we use for entry points, looking at magic bytes. Currently import of Wasm is gated behind the separate flag --experimental-wasm-modules, so it doesn’t matter all that much.

But I think we can support import of extensionless JavaScript. JavaScript can be the default for an import statement, where “extensionless = JavaScript with the format determined by type” for import like for entry point, with other potential formats needing a way to disambiguate. Those new types would either need explicit extensions (the most likely and obvious disambiguation) or they would need mandatory import attributes (like JSON) or they would need magic bytes; some way to disambiguate them from JavaScript. But I think we can unlock import of extensionless JavaScript without much risk of that prohibiting future enhancements.

[LiviaMedeiros]

I don’t think we can land this without some consensus that the other can land.

We should land this when we have consensus on this.

d78a9da has Wasm.

[GeoffreyBooth]

We should land this when we have consensus on this.

We can’t land this on its own without either consensus on an approach for Wasm, or consensus on agreeing that we don’t care if we foreclose the possibility of extensionless Wasm in the future. #31415 set the precedent that we want to preserve that possibility. We can certainly revisit that decision, but until we do I’m assuming that it still stands.

A “go big” PR would avoid that problem, since it would answer the question of how to handle extensionless Wasm. It would also avoid the problem of adding several flags, since it could combine multiple behaviors behind just one. @LiviaMedeiros Do you mind attempting a single PR that includes everything? And if it can’t land for some reason then we can revisit.

To copy from #49432 (comment), here’s what I think the big PR should include:

• When a .js file is outside of any defined package.json scope, because there are no package.json files in its folder or any folders above it, it will be treated as ESM JavaScript.

• When a .js file is in a package scope where its nearest parent package.json lacks a "type" field, it will be interpreted per the conditions we establish in Discussion: In an ESM-first mode, how should a package.json file with no type field be handled? #49494: when the package scope is under a folder named node_modules, the file will be treated as CommonJS; otherwise it will be treated as ESM.

• When an extensionless file is outside of any defined package.json scope, it can be run as an entry point per the conditions we establish in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431: if the file begins with the Wasm magic bytes, it will be evaluated as Wasm, and likewise for any other future file types with defined headers that don’t conflict with JavaScript; otherwise it will be evaluated as ESM JavaScript. This support for extensionless files would be limited to entry points; they still would not be allowed in import.

• The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed.

• Input from STDIN or --eval or --print would be treated as ESM unless --input-type=commonjs is passed. Or in other words, the default value of --input-type would flip from commonjs to module.

And this should all get put behind a flag. In #49541 I proposed --experimental-type=module / --experimental-type=commonjs, so we can use that for now in the PR and change it later if a better alternative emerges.

What do you think?

[LiviaMedeiros]

No, this is a separate feature that should be there no matter what flags are landed or enabled.

This also should be separated from Wasm support, so we can easily backport it, if needed.

We can’t land this on its own without either consensus on an approach for Wasm, or consensus on agreeing that we don’t care if we foreclose the possibility of extensionless Wasm in the future.

What is not in consensus about Wasm?

[GeoffreyBooth]

No, this is a separate feature that should be there no matter what flags are landed or enabled.

This also should be separated from Wasm support, so we can easily backport it, if needed.

I don’t think these statements are quite right. In my opinion:

• This needs to land behind an experimental flag, both because it has a potentially large impact and so that we can land it on 20.x and backport it.

• If we’re going to create a new flag I’d prefer it encompasses all of the “ESM-first” behaviors rather than us creating a set of flags to achieve all of them. One flag instead of many just feels like better UX.

• It can’t be separated from Wasm support because module: drop support for extensionless files in ESM #31415 established a consensus that we don’t want to support extensionless ESM entry points without also supporting extensionless Wasm entry points. But the inverse is also true: we want extensionless ESM entry points with extensionless Wasm entry points, and across the two PRs you’ve implemented a solution for doing so. Maybe people might reevaluate their positions in light of how much time has passed, but just taking the earlier comments at face value, what should be acceptable to everyone is if we land everything together. If someone objects to that then there’s no possible PR that fits everyone’s requirements and we’ll need to negotiate a compromise, but right now that doesn’t seem necessary to me: we can make a single PR that makes everyone happy.

@LiviaMedeiros do you have an objection to potentially doing everything in one PR? I was assuming not, since #49295 was a big PR with several ESM-first features grouped together. I can understand wanting to land separate PRs as a process concern, to maybe divide up the work and ease reviewing and such; we can do that and add “don’t land” labels so that none of the PRs get released until they’re all landed (and if any get blocked or abandoned, we’d have to revert the earlier ones that landed on main based on an assumption that everything would land). But based on the various PRs offered so far, a PR that puts them all together seems like it wouldn’t be so large as to be unwieldy or hard to review, and I think the issues we’ve opened so far make clear that there is likely to be a path forward for it. I or someone else can take over this task if you don’t want to do it, but since you’ve been the driving force of this so far I thought you would probably want to lead it.

[LiviaMedeiros]

Given that this is semver-major, I don't see any need to put it under an experimental flag.

If the concern is about potential breakage, let's try CITGM run on this.
This PR changes behaviour of case that currently throws, so it can't be presumed as breaking unless there are concrete examples of that (and if they are significant enough to justify flag despite semver-major).

It can't be combined with any flags that change type interpretation. Otherwise it would break in all CJS-first projects in ecosystem.

[GeoffreyBooth]

The idea of mixing multiple standalone features in single PR under single flag, so people who need particular features will have to break something else - I object that.

One of the “larger” features is supporting extensionless files as ESM where there is no package.json file. I don’t know what the use case is for supporting extensionless files only within package scopes. This smaller feature is so small that I’m not sure if it has merit to land on its own, or that it deserves its own flag. If anything it’s a footgun because it enables something similar to what people want but isn’t what people want.

We don’t need to split it out; I was suggesting that to be charitable. We could just land everything together in the big flag.

This PR is already a part of fixing problems listed in #49432. The idea of mixing multiple standalone features in single PR under single flag, so people who need particular features will have to break something else - I object that.

This is the approach that I think is most likely to get majority support. We’ll discuss tomorrow in nodejs/loaders#160 if you’d like to attend.

If individual components have merit on their own, they can be split out under separate flags; though if anything it might make more sense to further consolidate, like the suggestion to make --input-type control not just STDIN/eval but all ESM default behaviors.

[GeoffreyBooth]

I don’t think I was clear enough in my previous objection, so I’ll spell it out more here. I’m objecting to this based on all three of the bullet points from #49531 (comment):

1. I think this needs to land under an experimental flag at first.
2. That flag should encompass all the “default as ESM” behaviors we’re considering, not just extensionless files in a module scope.
3. That flag must include a solution for extensionless Wasm, such as magic bytes.

Once the above lands, we could consider creating a separate flag just for extensionless files as ESM (or even extensionless files within module scopes as ESM, though I don’t see the point of that) but the above needs to land first so that we’re sure we’ve solved the extensionless Wasm use case; and so that users have a single flag to handle extensionless files as ESM both within package scopes and without.

[mcollina]

I agree with @GeoffreyBooth.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53903/

[LiviaMedeiros]

I strongly disagree and I find a PR block that doesn't provide a reason why this PR can't land and instead proposes unfeasible alternative to be unreasonable.
Since the block persists without addressing the objections to the alternative, adding tsc-agenda Issues and PRs to discuss during the meetings of the TSC. .

To summarize the main problem of alternative, the effect of #49432 can work only outside of node_modules and only with addition of changing at least how default type is interpreted, what is default --input-type. Even after the proposed "flipping defaults", there would be a backward-compatibility flag like --type=commonjs that undoes it.

There is no reason to not allow developers to use extensionless ES modules inside of node_modules, in CJS-first projects that do not want default type to change, and in future-legacy projects that have to preserve the current default after the flip.

Additionally, if we combine the flags, the same problem will occur to extensionless Wasm that directly depends on this feature.

This PR does solve these problems, and opens path for both extensionless Wasm and extensionless ESM outside of pjson scopes, effectively achieving the same goal. Without flags with side effects, limitations, nor making CJS projects less usable.

Semverity is up to TSC; it also can land without runtime flag (#49629 is unflagged version of this).

[mcollina]

If we do allow extensionless imports behind the flag proposed in #49432, it will introduce huge impact on all CJS projects.
Before the "flipping default", using this feature in flagged projects would throw in dependents that don't use the flag, simply making it unusable in the wild.
After the "flipping default", every CJS project that uses --type=commonjs for compatibility with current defaults would throw if its dependencies use this as unflagged out-of-the-box feature.

I think we have covered this enough in #49494 not to be the case. None of the new flags will affect node_modules, which will be left untouched. That's the prerequisite of all this initiative: not break the ecosystem.

I prefer this change to happen alongside the change to the type default inside package.json and I think those should be controlled by the same flag to minimize the actual testing maintainers should do, e.g. "please verify everything works with node --esm".

I don't understand why the above is unreasonable.

[LiviaMedeiros]

I think we have covered this enough in #49494 not to be the case. None of the new flags will affect node_modules, which will be left untouched. That's the prerequisite of all this initiative: not break the ecosystem.

That's amazing, I'll update the summary assuming this and #49432 (comment).
There were other options proposed in that issue, but since this is the preferred one, the question becomes simpler: "why this feature should NOT be available within node_modules, and kept behind runtime flag that also enables unrelated effects?"

I prefer this change to happen alongside the change to the type default inside package.json and I think those should be controlled by the same flag to minimize the actual testing maintainers should do, e.g. "please verify everything works with node --esm".

We usually don't group multiple different flags in one to minimize testing, do we? Otherwise we could already have some --experimental-everything that enables all bleeding edge features at once.
This and changing type are completely independent features, just like --experimental-network-imports and --experimental-wasm-modules.

For the record, this one can be landed without any flag, unless there is a clear reason to not. So far, there was no example of potential breakage that would make it worse than semver-major. This feature doesn't remove or change anything; it only makes loading a module work in condition that otherwise throws ERR_UNKNOWN_FILE_EXTENSION. I'll open unflagged version separately to make it less confusing.

[GeoffreyBooth]

None of the new flags will affect node_modules, which will be left untouched. That’s the prerequisite of all this initiative: not break the ecosystem.

Well specifically, redefining how type behaves is something we’re excluding from packages underneath a node_modules folder. It wasn’t proposed to restrict how extensionless files under node_modules are interpreted, because as @LiviaMedeiros points out, doing so wouldn’t be a breaking change.

The reason to group them together is that from the user perspective, extensionless files are extensionless files; it’s not a separate “feature” how extensionless files are treated:

• Loose, outside of any scope defined by a package.json
• In the user’s app
• In a package in a dependency of the user’s app, under node_modules

I agree that we can land the “how extensionless files are treated within a module scope” part on its own and it wouldn’t be a breaking change. But if that landed before the user had some way of affecting extensionless files outside of a module scope, it would feel very frustrating. The extensionless files that users want to influence are the loose ones. It would be better to first ship a flag that lets them change all extensionless files, wherever they are, and then spin off from that the “within module scope” part to land on its own unflagged now that there’s already a way to influence loose ones.

We usually don’t group multiple different flags in one to minimize testing, do we?

We haven’t traditionally tried to control the scope of flags. We had --experimental-modules for years to enable the entire ESM implementation. I view this new “ESM-first” flag as in that vein. In some ways it’s easier to have one big flag than many little ones, as there are fewer combinations of toggles to test.

[LiviaMedeiros]

I agree that we can land the “how extensionless files are treated within a module scope” part on its own and it wouldn’t be a breaking change. But if that landed before the user had some way of affecting extensionless files outside of a module scope, it would feel very frustrating. The extensionless files that users want to influence are the loose ones.

From a user perspective with pjsonless extensionless scripts, I absolutely can't feel very bad about extensionless files outside of scopes for a very simple reason: they already work! But as CJS, which is completely understandable.

If, as user, I really want my loose extensionless scripts to work, I would be able to do it with this PR by doing echo '{"type":"module"}' > ~/my_esm_utils_that_are_in_PATH/package.json and to be happy.

But what will really make me feel very frustrating, is this:
I type node --type=module ~/my_esm_tools/my_tool and it throws random syntax error because it wants to import /root/.our_common_functions/func_a.js but this file is CJS.
Then I add ~/my_esm_tools/package.json with {"type":"module"} and naively expect it to work because why wouldn't it?
Then I type node ~/my_esm_tools/my_tool and get ERR_UNKNOWN_FILE_EXTENSION in my face.

This kind of scenario I would absolutely hate as user, because I can not comprehend WHY adding pjson doesn't help, and I have no choice than to either make my extensionless scripts not so extensionless, or make my own local copy of func_a.js in my own new local directory that must have {"type":"commonjs"} pjson.
Combining unrelated features under single flag will always bring situations where one feature is desirable and other is not, and I have absolutely no idea how to explain a user why it works like this if we ship it combined and they open issue with similar situation.

Also, as user I would be frustrated from just reading documentation of a flag that instead of just flipping default type has a long list of side effects, especially if I search for a particular feature that happened to be a part of it. Like, what kind of technical issue happened that Node.js couldn't have this and that effect by default?
I.e. overall as user I would prefer --type=module to be as close to "just flip default type" as possible: the fewer nuances it has, the less confusing it is, the better UX I get.

[RafaelGSS]

It's missing the stability: 1.1 Active Development

[mcollina]

As discussed in today's TSC meeting, I'm adding dont-land-on-v20 and dont-land-on-v18 labels.

[GeoffreyBooth]

Closing now that #49974 has landed.