module: self resolve bug fixes and esm ordering

[guybedford]

This resolves the self-resolve bugs #30633 and #30602. In addition the ESM implementation of self-resolve is slightly adjusted to apply before the node_modules lookup for better encapsulation, while the CJS implementation is left after the node_modules lookup for backwards compatibility.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[guybedford]

//cc @nodejs/modules-active-members

[MylesBorins]

I'm not sure that we should change the resolve order between ESM + CJS. This could result in unexpected behavior.

[guybedford]

@MylesBorins the alternatives here are:

1. Allow self-resolve to always be intercepted by node_modules lookups in ES modules. My concern with this is that self-resolve is no longer reliable but dependent on the surrounding node_modules structures, which doesn't provide the same absolute guarantee of self resolution not having to even check the file system.
2. Return to the great sigil debate of not using the package name string, since we have to support the node_modules lookups in the CommonJS implementation for the package name for backwards compatibility in existing CommonJS workflows.

I find this is enough of an edge case that the compat doesn't concern me personally, while the full encapsulation going forward I do think is important. Happy either way here though.

[weswigham]

This could result in unexpected behavior.

It'll behave strangely any time a package gets installed in its own node_modules folder (as the esm resolver will resolve to the local copy, while the cjs resolver will resolve to the node_modules copy). This used to happen when developing typescript, as typescript had a devDependency of tslint which had a dependency of typescript (which npm then flattens), so is a situation that happens in the real world. They probably shouldn't diverge.

[weswigham]

which doesn't provide the same absolute guarantee of self resolution not having to even check the file system.

It needs to check the fs to look for the enclosing package.jsons already - the node_modules folder state isn't that much more, once you're already hitting disk.

W.R.T. the sigil stuff, I, at least, forsee writing a

{
  "name": "~",
  "type": "module",
  "private": true
}

package.json file in my src folders, under my primary package.json. So you could say that the current implementation just allows choice of sigil, if you're so inclined. So if people are taking issue with the "meaning" of a sigil, I'm not sure the argument holds once this kind of "custom sigil" is possible. it may be better to just pick one, any one, just to save work in both resolvers.

[guybedford]

@weswigham it will check node_modules/[name] folders all the way to the root of the file system, is the issue.

When applying tooling for eg import maps generation, a root-level node_modules install affecting self-resolution isn't ideal.

[weswigham]

it will check node_modules/[name] folders all the way to the root of the file system, is the issue.

Are the discovered folder contents cached, like how package.json results are cached? If so, it shouldn't be too bad, since resolution will already be forced to generate that list for literally any other non-relative specifier.

[jkrems]

Return to the great sigil debate of not using the package name string, since we have to support the node_modules lookups in the CommonJS implementation for the package name for backwards compatibility in existing CommonJS workflows.

Using a sigil does unfortunately not really change this: Any sigil would be a valid sub-directory of node_modules and we're back to the same order concern for require. Afaik we haven't found any syntax that is backwards compatible with existing CJS resolution without checking node_modules first. The only solution would be an explicit opt-in like a useSelfReference in package.json which would work for both name and sigil.

[weswigham]

Afaik we haven't found any syntax that is both backwards compatible with existing CJS resolution without checking node_modules first.

I think the idea behind picking a sigil is that you can make the cjs loader pick the sigil first, and it's probably not a really breaking change, since you generally can't use sigils as a package name (especially if they can't be package names on npm); meanwhile if using the package's actual name, if you change the behavior, you change a scenario that actually crops up in real installations.

[MylesBorins]

Could we potentially tease the ordering change out of this PR and land the fixes? Or are they entangled?

[guybedford]

Further to the discussion from the meeting today I've gone ahead and pushed the following change:

• If the package scope has an "exports" property, then self resolution applies before any further package resolution lookup in node_modules
• If the package scope does not have an "exports" property, then the self resolution always only applies after trying the node_modules package resolution. self resolution does not apply.
• The implementation of the above is identical between CJS and ESM, bringing them back to parity.

The approach for this is basically to inject the self resolve method twice - once before and once after package resolution, with a flag to indicate if it must check for exports.

@jkrems does that alleviate your original concerns here? Would appreciate re-review when you can.

[jkrems]

As discussed out-of-band: LGTM in general although I'd like to see the 2nd attempt removed. Having a safe way of using this feature (set exports) and a default way that isn't as safe (post-node_modules lookup) seems dangerous. Thanks for working through this!

[guybedford]

In line with the suggestion by @jkrems I've made self-resolution always-opt-in on "exports" being present for both.

[jkrems]

Nice! I think there's some traces left in the algorithm of the previous variant.

[ljharb]

Wait, does this mean that self-export only works inside a package using "exports"?

[guybedford]

Wait, does this mean that self-export only works inside a package using "exports"?

As added in ff11612, yes. The first iteration before this commit was a bit messier, but allowed it without exports too. The diff is in #31009 (comment) to see what I mean.

Since "exports" is the new main which also offers encapsulation this approach seems to make the most sense. I'd advise checking the meeting notes on the discussion to avoid rehashing too much here, but happy to discuss further.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/27905/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/28010/

[guybedford]

Landed in 8a96d05.