module: add support for `node:`‑prefixed `require(…)` calls #37246

ExE-Boss · 2021-02-06T11:19:29Z

Fixes: #36098 (applies only to commit 1d8c8b7)
Refs: #37178 (applies only to commit b0fbc91)

aduh95 · 2021-02-06T13:27:25Z

You read my mind, I was precisely working on this. I noticed it produces an arguably better error when trying to load internal modules:

$ node -p "require('internal/test/binding')"
Error: Cannot find module 'internal/test/binding'
$ node -p "require('node:internal/test/binding')"
Error: Should not compile internal/test/binding for public use

Can you add tests in test/parallel/test-require-node-prefix.js to make sure requiring internal modules still throws please?

lib/internal/modules/cjs/loader.js

doc/api/esm.md

benjamingr · 2021-02-06T15:54:57Z

cc @nodejs/modules

test/parallel/test-require-node-prefix.js

bmeck · 2021-02-06T16:41:54Z

test/parallel/test-require-node-prefix.js

+  );
+}
+
+// `node:`-prefixed `require(...)` calls bypass the require cache:


this differs from other builtins, I love it, but want to be sure it is intentional.

this actually seems like an issue - it means someone who replaces a builtin intentionally can’t replace one of these (altho presumably mutating a builtin would be visible in both). That could cause an issue where someone wants to use a package to instrument or lock down fs (and thus, policies are not an ergonomic option, unless I’m misunderstanding how policies work), but accidentally leaves a gaping security hole when an attacker requires node:fs.

policies should intercept before any of the resolution helpers even get called

node/lib/internal/modules/cjs/helpers.js

Line 52 in 36cc0ee

if (redirects) {

, if any module redirection is in place (including complete shutdown) then it would never call out to module.require and get here without some kind of opt-in to the behavior (like dependencies:true).

At the very least this needs documentation and probably a strong indication people using policies + instrumenting fs now need to update the policy.

My point was that policies aren’t currently required to ensure this for CJS - a package (not the app itself) can do it. Forcing policies to be the mechanism means packages are no longer capable of abstracting this for users.

Totally understood. Given that policies are the superior mechanism for app developers to lock things down, what's the benefit of adding special, inconsistent, cache-bypassing behavior for require with a node: prefix?

@ljharb i have no strong opinion on if we should diverge, but cache first behavior of CJS isn't shared by ESM and isn't robust against things like core destructuring the original impls. My like is just that it isn't a confusing situation like with fs/.

but cache first behavior of CJS isn't shared by ESM and isn't robust against things like core destructuring the original impls.

This is demonstrated by:

const fs = require('fs'); const fakeModule = {}; require.cache.fs = { exports: fakeModule }; assert.strictEqual((await import('fs')).default, fs);

indeed, but that doesn’t mean it’s a good idea to make CJS inconsistent with itself.

I don't think CJS is able to be claimed as consistent with itself since in general the natives don't normally populate the cache. I think it was just ad-hoc written together and the cache behavior is evolutionary not intentional or well understood (even by me).

lib/internal/modules/cjs/helpers.js

guybedford · 2021-02-08T09:17:00Z

lib/internal/modules/cjs/loader.js

@@ -837,7 +850,8 @@ Module._load = function(request, parent, isMain) {
 };

 Module._resolveFilename = function(request, parent, isMain, options) {
-  if (NativeModule.canBeRequiredByUsers(request)) {
+  if (StringPrototypeStartsWith(request, 'node:') ||
+      NativeModule.canBeRequiredByUsers(request)) {


This bifurcation will have implications for mocking, but I can appreciate the benefit too.

I'd still prefer a more convergent path here eg, to return node:fs for both node:fs and fs inputs, and then to still populate a cache['fs'] entry in the node:fs case as the same object for backwards compatibility. Such a change would hopefully be mostly compatible but would probably need to be a separate major nontheless. Worth thinking about at least.

Existing mocking tools (such as Jest) provide their own require(…) implementation, since they need to be able to bypass the mocked module using jest.requireActual(…).

lib/internal/modules/esm/translators.js

doc/api/esm.md

lib/internal/modules/esm/translators.js

guybedford · 2021-02-08T14:00:33Z

lib/internal/modules/cjs/loader.js

+  if (StringPrototypeStartsWith(filename, 'node:')) {
+    // Slice 'node:' prefix
+    const id = StringPrototypeSlice(filename, 5);
+
+    const module = loadNativeModule(id, request);
+    if (!module?.canBeRequiredByUsers) {
+      throw new ERR_UNKNOWN_BUILTIN_MODULE(filename);
+    }
+
+    return module.exports;
+  }


What is the motivation for short-circuiting the cache here, something that has never been done previously for native modules?

I don't see why this scheme should be any different to any other in this regard.

On this topic, should we throw or defer to cache on unknown builtins?

doc/api/modules.md

aduh95

LGTM

benjamingr

Just making sure #37246 (comment) is addressed before landing :) (namely the caveat) - I'm appreciative of the work put in, thanks!

Also would like more @nodejs/modules and @nodejs/tsc eyes on this.

(The actual changes LGTM)

ExE-Boss force-pushed the lib/module/require-node-url branch from 8615225 to 4f61960 Compare February 6, 2021 11:21

ExE-Boss force-pushed the lib/module/require-node-url branch from e740b61 to 320ef5e Compare February 6, 2021 13:57

RaisinTen reviewed Feb 6, 2021

View reviewed changes

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

RaisinTen reviewed Feb 6, 2021

View reviewed changes

doc/api/esm.md Outdated Show resolved Hide resolved

bmeck reviewed Feb 6, 2021

View reviewed changes

test/parallel/test-require-node-prefix.js Outdated Show resolved Hide resolved

bmeck reviewed Feb 6, 2021

View reviewed changes

bmeck approved these changes Feb 6, 2021

View reviewed changes

jasnell added the module Issues and PRs related to the module subsystem. label Feb 6, 2021

ExE-Boss force-pushed the lib/module/require-node-url branch from 3634cd4 to d196d45 Compare February 6, 2021 19:37

guybedford reviewed Feb 8, 2021

View reviewed changes

aduh95 requested changes Feb 8, 2021

View reviewed changes

doc/api/esm.md Outdated Show resolved Hide resolved

ExE-Boss force-pushed the lib/module/require-node-url branch from 10b08bf to 8564814 Compare February 8, 2021 13:11

ExE-Boss requested a review from aduh95 February 8, 2021 13:12

guybedford reviewed Feb 8, 2021

View reviewed changes

aduh95 reviewed Feb 8, 2021

View reviewed changes

doc/api/modules.md Outdated Show resolved Hide resolved

doc/api/modules.md Outdated Show resolved Hide resolved

aduh95 mentioned this pull request Feb 9, 2021

module: disable package self resolution when name contains a colon #37290

Closed

ExE-Boss mentioned this pull request Feb 12, 2021

[node] Add node:‑prefixed module aliases DefinitelyTyped/DefinitelyTyped#51107

Merged

8 tasks

This was referenced Feb 15, 2021

feat: add support for additional node.js builtin modules webpack/webpack#12181

Merged

feat: add node: prefixed modules webpack/webpack#12693

Merged

ExE-Boss force-pushed the lib/module/require-node-url branch from 8564814 to b0fbc91 Compare March 5, 2021 10:48

ExE-Boss requested review from aduh95, RaisinTen, benjamingr, guybedford, ljharb and bmeck March 5, 2021 10:50

aduh95 approved these changes Mar 5, 2021

View reviewed changes

benjamingr requested changes Mar 5, 2021

View reviewed changes

targos mentioned this pull request Sep 4, 2021

v14.18.0 release proposal #39990

Merged

kachkaev mentioned this pull request Oct 3, 2021

Usage of node:path is incompatible with declared engines sindresorhus/find-up#55

Closed

mortynex mentioned this pull request Oct 29, 2021

Support for "node:" protocol for importing builtin nodejs modules stackblitz/webcontainer-core#369

Closed

LinusU mentioned this pull request Dec 2, 2021

node: Add support for node: prefix on v14 DefinitelyTyped/DefinitelyTyped#57498

Merged

8 tasks

mattcollier mentioned this pull request Jan 25, 2022

Use new node: URLs where applicable? digitalbazaar/bedrock#96

Closed

dominics mentioned this pull request Feb 3, 2022

Feature: support for node: prefixed URL-style imports FormidableLabs/trace-deps#75

Closed

trentm mentioned this pull request Jul 8, 2022

core modules imported with require('node:...') are not instrumented elastic/apm-agent-nodejs#2816

Closed

DeeDeeG mentioned this pull request Jul 11, 2022

doc: revert anachronistic 'node:' prefixed module require()s in API history notes #43768

Merged

cristiand391 mentioned this pull request Sep 28, 2022

Any sfdx command fails after npm update forcedotcom/cli#1728

Closed

titanism mentioned this pull request Feb 10, 2023

[fix] preview-email is failing due to require('node:child_process') is not getting processed on mac intel forwardemail/test-preview-emails-cross-browsers-ios-simulator-nodejs-javascript#35

Closed

3 tasks

fengmk2 mentioned this pull request Mar 25, 2023

feat: build target up to ES2022 node-modules/urllib#439

Merged

ExE-Boss mentioned this pull request Sep 16, 2023

Support node: prefix for CJS dependencies parcel-bundler/parcel#9244

Merged

Phillip9587 mentioned this pull request Jan 14, 2025

refactor: prefix built-in node module imports expressjs/body-parser#573

Merged

Uh oh!

module: add support for node:‑prefixed require(…) calls #37246

module: add support for node:‑prefixed require(…) calls #37246

Uh oh!

Conversation

ExE-Boss commented Feb 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aduh95 commented Feb 6, 2021

Uh oh!

Uh oh!

Uh oh!

benjamingr commented Feb 6, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ExE-Boss Feb 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aduh95 left a comment

Choose a reason for hiding this comment

Uh oh!

benjamingr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

module: add support for `node:`‑prefixed `require(…)` calls #37246

module: add support for `node:`‑prefixed `require(…)` calls #37246

ExE-Boss commented Feb 6, 2021 •

edited

Loading

ExE-Boss Feb 8, 2021 •

edited

Loading