module: implement NODE_COMPILE_CACHE for automatic on-disk code caching

[joyeecheung]

This is the initial PR for implementing the idea proposed in #47472. It takes inspiration from ccache, the v8-compile-cache npm package (but faster, supports ESM and replaces CJS loader moneky-patching in user land) and v8 code caching in Blink. The first commit is backported from https://chromium-review.googlesource.com/c/v8/v8/+/5401780 which has already landed in the upstream (this is required for import() to work in deserialized code). For more motivation and background of this PR, check out #47472

Local numbers:

❯ hyperfine "./out/Release/node test/fixtures/snapshot/typescript.js" --warmup 5
Benchmark 1: ./out/Release/node test/fixtures/snapshot/typescript.js
  Time (mean ± σ):     124.3 ms ±  13.2 ms    [User: 109.6 ms, System: 11.6 ms]
  Range (min … max):   116.1 ms … 170.2 ms    21 runs

❯ export NODE_COMPILE_CACHE=/tmp/.node_compile_cache
❯ hyperfine "./out/Release/node test/fixtures/snapshot/typescript.js" --warmup 5
Benchmark 1: ./out/Release/node test/fixtures/snapshot/typescript.js
  Time (mean ± σ):      72.3 ms ±   3.2 ms    [User: 62.3 ms, System: 10.2 ms]
  Range (min … max):    70.5 ms …  87.2 ms    40 runs

There are more numbers from folks trying on an earlier iteration of this work (although this branch now hashes the code and the cache as an extra check, but crc32 should be fast enough that it's not too big of an overhead). For example from #47472 (comment) tsc --version can go from ~90ms to ~40ms, yarn --version can go from ~190ms to ~135ms, from #47472 (comment) npm run echo with a ncc-bundled npm may go from ~150ms to ~110ms

---

summary

This patch implements automatic on-disk code caching that can be enabled
via an environment variable NODE_COMPILE_CACHE.

When set, whenever Node.js compiles a CommonJS or a ECMAScript Module,
it will use on-disk V8 code cache persisted in the specified directory
to speed up the compilation. This may slow down the first load of a
module graph, but subsequent loads of the same module graph may get
a significant speedup if the contents of the modules do not change.
Locally, this speeds up loading of test/fixtures/snapshot/typescript.js
from ~130ms to ~80ms.

To clean up the generated code cache, simply remove the directory.
It will be recreated the next time the same directory is used for
NODE_COMPILE_CACHE.

Compilation cache generated by one version of Node.js may not be used
by a different version of Node.js. Cache generated by different versions
of Node.js will be stored separately if the same directory is used
to persist the cache, so they can co-exist.

Caveat: currently when using this with V8 JavaScript code coverage, the
coverage being collected by V8 may be less precise in functions that are
deserialized from the code cache. It's recommended to turn this off when
running tests to generate precise coverage.

Implementation details:

There is one cache file per module on disk. The directory layout
is:

• Compile cache directory (from NODE_COMPILE_CACHE)
  • 8b23c8fe: CRC32 hash of CachedDataVersionTag + NODE_VERESION
  • 2ea3424d:
    • 10860e5a: CRC32 hash of filename + module type
    • 431e9adc: ...
    • ...

Inside the cache file, there is a header followed by the actual
cache content:

[uint32_t] code size
[uint32_t] code hash
[uint32_t] cache size
[uint32_t] cache hash
... compile cache content ...

When reading the cache file, we'll also check if the code size
and code hash match the code that the module loader is loading
and whether the cache size and cache hash match the file content
read. If they don't match, or if V8 rejects the cache passed,
we'll ignore the mismatch cache, and regenerate the cache after
compilation succeeds and rewrite it to disk.

There are still a bunch of TODOs for this feature, some of them are documented as TODO comments. In addition we can look into idle-time code cache serialization/writing which is also done by Blink to reduce the impact on first load. But this should be good enough as an initial iteration.

Note: why CRC32? Because we already have it in the dependencies (in zlib.h), it's fast and can avoid enough collisions for this use case. We do have some existing hashing functions available via openssl but those won't be available on no-crypto builds. We can revisit if CRC32 turns out to be inadequate for this feature. We can also revisit the directory structure or use an actual db, but for now the file I/O cost doesn't really show up when I profile it locally. The remaining bottleneck is the repeated UTF8 encoding, which I plan to get rid of in a follow-up (because it needs to dance with monkey-patching).

Refs: #47472

[nodejs-github-bot]

Review requested:

• @nodejs/gyp
• @nodejs/loaders
• @nodejs/security-wg
• @nodejs/v8-update
• @nodejs/vm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58387/

[joyeecheung]

When enabling NODE_COMPILE_CACHE and running tools/test.py parallel locally, three tests would fail:

Failed tests:
out/Release/node /Users/joyee/projects/node/test/parallel/test-runner-coverage.js
out/Release/node /Users/joyee/projects/node/test/parallel/test-runner-output.mjs
out/Release/node /Users/joyee/projects/node/test/parallel/test-v8-coverage.js

The failed tests are all related to less precise coverage which is currently a known caveat of code cache + v8 native coverage collection (e.g. see #51672). So I just documented that a bit in the docs that users should not use the two together for now. I could spare some time later to fix it in V8, but I don't think it's a blocker for this PR because this feature is currently opt-in. User-land code caching can run into this problem too, and it's technically a V8 bug.

[merceyz]

There are more numbers from folks trying on an earlier iteration of this work

Some up-to-date results with f7806b3:

$ hyperfine -w 10 "NODE_COMPILE_CACHE='' ./node ./.yarn/releases/yarn-4.1.1.cjs --version" "NODE_COMPILE_CACHE='/tmp/node-cache' ./node ./.yarn/releases/yarn-4.1.1.cjs --version"
Benchmark 1: NODE_COMPILE_CACHE='' ./node ./.yarn/releases/yarn-4.1.1.cjs --version
  Time (mean ± σ):     165.5 ms ±   1.9 ms    [User: 156.5 ms, System: 25.4 ms]
  Range (min … max):   162.2 ms … 169.3 ms    18 runs

Benchmark 2: NODE_COMPILE_CACHE='/tmp/node-cache' ./node ./.yarn/releases/yarn-4.1.1.cjs --version
  Time (mean ± σ):     116.0 ms ±   1.3 ms    [User: 109.2 ms, System: 23.5 ms]
  Range (min … max):   113.8 ms … 119.7 ms    25 runs

Summary
  NODE_COMPILE_CACHE='/tmp/node-cache' ./node ./.yarn/releases/yarn-4.1.1.cjs --version ran
    1.43 ± 0.02 times faster than NODE_COMPILE_CACHE='' ./node ./.yarn/releases/yarn-4.1.1.cjs --version

[jdalton]

This is so rad! Having had to manage and hook into systems for this having it built-in is very nice! ❤️ it.

[benjamingr]

This description is very technical - I would use more layman terms for the first paragraph.

[joyeecheung]

Do you have specific suggestions? I don't want to make it sound too magical to avoid false advertisement.

[benjamingr]

i'll do a follow up PR after this lands to not hold it but ideally something that focuses more on what it does and not the technical implementation

[benjamingr]

LGTM as experimental/active development. Have a few questions:

• how does this interact with SEA (would it make sense or be possible to only eventually bundle the code cache and not the code?)
• What happens on overflows or tampering with the cache file? Is there a security concern here regarding passing different code through the cache file (since the hash is just a crc32)?
• How does this feature interact with loaders/require hooks that modify the file contents and in particular passing them in different order? (reading the code it should work but want to make sure)

[benjamingr]

(Not blocking for this PR - we should have better user facing errors if there are permission issues creating/reading/writing the cache)

[joyeecheung]

This should not error at all, the idea is that if caching cannot be done, this is just a no-op. There can be some logging to help users figure out why the cache isn't hit to ensure better performance, but the cache not being hit only leads to "not speeding up module loading", and it's not an error (just like when you write code that V8 cannot optimize or have to deopt with, there is no error either).

[benjamingr]

It's literally caching I get that but when it won't work for users and they'll want to figure out why they'll ask us. It would be useful to at least trace it as as. warning or add DEBUG* like logs.

[joyeecheung]

There are already traces. The handful of tests here are testing against the traces .

[joyeecheung]

how does this interact with SEA (would it make sense or be possible to only eventually bundle the code cache and not the code?)

Currently SEA only supports bundling a single script and it's not compiled using the normal CJS loader, but there's already a dedicated option in the SEA config to enable code cache for that one file ("useCodeCache": true). Neither static import nor dynamic import is not currently supported in SEA. Users can load on-disk CJS files via createRequire() and in that case if the environment variable is enabled, the cache of on-disk modules loaded through the created require() will be put in that directory. This is all separated from the SEA itself.

What happens on overflows or tampering with the cache file? Is there a security concern here regarding passing different code through the cache file (since the hash is just a crc32)?

The cache will be silently ignored if the hash doesn't match. Under the Node.js threat model, code loaded from the file system is already trusted. If you can gain access to the file system, you might as well just replace the source file on disk that'll be executed and do whatever you want in that replaced file, instead of wasting all your time figuring out how to mutate the code cache that meets all the requirement below:

1. Can do what you need (this means you need to know how to hand-write V8 bytecode to do what you need. Also, it must be run with the right V8 version since the bytecode format changes all the time)
2. The mutated code cache is still of the same length of the original code cache (so if you want to do a lot of additional stuff in that bytecode, this is unlikely to pass)
3. It passes the additional checks done by V8, and V8 can actually run it just fine (and once the V8 version in Node.js is upgraded this is likely to stop working)
4. The CRC32 of this new code cache must also conflict with the original code cache (finding CRC32 conflicts isn't that hard, but finding CRC32 conflicts that meet 1-3 is hard)

TBH it seems a bit silly for an attacker to waste all their effort on this when they can already just replace the source code on disk if they have access to the file system. Also, this is already what people have been doing in the wild via e.g. https://www.npmjs.com/package/v8-compile-cache with 12M downloads/week, so if there are any security concerns, well it's already out there for many years, and it's unlikely that the ecosystem will just stop relying on a capability like this.

How does this feature interact with loaders/require hooks that modify the file contents and in particular passing them in different order? (reading the code it should work but want to make sure)

If they are using the "official" hooks, e.g. module.register the modules will be cached and the resource name & module content cached match the results coming out of the loaders. If it's monkey-patching CJS hooks this will be ignored if they patch Module.wrap, Module.wrapper or Module.prototype._compile (without calling the original version, this means it's a no-op if v8-compile-cache is used since that package does this), but otherwise it should work fine.

[merceyz]

Would it be feasible to add a way to enable this for the current process only, perhaps as a follow-up?

For context Corepack uses v8-compile-cache to speed up the loading of package managers but doesn't modify process.env to avoid affecting anything said package managers might spawn.

[joyeecheung]

Would it be feasible to add a way to enable this for the current process only, perhaps as a follow-up?

Yeah I think we can discuss other ways to enable this (or a possible solution to enable it by default and pointing it to somewhere that's less likely to fill up the disk e.g. the tmpdir) in a follow-up. For the initial PR an opt-in environment variable should be enough.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58449/

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @joyeecheung.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

Joyee: See the PR description

[lirantal]

Good discussion and a cool performance improvement @joyeecheung, congrats!

Expressing some of my security concerns thoughts:

What happens on overflows or tampering with the cache file? Is there a security concern here regarding passing different code through the cache file (since the hash is just a crc32)?

The cache will be silently ignored if the hash doesn't match

It would be helpful for security audit trail to record in some way (warning, trace, etc) that a cache entry failed matching.

Other aspects worth considering, even if only through mentions in the documentation so that these risks are well communicated:

• Would child processes also inherit the enabled cache mode?
• It seems like this shouldn't conflict with experimental policies but raising just in case. For example, would the experimental policy throw an error for loaded cached code because there's no integrity specified/calculated for it?
• What are the security controls guarding against a denial of service? for example, attackers create large code files that would cause more cached code files to be created. Since the threat model already trusts the OS I get this is not in regular scope but potentially be in effect if child processes are inheriting code cache and in between an attacker creates this scenario (still out of scope I guess?)
• Where are the code cache files stored and how are they protected? If they are created in a predictable path and a vulnerability such as path traversal exists in the application, it could be used to exfiltrate the compiled code.

[joyeecheung]

It would be helpful for security audit trail to record in some way (warning, trace, etc) that a cache entry failed matching.

It can be traced using NODE_DEBUG_NATIVE=COMPILE_CACHE, see the tests

Would child processes also inherit the enabled cache mode?

Only if they inherit the environment variables (child process env options are defaulted to process.env, but once you specify it you need to copy process.env into the new options).

For example, would the experimental policy throw an error for loaded cached code because there's no integrity specified/calculated for it?

We can just skip it #52577 - the cache should be quiet by default, failures would just lead to no-ops.

What are the security controls guarding against a denial of service? for example, attackers create large code files that would cause more cached code files to be created.

If they already have access to the file system they might as well just modify the original module to run an infinite loop?

Where are the code cache files stored and how are they protected?

This is up to the user, it's currently opt-in via the environment variable NODE_COMPILE_CACHE, and the names of subdirectories change depending on the Node.js version and the filename and the module type. If an attacker can already get the filename of the module, they might as well just...modify the module?

[lirantal]

Thanks Joyee, appreciate all the input.

[WontonSam]

Review requested:

• @nodejs/gyp
• @nodejs/loaders
• @nodejs/security-wg
• @nodejs/v8-update
• @nodejs/vm