process: initial SourceMap support via NODE_V8_COVERAGE #28960

bcoe · 2019-08-04T20:43:47Z

Adds support for caching source maps when they're observed in footer of script file, and outputting them to the coverage directory if NODE_V8_COVERAGE is enabled.

This is seen as an initial step towards more thorough support for source maps inside Node.js itself.

Goal of this PR

Get it to the point where upstream tooling can detect source maps, even in the cases where they've been inserted by custom loaders or require.extensions.

Future Work (outside the scope of this PR)

expose source maps through an API, so that tooling can interact with them programmatically, e.g., fetching a SourceMap, or applying a SourceMap to a line/column position.
apply source maps to Node.js' annotations of error objects.
work with V8 folks towards adding source map support into V8 itself; with some of the functionality in Node.js (e.g., caching the source maps) and some of the functionality in V8 (e.g., having correct stack traces).

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

CC: @schuay, @snehasi (I've left some tests blank and I think this would be a great starting point for some of the work we're doing in the mentorship program).

Remaining Work

~~discuss future API direction (pull together design doc?).~~
- I don't think I'll be able to write a design document up for a few weeks, but I think this initial implementation is a step in the right direction.
make sure collecting source-maps isn't a major performance regression for coverage.
add support for base64 encoded data URIs.

joyeecheung

Is this supposed to be a WIP? Are we waiting for some V8 API addition that we can feed sourcemaps to to pair with this?

lib/internal/source_map.js

test/fixtures/v8-coverage/sigint.js

bcoe · 2019-08-05T19:51:26Z

Is this supposed to be a WIP? Are we waiting for some V8 API addition that we can feed sourcemaps to to pair with this?

@joyeecheung I've been thinking about this design problem for a little bit of time now, and the conclusion I've been coming to is we'd need to support source maps in both Node.js and V8:

Role I feel V8 should serve in supporting source maps

it would be nice if the stack traces thrown from V8 applied source maps (perhaps it includes both the transpiled position and original position, if a script has been created with a source map).
it would potentially be nice if source maps were automatically applied to the coverage iniformation output by V8 (this would simplify writing upstream tooling ... my only concern is that it's perhaps a lot of engineering for the value we'd get, if tooling could instead be applied a layer above).

Role I feel Node.js should serve in supporting source maps

it should detect source maps in a script being executed: loading the file if it's on disk; decoding data-URLs from inline source maps.
Node.js should pass the decoded source map to the script created in V8, so that it can be applied appropriately to stack traces.
Assuming V8 doesn't apply source maps to coverage generated (at least for some time) it would be nice if we could output source map information at exit, similarly to how we output coverage information (this would allow tooling to take into account source maps inserted at run time).
It would be nice to maintain a cache of the source maps loaded for files, and to expose this as an API, so that upstream tooling could interact with the source maps we've extracted for scripts.
Node.js performs post processing on the errors returned by V8, even if V8 applies source maps, it would be nice to be able to also correct this error output.

The scope of this proposal

I propose we start simple, by collecting source maps from source files, caching them, and exposing them similarly to how we expose coverage.

In follow up pull requests we could then gradually introduce additional behavior:

rewriting stack traces (which we could shim in Node.js until it's supported in V8).
exposing the source map cache as an API.

bcoe · 2019-08-19T05:31:59Z

@joyeecheung l added the method WriteSourceMapCache, which writes the source map cache to disk at the same time that the coverage data is written.

To me, it seemed like the most straight forward place to expose source maps from their JavaScript implementation in source_map.js to inspector_profiler.cc for serializing, was as an accessor on the global process object.

As you've recommended, I intend to extend on the source-map support further to:

load source-maps from file paths.
parse base64 encoded data-urls.
load source maps from other contexts, such as ESM modules.

Which is why I'd like to keep much of the implementation in source_map.js, taking advantage of higher level APIs, ideally just performing serialization in C++, if we can find an approach that makes folks happy.

lib/internal/bootstrap/node.js

lib/internal/source_map.js

src/inspector_profiler.cc

lib/internal/bootstrap/node.js

lib/internal/source_map.js

lib/internal/modules/cjs/loader.js

src/inspector_profiler.cc

doc/api/cli.md

lib/internal/bootstrap/node.js

lib/internal/source_map.js

nodejs-github-bot · 2019-09-21T19:42:12Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25623/

nodejs-github-bot · 2019-09-21T21:19:15Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25624/

PR-URL: #28960 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: David Carlier <[email protected]>

addaleax · 2019-09-21T22:40:08Z

Landed in 8f06773 🎉

PR-URL: #28960 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: David Carlier <[email protected]>

Notable changes: * crypto: * Add `oaepLabel` option #29489 * deps: * Update V8 to 7.7.299.11 #28918 * More efficient memory handling * Stack trace serialization got faster * The `Intl.NumberFormat` API gained new functionality * For more information: https://v8.dev/blog/v8-release-77 * events: * Add support for `EventTarget` in `once` #29498 * fs: * Expose memory file mapping flag `UV_FS_O_FILEMAP` #29260 * inspector: * New API - `Session.connectToMainThread` #28870 * process: * Initial SourceMap support via `env.NODE_V8_COVERAGE` #28960 * stream: * Make `_write()` optional when `_writev()` is implemented #29639 * tls: * Add option to override signature algorithms #29598 * util: * Add `encodeInto` to `TextEncoder` #29524 * worker: * The `worker_thread` module is now stable #29512

PR-URL: #28960 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: David Carlier <[email protected]>

Notable changes: * crypto: * Add `oaepLabel` option #29489 * deps: * Update V8 to 7.7.299.11 #28918 * More efficient memory handling * Stack trace serialization got faster * The `Intl.NumberFormat` API gained new functionality * For more information: https://v8.dev/blog/v8-release-77 * events: * Add support for `EventTarget` in `once` #29498 * fs: * Expose memory file mapping flag `UV_FS_O_FILEMAP` #29260 * inspector: * New API - `Session.connectToMainThread` #28870 * process: * Initial SourceMap support via `env.NODE_V8_COVERAGE` #28960 * stream: * Make `_write()` optional when `_writev()` is implemented #29639 * tls: * Add option to override signature algorithms #29598 * util: * Add `encodeInto` to `TextEncoder` #29524 * worker: * The `worker_thread` module is now stable #29512 PR-URL: #29695

bcoe requested review from addaleax, hashseed and joyeecheung August 4, 2019 20:43

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Aug 4, 2019

joyeecheung reviewed Aug 5, 2019

View reviewed changes

lib/internal/source_map.js Outdated Show resolved Hide resolved

test/fixtures/v8-coverage/sigint.js Outdated Show resolved Hide resolved

bcoe added the wip Issues and PRs that are still a work in progress. label Aug 13, 2019

bcoe force-pushed the source-map branch 2 times, most recently from fd155c3 to 3b07d5b Compare August 19, 2019 05:16

jasnell reviewed Aug 19, 2019

View reviewed changes

lib/internal/bootstrap/node.js Outdated Show resolved Hide resolved

jasnell reviewed Aug 20, 2019

View reviewed changes

lib/internal/source_map.js Show resolved Hide resolved

jasnell reviewed Aug 20, 2019

View reviewed changes

src/inspector_profiler.cc Outdated Show resolved Hide resolved

jasnell reviewed Aug 20, 2019

View reviewed changes

src/inspector_profiler.cc Outdated Show resolved Hide resolved

Trott added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2019

joyeecheung reviewed Aug 22, 2019

View reviewed changes

lib/internal/bootstrap/node.js Outdated Show resolved Hide resolved

lib/internal/source_map.js Outdated Show resolved Hide resolved

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

bcoe commented Aug 23, 2019

View reviewed changes

src/inspector_profiler.cc Outdated Show resolved Hide resolved

bcoe force-pushed the source-map branch from 76d0c21 to a96165d Compare August 24, 2019 00:33

joyeecheung reviewed Aug 24, 2019

View reviewed changes

doc/api/cli.md Outdated Show resolved Hide resolved

doc/api/cli.md Outdated Show resolved Hide resolved

lib/internal/bootstrap/node.js Outdated Show resolved Hide resolved

lib/internal/source_map.js Outdated Show resolved Hide resolved

lib/internal/source_map.js Show resolved Hide resolved

nodejs deleted a comment from nodejs-github-bot Aug 26, 2019

bcoe force-pushed the source-map branch from 18761ad to 4dab8e3 Compare September 1, 2019 02:15

bcoe changed the title ~~sourcemap: initial SourceMap support~~ process: initial SourceMap support Sep 2, 2019

bcoe added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. and removed wip Issues and PRs that are still a work in progress. labels Sep 2, 2019

bcoe changed the title ~~process: initial SourceMap support~~ process: initial SourceMap support tied to NODE_V8_COVERAGE Sep 2, 2019

bcoe added the c++ Issues and PRs that require attention from people who are familiar with C++. label Sep 2, 2019

bcoe changed the title ~~process: initial SourceMap support tied to NODE_V8_COVERAGE~~ process: initial SourceMap support via NODE_V8_COVERAGE Sep 2, 2019

nodejs deleted a comment from nodejs-github-bot Sep 2, 2019

nodejs deleted a comment from nodejs-github-bot Sep 20, 2019

nodejs deleted a comment from Freya716 Sep 20, 2019

nodejs deleted a comment from nodejs-github-bot Sep 20, 2019

nodejs deleted a comment from nodejs-github-bot Sep 21, 2019

addaleax mentioned this pull request Sep 21, 2019

src: fix closing weak HandleWraps on GC #29640

Closed

3 tasks

bcoe force-pushed the source-map branch from b90717a to 4329191 Compare September 21, 2019 19:35

nodejs deleted a comment from nodejs-github-bot Sep 21, 2019

process: initial SourceMap support via NODE_V8_COVERAGE

2a6dcde

bcoe force-pushed the source-map branch from 4329191 to 2a6dcde Compare September 21, 2019 20:58

addaleax closed this Sep 21, 2019

bcoe deleted the source-map branch September 22, 2019 18:02

BridgeAR mentioned this pull request Sep 24, 2019

v12.11.0 proposal #29695

Merged

7rulnik mentioned this pull request Sep 27, 2019

Future of rewriteFrames? getsentry/sentry-javascript#2254

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

process: initial SourceMap support via NODE_V8_COVERAGE #28960

process: initial SourceMap support via NODE_V8_COVERAGE #28960

bcoe commented Aug 4, 2019 •

edited

Loading

joyeecheung left a comment

bcoe commented Aug 5, 2019 •

edited

Loading

bcoe commented Aug 19, 2019

nodejs-github-bot commented Sep 21, 2019

nodejs-github-bot commented Sep 21, 2019

addaleax commented Sep 21, 2019

process: initial SourceMap support via NODE_V8_COVERAGE #28960

process: initial SourceMap support via NODE_V8_COVERAGE #28960

Conversation

bcoe commented Aug 4, 2019 • edited Loading

Goal of this PR

Future Work (outside the scope of this PR)

Checklist

Remaining Work

joyeecheung left a comment

Choose a reason for hiding this comment

bcoe commented Aug 5, 2019 • edited Loading

Role I feel V8 should serve in supporting source maps

Role I feel Node.js should serve in supporting source maps

The scope of this proposal

bcoe commented Aug 19, 2019

nodejs-github-bot commented Sep 21, 2019

nodejs-github-bot commented Sep 21, 2019

addaleax commented Sep 21, 2019

bcoe commented Aug 4, 2019 •

edited

Loading

bcoe commented Aug 5, 2019 •

edited

Loading