-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bootstrap: support configure-time user-land snapshot #42466
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refresh the options map during pre-execution to pave the way for user land snapshots which may need to access run-time options at snapshot-building time. The default embedded bootstrap snapshot is still prevented from accessing them at snapshot building time since it serves a wider audience and is ignorant of application states, while the user-land snapshots are meant to be application-specific and so it makes sense for them to access runtime states as long as the snapshotted-code works with re-initialized runtime states.
This adds a --build-snapshot runtime option which is currently only supported by the node_mksnapshot binary, and a --node-snapshot-main configure option that makes use it to run a custom script when building the embedded snapshot. The idea is to have this experimental feature in core as a configure-time feature for now, and investigate the renaming V8 bugs before we make it available to more users via making it a runtime option.
Use the mksnapshot cleanup hooks to release the references to the native streams so that they can be destroyed right before the snapshot is taken, and move the initialization of the global console to pre-execution so that they can be re-initialized during snapshot dehydration. This makes it possible for user-land snapshots to use the I/O during snapshot building.
This makes --inspect and stdin/out functional in the embedded snapshot, currently it's not guaranteed that these work perfectly, the plan is to investigate any out-of-sync states that might appear in user land snapshots with this while the it is a configure-time feature.
Review requested:
|
nodejs-github-bot
added
lib / src
Issues and PRs related to general changes in the lib or src directory.
needs-ci
PRs that need a full CI run.
labels
Mar 25, 2022
26 tasks
This comment was marked as outdated.
This comment was marked as outdated.
26 tasks
RaisinTen
reviewed
Mar 27, 2022
24 tasks
This comment was marked as outdated.
This comment was marked as outdated.
12 tasks
This comment was marked as outdated.
This comment was marked as outdated.
RaisinTen
reviewed
Mar 29, 2022
RaisinTen
approved these changes
Mar 29, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
11 tasks
VoltrexKeyva
approved these changes
Mar 30, 2022
VoltrexKeyva
added
the
author ready
PRs that have at least one approval, no pending requests for changes, and a CI started.
label
Mar 30, 2022
XadillaX
approved these changes
Mar 30, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👏🏻×1024
danielleadams
pushed a commit
that referenced
this pull request
Jun 27, 2022
PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
Refresh the options map during pre-execution to pave the way for user land snapshots which may need to access run-time options at snapshot-building time. The default embedded bootstrap snapshot is still prevented from accessing them at snapshot building time since it serves a wider audience and is ignorant of application states, while the user-land snapshots are meant to be application-specific and so it makes sense for them to access runtime states as long as the snapshotted-code works with re-initialized runtime states. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
This adds a --build-snapshot runtime option which is currently only supported by the node_mksnapshot binary, and a --node-snapshot-main configure option that makes use it to run a custom script when building the embedded snapshot. The idea is to have this experimental feature in core as a configure-time feature for now, and investigate the renaming V8 bugs before we make it available to more users via making it a runtime option. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
Use the mksnapshot cleanup hooks to release the references to the native streams so that they can be destroyed right before the snapshot is taken, and move the initialization of the global console to pre-execution so that they can be re-initialized during snapshot dehydration. This makes it possible for user-land snapshots to use the I/O during snapshot building. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
This makes --inspect and stdin/out functional in the embedded snapshot, currently it's not guaranteed that these work perfectly, the plan is to investigate any out-of-sync states that might appear in user land snapshots with this while the it is a configure-time feature. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
Refresh the options map during pre-execution to pave the way for user land snapshots which may need to access run-time options at snapshot-building time. The default embedded bootstrap snapshot is still prevented from accessing them at snapshot building time since it serves a wider audience and is ignorant of application states, while the user-land snapshots are meant to be application-specific and so it makes sense for them to access runtime states as long as the snapshotted-code works with re-initialized runtime states. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
This adds a --build-snapshot runtime option which is currently only supported by the node_mksnapshot binary, and a --node-snapshot-main configure option that makes use it to run a custom script when building the embedded snapshot. The idea is to have this experimental feature in core as a configure-time feature for now, and investigate the renaming V8 bugs before we make it available to more users via making it a runtime option. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
Use the mksnapshot cleanup hooks to release the references to the native streams so that they can be destroyed right before the snapshot is taken, and move the initialization of the global console to pre-execution so that they can be re-initialized during snapshot dehydration. This makes it possible for user-land snapshots to use the I/O during snapshot building. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
This makes --inspect and stdin/out functional in the embedded snapshot, currently it's not guaranteed that these work perfectly, the plan is to investigate any out-of-sync states that might appear in user land snapshots with this while the it is a configure-time feature. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 11, 2022
PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 31, 2022
Refresh the options map during pre-execution to pave the way for user land snapshots which may need to access run-time options at snapshot-building time. The default embedded bootstrap snapshot is still prevented from accessing them at snapshot building time since it serves a wider audience and is ignorant of application states, while the user-land snapshots are meant to be application-specific and so it makes sense for them to access runtime states as long as the snapshotted-code works with re-initialized runtime states. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 31, 2022
This adds a --build-snapshot runtime option which is currently only supported by the node_mksnapshot binary, and a --node-snapshot-main configure option that makes use it to run a custom script when building the embedded snapshot. The idea is to have this experimental feature in core as a configure-time feature for now, and investigate the renaming V8 bugs before we make it available to more users via making it a runtime option. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 31, 2022
Use the mksnapshot cleanup hooks to release the references to the native streams so that they can be destroyed right before the snapshot is taken, and move the initialization of the global console to pre-execution so that they can be re-initialized during snapshot dehydration. This makes it possible for user-land snapshots to use the I/O during snapshot building. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 31, 2022
This makes --inspect and stdin/out functional in the embedded snapshot, currently it's not guaranteed that these work perfectly, the plan is to investigate any out-of-sync states that might appear in user land snapshots with this while the it is a configure-time feature. PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
targos
pushed a commit
that referenced
this pull request
Jul 31, 2022
PR-URL: #42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this pull request
Oct 10, 2022
Refresh the options map during pre-execution to pave the way for user land snapshots which may need to access run-time options at snapshot-building time. The default embedded bootstrap snapshot is still prevented from accessing them at snapshot building time since it serves a wider audience and is ignorant of application states, while the user-land snapshots are meant to be application-specific and so it makes sense for them to access runtime states as long as the snapshotted-code works with re-initialized runtime states. PR-URL: nodejs/node#42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this pull request
Oct 10, 2022
This adds a --build-snapshot runtime option which is currently only supported by the node_mksnapshot binary, and a --node-snapshot-main configure option that makes use it to run a custom script when building the embedded snapshot. The idea is to have this experimental feature in core as a configure-time feature for now, and investigate the renaming V8 bugs before we make it available to more users via making it a runtime option. PR-URL: nodejs/node#42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this pull request
Oct 10, 2022
Use the mksnapshot cleanup hooks to release the references to the native streams so that they can be destroyed right before the snapshot is taken, and move the initialization of the global console to pre-execution so that they can be re-initialized during snapshot dehydration. This makes it possible for user-land snapshots to use the I/O during snapshot building. PR-URL: nodejs/node#42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this pull request
Oct 10, 2022
This makes --inspect and stdin/out functional in the embedded snapshot, currently it's not guaranteed that these work perfectly, the plan is to investigate any out-of-sync states that might appear in user land snapshots with this while the it is a configure-time feature. PR-URL: nodejs/node#42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this pull request
Oct 10, 2022
PR-URL: nodejs/node#42466 Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Khaidi Chu <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>
codebytere
added a commit
to electron/electron
that referenced
this pull request
Oct 12, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Oct 17, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Oct 19, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Oct 24, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 8, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 8, 2022
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 10, 2022
* chore: update to Node.js v18 * child_process: improve argument validation nodejs/node#41305 * bootstrap: support configure-time user-land snapshot nodejs/node#42466 * chore: update GN patch * src: disambiguate terms used to refer to builtins and addons nodejs/node#44135 * src: use a typed array internally for process._exiting nodejs/node#43883 * chore: lib/internal/bootstrap -> lib/internal/process * src: disambiguate terms used to refer to builtins and addons nodejs/node#44135 * chore: remove redudant browserGlobals patch * chore: update BoringSSL patch * src: allow embedder-provided PageAllocator in NodePlatform nodejs/node#38362 * chore: fixup Node.js crypto tests - nodejs/node#44171 - nodejs/node#41600 * lib: add Promise methods to avoid-prototype-pollution lint rule nodejs/node#43849 * deps: update V8 to 10.1 nodejs/node#42657 * src: add kNoBrowserGlobals flag for Environment nodejs/node#40532 * chore: consolidate asar initialization patches * deps: update V8 to 10.1 nodejs/node#42657 * deps: update V8 to 9.8 nodejs/node#41610 * src,crypto: remove AllocatedBuffers from crypto_spkac nodejs/node#40752 * build: enable V8's shared read-only heap nodejs/node#42809 * src: fix ssize_t error from nghttp2.h nodejs/node#44393 * chore: fixup ESM patch * chore: fixup patch indices * src: merge NativeModuleEnv into NativeModuleLoader nodejs/node#43824 * [API] Pass OOMDetails to OOMErrorCallback https://chromium-review.googlesource.com/c/v8/v8/+/3647827 * src: iwyu in cleanup_queue.cc * src: return Maybe from a couple of functions nodejs/node#39603 * src: clean up embedder API nodejs/node#35897 * src: refactor DH groups to delete crypto_groups.h nodejs/node#43896 * deps,src: use SIMD for normal base64 encoding nodejs/node#39775 * chore: remove deleted source file * chore: update patches * chore: remove deleted source file * lib: add fetch nodejs/node#41749 * chore: remove nonexistent node specs * test: split report OOM tests nodejs/node#44389 * src: trace fs async api nodejs/node#44057 * http: trace http request / response nodejs/node#44102 * test: split test-crypto-dh.js nodejs/node#40451 * crypto: introduce X509Certificate API nodejs/node#36804 * src: split property helpers from node::Environment nodejs/node#44056 * nodejs/node#38905 bootstrap: implement run-time user-land snapshots via --build-snapshot and --snapshot-blob * lib,src: implement WebAssembly Web API nodejs/node#42701 * fixup! deps,src: use SIMD for normal base64 encoding * fixup! src: refactor DH groups to delete crypto_groups.h * chore: fixup base64 GN file * fix: check that node::InitializeContext() returns true * chore: delete _noBrowserGlobals usage * chore: disable fetch in renderer procceses * dns: default to verbatim=true in dns.lookup() nodejs/node#39987 Co-authored-by: PatchUp <73610968+patchup[bot]@users.noreply.github.com>
khalwa
pushed a commit
to solarwindscloud/electron
that referenced
this pull request
Feb 22, 2023
* chore: update to Node.js v18 * child_process: improve argument validation nodejs/node#41305 * bootstrap: support configure-time user-land snapshot nodejs/node#42466 * chore: update GN patch * src: disambiguate terms used to refer to builtins and addons nodejs/node#44135 * src: use a typed array internally for process._exiting nodejs/node#43883 * chore: lib/internal/bootstrap -> lib/internal/process * src: disambiguate terms used to refer to builtins and addons nodejs/node#44135 * chore: remove redudant browserGlobals patch * chore: update BoringSSL patch * src: allow embedder-provided PageAllocator in NodePlatform nodejs/node#38362 * chore: fixup Node.js crypto tests - nodejs/node#44171 - nodejs/node#41600 * lib: add Promise methods to avoid-prototype-pollution lint rule nodejs/node#43849 * deps: update V8 to 10.1 nodejs/node#42657 * src: add kNoBrowserGlobals flag for Environment nodejs/node#40532 * chore: consolidate asar initialization patches * deps: update V8 to 10.1 nodejs/node#42657 * deps: update V8 to 9.8 nodejs/node#41610 * src,crypto: remove AllocatedBuffers from crypto_spkac nodejs/node#40752 * build: enable V8's shared read-only heap nodejs/node#42809 * src: fix ssize_t error from nghttp2.h nodejs/node#44393 * chore: fixup ESM patch * chore: fixup patch indices * src: merge NativeModuleEnv into NativeModuleLoader nodejs/node#43824 * [API] Pass OOMDetails to OOMErrorCallback https://chromium-review.googlesource.com/c/v8/v8/+/3647827 * src: iwyu in cleanup_queue.cc * src: return Maybe from a couple of functions nodejs/node#39603 * src: clean up embedder API nodejs/node#35897 * src: refactor DH groups to delete crypto_groups.h nodejs/node#43896 * deps,src: use SIMD for normal base64 encoding nodejs/node#39775 * chore: remove deleted source file * chore: update patches * chore: remove deleted source file * lib: add fetch nodejs/node#41749 * chore: remove nonexistent node specs * test: split report OOM tests nodejs/node#44389 * src: trace fs async api nodejs/node#44057 * http: trace http request / response nodejs/node#44102 * test: split test-crypto-dh.js nodejs/node#40451 * crypto: introduce X509Certificate API nodejs/node#36804 * src: split property helpers from node::Environment nodejs/node#44056 * nodejs/node#38905 bootstrap: implement run-time user-land snapshots via --build-snapshot and --snapshot-blob * lib,src: implement WebAssembly Web API nodejs/node#42701 * fixup! deps,src: use SIMD for normal base64 encoding * fixup! src: refactor DH groups to delete crypto_groups.h * chore: fixup base64 GN file * fix: check that node::InitializeContext() returns true * chore: delete _noBrowserGlobals usage * chore: disable fetch in renderer procceses * dns: default to verbatim=true in dns.lookup() nodejs/node#39987 Co-authored-by: PatchUp <73610968+patchup[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
author ready
PRs that have at least one approval, no pending requests for changes, and a CI started.
lib / src
Issues and PRs related to general changes in the lib or src directory.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a less-ambitious version of #38905 which only supports user-land snapshot at configure time. I've made the last two commits separate in case they cause any regressions and it would be easier to revert them to test things out.
A quick example of the feature:
typescript.js
: A modified version of https://github.com/microsoft/TypeScript/blob/main/lib/typescript.js which stores the initialized ts namespace in globalThis, see #38905For now the user can initialize the data/apps to be snapshotted in the entry point specified by
--node-snapshot-main
and hang this somewhere they can retrieve back after the snapshot is deserialized e.g. globalThis, we can invent better APIs in subsequent PRs (e.g. allowing the user to add a module into the module graph at build time that can berequire
ed back at runtime). Similar to #38905 the user has to bundle their app into one script to be built into the binary, as user-land modules are not yet supported (we'll need to do more refactoring like #42191 to support user modules in the snapshot).bootstrap: refresh options in pre-execution
Refresh the options map during pre-execution to pave the way for
user land snapshots which may need to access run-time options at
snapshot-building time. The default embedded bootstrap snapshot
is still prevented from accessing them at snapshot building time
since it serves a wider audience and is ignorant of application
states, while the user-land snapshots are meant to be
application-specific and so it makes sense for them to access
runtime states as long as the snapshotted-code
works with re-initialized runtime states.
build: add --node-snapshot-main configure option
This adds a --build-snapshot runtime option which is currently only
supported by the node_mksnapshot binary, and a --node-snapshot-main
configure option that makes use it to run a custom script when
building the embedded snapshot. The idea is to have this experimental
feature in core as a configure-time feature for now, and investigate
the renaming V8 bugs before we make it available to more users via
making it a runtime option.
bootstrap: make I/O streams work with user-land snapshot
Use the mksnapshot cleanup hooks to release the references
to the native streams so that they can be destroyed right
before the snapshot is taken, and move the initialization
of the global console to pre-execution so that they can be
re-initialized during snapshot dehydration. This makes
it possible for user-land snapshots to use the I/O during
snapshot building.
bootstrap: run inspector and event loop in snapshot builder
This makes --inspect and stdin/out functional in the embedded snapshot,
currently it's not guaranteed that these work perfectly, the
plan is to investigate any out-of-sync states that might appear
in user land snapshots with this while the it is a configure-time
feature.
Refs: #35711