Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build,tools: make addons tests work with GN #50737

Closed
wants to merge 3 commits into from

Conversation

zcbenz
Copy link
Contributor

@zcbenz zcbenz commented Nov 15, 2023

This PR refactors the install and build_addons scripts so they can be reused by node-ci and electron projects to build the addons tests.

In detail, this PR does following things:

  1. Minor modifications of the GN configurations to fix loading native modules.
  2. Add some flags to tools/install.py script to allow customize locations of config.gypi, v8, etc. With some refactoring to remove global variables.
  3. Rewrite the tools/build_addons.py script:
    1. Rewrite it from js to python, because implementing cmd args handling with vanilla Node.js would take much longer time than a simple rewrite.
    2. Add a few more flags to make it easier to use for embedders.
  4. Modify the Makefile to generate headers in out dir first, and then calls build_addons.py --headers-dir=out_dir/headers (which calls node-gyp --nodedir=out_dir/headers) to build addons. Previously the build system was just calling node-gyp --nodedir=., which relied on quirky behavior of node-gyp and did not really test whether the generated headers work.

Users that rely on make or vcbuild.bat to work on Node are not affected.

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/tsc

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to build files or the CI. dependencies Pull requests that update a dependency file. libuv Issues and PRs related to the libuv dependency or the uv binding. needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform. labels Nov 15, 2023
@zcbenz zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 507f7ab to 353afc9 Compare November 15, 2023 11:06
@zcbenz zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 0940864 to 13f87fc Compare November 16, 2023 07:32
@zcbenz
Copy link
Contributor Author

zcbenz commented Nov 16, 2023

I have separated some changes out of this PR so review should be easier.

/cc @nodejs/gyp @joyeecheung @richardlau for reviewing changes on tools.

tools/build_addons.py Outdated Show resolved Hide resolved
@gengjiawen gengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 17, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 17, 2023
@nodejs-github-bot
Copy link
Collaborator

@zcbenz
Copy link
Contributor Author

zcbenz commented Nov 28, 2023

Ping @nodejs/gyp @joyeecheung @anonrig for review after the holiday.

Copy link
Member

@joyeecheung joyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % a correction but I think ideally we should have someone from @nodejs/build to confirm that it works with our releases.

Makefile Outdated Show resolved Hide resolved
@gengjiawen gengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023
@nodejs-github-bot
Copy link
Collaborator

@zcbenz zcbenz force-pushed the build-addons-gn branch 3 times, most recently from 974401b to 3be1111 Compare January 4, 2024 02:28
@zcbenz zcbenz deleted the build-addons-gn branch February 23, 2024 07:17
marco-ippolito pushed a commit that referenced this pull request Feb 26, 2024
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 26, 2024
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 27, 2024
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
@batrla
Copy link
Contributor

batrla commented Feb 28, 2024

Sorry to report it, but this PR caused regression, build in --shared mode now fails:

11:02:41: stdout installing /some_dir/build/prototype/i386/some_bindir/node
11:02:41: stderr Traceback (most recent call last):
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 419, in <module>
11:02:41: stderr     run(parse_options(sys.argv[1:]))
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 372, in run
11:02:41: stderr     files(options, install)
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 179, in files
11:02:41: stderr     action(options, [os.path.join(output_prefix, output_lib)],
11:02:41: stderr                                   ^^^^^^^^^^^^^
11:02:42: stderr UnboundLocalError: cannot access local variable 'output_prefix' where it is not associated with a value   

The problem in source code is obvious, the variable 'output_prefix' is only defined in some branches.
Later in code the variable is used in a branch where it was not defined before.

@zcbenz
Copy link
Contributor Author

zcbenz commented Feb 28, 2024

Sorry for breaking it! I'll create a fix soon.

zcbenz added a commit to zcbenz/node that referenced this pull request Feb 28, 2024
Fix a bug caused by nodejs#50737 that,
make install fails for --shared mode.
@marco-ippolito marco-ippolito mentioned this pull request Mar 1, 2024
nodejs-github-bot pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
zcbenz pushed a commit to photoionization/node_ci that referenced this pull request Mar 6, 2024
- Use node in-tree gn configs when possible.
  Some in-tree configs (deps/openssl) were modified in our mirror until
  they are fixed in the upstream node repo [1].
- Trigger landmine to clean out directory (required due to changed
  targets with new build files)
- Update to node-ci-2024-01-17
- Add fp16 to update_deps.py

[1] nodejs/node#50737.

Bug: v8:14572

Change-Id: I86cc19d23151b2a64d7173cf8f0aa6adc417c091
Reviewed-on: https://chromium-review.googlesource.com/c/v8/node-ci/+/5203209
Commit-Queue: Patrick Thier <[email protected]>
Reviewed-by: Victor Gomes <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
@richardlau richardlau mentioned this pull request Mar 25, 2024
rdw-msft pushed a commit to rdw-msft/node that referenced this pull request Mar 26, 2024
PR-URL: nodejs#50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
rdw-msft pushed a commit to rdw-msft/node that referenced this pull request Mar 26, 2024
Fix a bug caused by nodejs#50737 that,
make install fails for --shared mode.

PR-URL: nodejs#51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
codebytere added a commit to electron/electron that referenced this pull request Oct 16, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 22, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 28, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 29, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 31, 2024
jkleinsc pushed a commit to electron/electron that referenced this pull request Nov 4, 2024
* chore: bump Node.js to v22.9.0

* build: drop base64 dep in GN build

nodejs/node#52856

* build,tools: make addons tests work with GN

nodejs/node#50737

* fs: add fast api for InternalModuleStat

nodejs/node#51344

* src: move package_json_reader cache to c++

nodejs/node#50322

* crypto: disable PKCS#1 padding for privateDecrypt

nodejs-private/node-private#525

* src: move more crypto code to ncrypto

nodejs/node#54320

* crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey

nodejs/node#50234

* src: shift more crypto impl details to ncrypto

nodejs/node#54028

* src: switch crypto APIs to use Maybe<void>

nodejs/node#54775

* crypto: remove DEFAULT_ENCODING

nodejs/node#47182

* deps: update libuv to 1.47.0

nodejs/node#50650

* build: fix conflict gyp configs

nodejs/node#53605

* lib,src: drop --experimental-network-imports

nodejs/node#53822

* esm: align sync and async load implementations

nodejs/node#49152

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* module: detect ESM syntax by trying to recompile as SourceTextModule

nodejs/node#52413

* test: adapt debugger tests to V8 11.4

nodejs/node#49639

* lib: update usage of always on Atomics API

nodejs/node#49639

* test: adapt test-fs-write to V8 internal changes

nodejs/node#49639

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* deps: update libuv to 1.47.0

nodejs/node#50650

* src: use non-deprecated v8::Uint8Array::kMaxLength

nodejs/node#50115

* src: update default V8 platform to override functions with location

nodejs/node#51362

* src: add missing TryCatch

nodejs/node#51362

* lib,test: handle new Iterator global

nodejs/node#51362

* src: use non-deprecated version of CreateSyntheticModule

nodejs/node#50115

* src: remove calls to recently deprecated V8 APIs

nodejs/node#52996

* src: use new V8 API to define stream accessor

nodejs/node#53084

* src: do not use deprecated V8 API

nodejs/node#53084

* src: do not use soon-to-be-deprecated V8 API

nodejs/node#53174

* src: migrate to new V8 interceptors API

nodejs/node#52745

* src: use supported API to get stalled TLA messages

nodejs/node#51362

* module: print location of unsettled top-level await in entry points

nodejs/node#51999

* test: make snapshot comparison more flexible

nodejs/node#54375

* test: do not set concurrency on parallelized runs

nodejs/node#52177

* src: move FromNamespacedPath to path.cc

nodejs/node#53540

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* build: add option to enable clang-cl on Windows

nodejs/node#52870

* chore: fixup patch indices

* chore: add/remove changed files

* esm: drop support for import assertions

nodejs/node#54890

* build: compile with C++20 support

nodejs/node#52838

* deps: update nghttp2 to 1.62.1

nodejs/node#52966

* src: parse inspector profiles with simdjson

nodejs/node#51783

* build: add GN build files

nodejs/node#47637

* deps,lib,src: add experimental web storage

nodejs/node#52435

* build: add missing BoringSSL dep

* src: rewrite task runner in c++

nodejs/node#52609

* fixup! build: add GN build files

* src: stop using deprecated fields of v8::FastApiCallbackOptions

nodejs/node#54077

* fix: shadow variable

* build: add back incorrectly removed SetAccessor patch

* fixup! fixup! build: add GN build files

* crypto: fix integer comparison in crypto for BoringSSL

* src,lib: reducing C++ calls of esm legacy main resolve

nodejs/node#48325

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* chore: fixup GN files for previous commit

* src: move more crypto code to ncrypto

nodejs/node#54320

* Fixup Perfetto ifdef guards

* fix: missing electron_natives dep

* fix: node_use_node_platform = false

* fix: include src/node_snapshot_stub.cc in libnode

* 5507047: [import-attributes] Remove support for import assertions

https://chromium-review.googlesource.com/c/v8/v8/+/5507047

* fix: restore v8-sandbox.h in filenames.json

* fix: re-add original-fs generation logic

* fix: ngtcp2 openssl dep

* test: try removing NAPI_VERSION undef

* chore(deps): bump @types/node

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* buffer: fix out of range for toString

nodejs/node#54553

* lib: rewrite AsyncLocalStorage without async_hooks

nodejs/node#48528

* module: print amount of load time of a cjs module

nodejs/node#52213

* test: skip reproducible snapshot test on 32-bit

nodejs/node#53592

* fixup! src: move more crypto_dh.cc code to ncrypto

* test: adjust emittedUntil return type

* chore: remove redundant wpt streams patch

* fixup! chore(deps): bump @types/node

* fix: gn executable name on Windows

* fix: build on Windows

* fix: rename conflicting win32 symbols in //third_party/sqlite

On Windows otherwise we get:

lld-link: error: duplicate symbol: sqlite3_win32_write_debug
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:47987
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_sleep
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48042
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_is_nt
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48113
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_unicode
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48470
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_unicode_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48486
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48502
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48518
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48534
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48550
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

* docs: remove unnecessary ts-expect-error after types bump

* src: move package resolver to c++

nodejs/node#50322

* build: set ASAN detect_container_overflow=0

nodejs/node#55584

* chore: fixup rebase

* test: disable failing ASAN test

* win: almost fix race detecting ESRCH in uv_kill

libuv/libuv#4341
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. commit-queue-failed An error occurred while landing this pull request using GitHub Actions. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. dependencies Pull requests that update a dependency file. libuv Issues and PRs related to the libuv dependency or the uv binding. needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants