build: fix ninja build failure #12484

danbev · 2017-04-18T09:52:02Z

When working on commit 6a09a69
("build: enable cctest to use generated objects") I did not take into
account building with ninja:

$ ./configure
$ tools/gyp_node.py -f ninja
$ ninja -C out/Release
$ ln -fs out/Release/node node

When ninja generated the ninja build files, src files that are
relative to the src directory will be named with a dot instead of a
path separator, for example:

out/Release/obj/src/node/node.o

would instead become:

out/Release/obj/src/node.node.o

This commit adds an additional variable for the typ of object separator
used for this case.

Also, the LIBS specified in the 'libraries' section are not
being included in the --start-group --end-group section which
means that these libraries will not be searched causing issue
with linkers where the order matters.

Fixes: #12448

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

Affected core subsystem(s)

build

danbev · 2017-04-18T09:53:37Z

I'm not really sure if this is a good solution but want to see if this passes all the CI runs before looking into something better (perhaps making a change to ninja.py).

CI: https://ci.nodejs.org/job/node-test-pull-request/7469/

sam-github · 2017-04-18T13:46:51Z

Nice commit message, but " typ of object separator" has a typo ("type").

benjamingr

LGTM, I don't think the fix is that ugly fwiw and it fixes the problem locally for me and in the CI - but I'm not a very skilled ninja user by any means.

refack · 2017-04-18T14:52:02Z

node.gyp

+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)agent.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)node_trace_buffer.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)node_trace_writer.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)trace_event.<(OBJ_SUFFIX)',


Is this necessary? GYP usually knows how to handle slashes?

??? The object separator is .node on ninja or / on gyp, not / or \. This is the bit that actually fixes building with ninja, why wouldn't it be necessary?

Sorry I missed the 'OBJ_SEPARATOR': 'node.',

Still it probably should be 'OBJ_SEPARATOR': '/node.', with no trailing slashes in other variables.

refack · 2017-04-18T14:53:34Z

node.gyp

        'conditions': [
+          ['GENERATOR=="ninja"', {
+            'OBJ_PATH': '<(OBJ_DIR)/src/',


don't end paths with a / it breaks on Windows (when it's the end of the final argument)

refack · 2017-04-18T14:57:10Z

I have a feeling the slash manipulation change in node.gyp is not necessary (or might even degrade ninja output on Windows). Is there a way for you to gist a before and after outputs? for us to compare?

danbev · 2017-04-18T15:12:09Z

I have a feeling the slash manipulation change in node.gyp is not necessary (or might even degrade ninja output on Windows).

I'll run the ninja build on windows to verify that it works there too.

Fishrock123 · 2017-04-18T15:20:14Z

FWIW building with ninja on master works with or without this for me. macOS 10.12.4, ninja 1.6.0.

danbev · 2017-04-19T06:30:43Z

FWIW building with ninja on master works with or without this for me. macOS 10.12.4, ninja 1.6.0.

Is it possible that you already have built the project, which would mean that you have existing object files in out/Release/obj.target/node/src and the ninja build would in that case succeed?
I think doing a clean will show this error.

Fishrock123 · 2017-04-19T15:08:24Z

Ah, I thought I did make clean yesterday but apparently not. Yes, I also have the issue and this fixes it.

danbev · 2017-04-19T16:37:04Z

I've managed to run this on windows now and I've identified two issues. One is a simple fix where the win settings were overriding the ones set by the ninja condition.

The second issue I found was that the libraries in the cctest target in node.gyp, which in our case are object files, are getting a .lib extension in out/Release/obj/cctest.ninja. This causes a failure as there are no such files.
I've added a condition to tools/gyp/pylib/gyp/msvs_emulation.py to check for this situation. Not sure if this is a good solution or not but perhaps others can chime in now that they know the issue.

These are the commands I used to build:

> python configure --debug --ninja --dest-cpu=x86 --without-intl
> tools\gyp_node.py -f ninja
> ninja -C out\Release
> out\Release\cctest.exe

danbev · 2017-04-19T18:01:07Z

Rebased CI: https://ci.nodejs.org/job/node-test-pull-request/7526/

refack

Try to remove trailing slashes, and use prefixed slashes instead.

refack · 2017-04-19T20:15:20Z

node.gyp

+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)agent.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)node_trace_buffer.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)node_trace_writer.<(OBJ_SUFFIX)',
+        '<(OBJ_TRACING_PATH)<(OBJ_SEPARATOR)trace_event.<(OBJ_SUFFIX)',


Still it probably should be 'OBJ_SEPARATOR': '/node.', with no trailing slashes in other variables.

refack · 2017-04-19T20:22:36Z

@danbev sorry you got into GYP hell. If When you get frustrated, my best advice, do like @bnoordhuis in #12231, move the crazy logic into a python file. You can then call it with <! or <!@ https://gyp.gsrc.io/docs/InputFormatReference.md#Command-Expansions

refack · 2017-04-19T20:39:52Z

@danbev another tip: remember that GYP is just the Makefile generator. Running configure is cheap. Then rename output dir (out or Release) change .gyp files, rerun and text compare.

refack · 2017-04-19T22:57:11Z

@gibfahn @danbev @Fishrock123 I'm knee deep in the GYP code (they want the ninja generator to be VS2017 compatible), any thing I can fix while I'm there?

danbev · 2017-04-20T07:31:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/7544/

danbev · 2017-04-20T07:33:14Z

Try to remove trailing slashes, and use prefixed slashes instead.

@refack I've used a prefixed slash now, let me know what you think.

gibfahn · 2017-04-20T07:53:41Z

any thing I can fix while I'm there?

@refack the thing that irritates me with ninja is that if you do a ninja build and then a make test, make doesn't understand that the ninja build happened and rebuilds everything. I'm not sure how easy that would be to fix though, I suspect that might be beyond what make can do.

Also there could be a/some ninja based make target(s), but I don't know if collaborators would be okay with that.

The other thing is nodejs/node-gyp#1057, but that's mac specific, and I think upstream GYP aren't doing the xcodebuild stuff any more, so IDK if there's a way we can update gyp and avoid the warnings. In any case, warnings every time you configure are a pain.

bnoordhuis · 2017-04-20T08:01:01Z

Also there could be a/some ninja based make target(s), but I don't know if collaborators would be okay with that.

The Makefile used to have them but I removed it in de224d6. I don't have anything against ninja but approximately no one was using it at the time and it was slowing down the make build.

gibfahn · 2017-04-20T10:54:42Z

approximately no one was using it at the time and it was slowing down the make build.

How much does it slow down the build? It's interesting that people don't use it, I find it faster and cleaner to use, the only issue is that it doesn't play well with other make targets.

gibfahn · 2017-04-20T10:56:09Z

@refack see #12531 regarding GYP errors on macOS.

bnoordhuis · 2017-04-20T11:27:07Z

It's interesting that people don't use it, I find it faster and cleaner to use

I speculate that most people invoke ninja directly rather than through make (at least that is what I did - you use ninja because it's faster whereas the Makefile is a lumbering behemoth.)

refack · 2017-04-20T12:25:34Z

In #12425 I'm suggesting replacing make with ninja in order to eliminate the win/posix build disparity. Also it's faster (even than make).

refack · 2017-04-20T17:33:22Z

Code looks better, but ninja on windows fails:

D:\code\node\out.ninja\out\Debug$ ninja -j 3 -v -f build.ninja
[1/3] C:\bin\dev\python27\python.exe gyp-win-tool link-with-manifests environment.x64 True node.exe "C:\bin\dev\python27\python.exe gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:node.exe @node.exe.rsp" 1 mt.exe rc.exe "obj\node.node.exe.intermediate.manifest" obj\node.node.exe.generated.manifest ..\..\..\src\res\node.exe.extra.manifest
[2/3] C:\bin\dev\python27\python.exe gyp-win-tool stamp obj\cctest.actions_depends.stamp
[3/3] C:\bin\dev\python27\python.exe gyp-win-tool link-with-manifests environment.x64 True cctest.exe "C:\bin\dev\python27\python.exe gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:cctest.exe @cctest.exe.rsp" 1 mt.exe rc.exe "obj\cctest.cctest.exe.intermediate.manifest" obj\cctest.cctest.exe.generated.manifest ..\..\..\src\res\node.exe.extra.manifest
FAILED: cctest.exe cctest.exe.pdb
C:\bin\dev\python27\python.exe gyp-win-tool link-with-manifests environment.x64 True cctest.exe "C:\bin\dev\python27\python.exe gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:cctest.exe @cctest.exe.rsp" 1 mt.exe rc.exe "obj\cctest.cctest.exe.intermediate.manifest" obj\cctest.cctest.exe.generated.manifest ..\..\..\src\res\node.exe.extra.manifest
LINK : fatal error LNK1104: cannot open file 'obj/gen/node.node_javascript.obj.lib'

But since I have my knees deep in GYP I'm following up aswell.

danbev · 2017-04-20T17:39:35Z

Code looks better, but ninja on windows fails:

LINK : fatal error LNK1104: cannot open file 'obj/gen/node.node_javascript.obj.lib'

This does not look like you have the the changes in 5ddca82. Could you double check?

refack · 2017-04-20T17:49:47Z

This does not look like you have the the changes in 5ddca82. Could you double check?

Clashing GYP patches. I have patched that line to fix a different bug.
I feel like if you have to patch GYP in order to enable a feature (#11956), you might be doing something wrong 😉

Really no disrespect intended, GYP is super fragile, and it's baby .gyp files are super convoluted.
What do you say we roll back #11956, and work on it from a wider perspective together?

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: #12484 Fixes: #12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

danbev · 2017-05-11T12:18:23Z

@refack Thanks!

refack · 2017-05-11T12:18:34Z

Post land CI: https://ci.nodejs.org/job/node-test-commit/9810/

When working on commit 6a09a69 ("build: enable cctest to use generated objects") I did not take into account building with ninja: $ ./configure $ tools/gyp_node.py -f ninja $ ninja -C out/Release $ ln -fs out/Release/node node When ninja generated the ninja build files, src files that are relative to the src directory will be named with a dot instead of a path separator, for example: out/Release/obj/src/node/node.o would instead become: out/Release/obj/src/node.node.o This commit adds an additional variable for the type of object separator used for this case. Currently the check for if ninja is being used is a normal if statement as are the following os checks (win and aix). But the win and aix ones should only be evaluated if the build is not generated by ninja. This commit turns this logic into an if ninja else statement. PR-URL: nodejs#12484 Fixes: nodejs#12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: nodejs#12484 Fixes: nodejs#12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

gibfahn · 2017-06-20T23:13:40Z

Fix for #11956, so should be backported with that.

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: nodejs#12484 Fixes: nodejs#12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: nodejs/node#12484 Fixes: nodejs/node#12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: #12484 Fixes: #12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

Currently the files specified in libraries in node.gyp `cctest` target are getting a '.lib' extension on windows when generated with ninja. This commit adds a check to see if a file has a '.obj' extension and in that case no '.lib' extension will be added. Also, the LIBS specified in the 'libraries' section are not being included in the --start-group --end-group section which means that these libraries will not be searched causing issue with linkers where the order matters. PR-URL: nodejs/node#12484 Fixes: nodejs/node#12448 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

nodejs-github-bot added build Issues and PRs related to build files or the CI. tools Issues and PRs related to the tools directory. labels Apr 18, 2017

benjamingr approved these changes Apr 18, 2017

View reviewed changes

jasnell approved these changes Apr 18, 2017

View reviewed changes

refack suggested changes Apr 18, 2017

View reviewed changes

danbev force-pushed the ninja-build branch from 74c12ba to 5ddca82 Compare April 19, 2017 16:36

refack suggested changes Apr 19, 2017

View reviewed changes

jasnell mentioned this pull request May 28, 2017

8.0.0 Release Proposal #12220

Closed

gibfahn mentioned this pull request Jun 15, 2017

Auditing for 6.11.1 nodejs/Release#230

Closed

3 tasks

gibfahn added the backport-requested-v6.x label Jun 20, 2017

gibfahn mentioned this pull request Jun 20, 2017

build: enable cctest to use generated objects #11956

Closed

2 tasks

danbev deleted the ninja-build branch June 28, 2017 05:37

refack added the embedding Issues and PRs related to embedding Node.js in another project. label Aug 17, 2017

MylesBorins mentioned this pull request Sep 10, 2017

v8.5.0 proposal #15308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: fix ninja build failure #12484

build: fix ninja build failure #12484

danbev commented Apr 18, 2017

danbev commented Apr 18, 2017

sam-github commented Apr 18, 2017

benjamingr left a comment

refack Apr 18, 2017

gibfahn Apr 19, 2017

refack Apr 19, 2017

refack Apr 19, 2017

refack Apr 18, 2017

refack commented Apr 18, 2017

danbev commented Apr 18, 2017

Fishrock123 commented Apr 18, 2017

danbev commented Apr 19, 2017

Fishrock123 commented Apr 19, 2017

danbev commented Apr 19, 2017

danbev commented Apr 19, 2017

refack left a comment

refack Apr 19, 2017

refack commented Apr 19, 2017 •

edited

Loading

refack commented Apr 19, 2017

refack commented Apr 19, 2017

danbev commented Apr 20, 2017

danbev commented Apr 20, 2017 •

edited

Loading

gibfahn commented Apr 20, 2017

bnoordhuis commented Apr 20, 2017

gibfahn commented Apr 20, 2017

gibfahn commented Apr 20, 2017

bnoordhuis commented Apr 20, 2017

refack commented Apr 20, 2017

refack commented Apr 20, 2017

danbev commented Apr 20, 2017

refack commented Apr 20, 2017

danbev commented May 11, 2017

refack commented May 11, 2017

gibfahn commented Jun 20, 2017

build: fix ninja build failure #12484

build: fix ninja build failure #12484

Conversation

danbev commented Apr 18, 2017

Checklist

Affected core subsystem(s)

danbev commented Apr 18, 2017

sam-github commented Apr 18, 2017

benjamingr left a comment

Choose a reason for hiding this comment

refack Apr 18, 2017

Choose a reason for hiding this comment

gibfahn Apr 19, 2017

Choose a reason for hiding this comment

refack Apr 19, 2017

Choose a reason for hiding this comment

refack Apr 19, 2017

Choose a reason for hiding this comment

refack Apr 18, 2017

Choose a reason for hiding this comment

refack commented Apr 18, 2017

danbev commented Apr 18, 2017

Fishrock123 commented Apr 18, 2017

danbev commented Apr 19, 2017

Fishrock123 commented Apr 19, 2017

danbev commented Apr 19, 2017

danbev commented Apr 19, 2017

refack left a comment

Choose a reason for hiding this comment

refack Apr 19, 2017

Choose a reason for hiding this comment

refack commented Apr 19, 2017 • edited Loading

refack commented Apr 19, 2017

refack commented Apr 19, 2017

danbev commented Apr 20, 2017

danbev commented Apr 20, 2017 • edited Loading

gibfahn commented Apr 20, 2017

bnoordhuis commented Apr 20, 2017

gibfahn commented Apr 20, 2017

gibfahn commented Apr 20, 2017

bnoordhuis commented Apr 20, 2017

refack commented Apr 20, 2017

refack commented Apr 20, 2017

danbev commented Apr 20, 2017

refack commented Apr 20, 2017

danbev commented May 11, 2017

refack commented May 11, 2017

gibfahn commented Jun 20, 2017

refack commented Apr 19, 2017 •

edited

Loading

danbev commented Apr 20, 2017 •

edited

Loading