The future of GYP in Node.js

[rvagg]

Inspired by a discussion with @indutny, but this has been brewing for a long time now:

It's clear that Chromium has effectively deprecated GYP and stopped active development on it, with effort moving over to GN. @domenic reports that this is being finished up "this month".

GYP is stuck on Python ^2.7 and we get a regular stream of issues filed against node-gyp because of the pain people feel when using a default installed Python 3. In fact, the strict limited range of Python support (must be at least 2.7) causes pain on our build infra, where very old EL (EL5 / CentOS 5) comes with an older version and much of its system tools are tied to that version so introducing a new Python becomes a bit hacky.

We don't "support" GYP, we regularly get pull requests against node-gyp and nodejs/node that adjust the GYP source and our normal response is to take it upstream, where it sometimes succeeds through the Chromium gauntlet and is sometimes dropped. We've been (generally) unwilling to float patches, which puts us, and therefore our users, entirely at the mercy of Chromium developers. This is obviously an increasing concern as they move to GN and GYP becomes a legacy concern that they need to detach themselves from.

So, we need to come up with a plan for how to deal with GYP, or at least a general goal that we can encourage effort on where appropriate. If we are not proactive about this we're going to be left with a lot of pain, in fact, we've probably left it far too late as it is.

There are two very distinct issues to discuss and we need to make sure we keep them both in mind when discussing:

• Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.
• Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

Some beginning talking points:

• @chrisdickinson has collected some useful links on GN @ collecting useful links chrisdickinson/nojs#3 and that project uses GN
• This thread on Twitter discusses how difficult it is to extract GN from the Chromium codebase and reuse it elsewhere, it's not a separate project we can just pick off the shelf although @chrisdickinson is doing some work on that.
• We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.
• CMake often gets proposed as an alternative, I'm not a huge fan but I also don't have a ton of experience so can't be objective. It's worth discussing since it's a mature solution.
• There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

[saghul]

require('this_comment');

I think replacing GYP (the build file format thingy) would break things too much. We can keep GYP (the syntax) and replace the project with a pure JS thing which just generates Ninja files. GYP and GN files are similar enough that even if Google drops it Node can keep up. And be independent.

We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.

I did that, and called it GYN and I was able to compile iojs with it. Original post. I still volunteer to finish the work, if that's the approach everyone agrees upon.

Note that this is not orthogonal to replacing GYP with a pure JS thing. Since it's the same codebase, GYP could be replaced with GYN, which gives us the advantage of working on Python 3 and switching to Ninja. Then if GYN is replaced with a pure JS implementation, the Python dependency would go away.

/saghul's 2 cents

[seishun]

Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.

We've already survived migration from waf to gyp. Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

So in the end it depends on what V8 does, doesn't it? I'm not sure how much V8 team cares about third-party users, but if they do, I find it unlikely that they would switch to gn in its current state.

There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

Not a huge fan of introducing another prerequisite for building Node.js.

[bnoordhuis]

We've already survived migration from waf to gyp.

The ecosystem was a lot smaller those days, though.

Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

But only in one other project (Chromium), just like GYP in 2012.

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

[seishun]

But only in one other project (Chromium), just like GYP in 2012.

What about V8?

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

Do you mean addons or Node.js itself? I'm not sure what would be worse, using different build systems for addons and Node.js itself, or using a different build system for Node.js than for V8.

[indutny]

I'm working on a JS port of GYP at this very moment. Once it is done - it should be easier for the contributors to support it, and we could possibly modify it to fit our needs better.

[ofrobots]

Some other discussions on this: nodejs/node#6089 and nodejs/NG#24.

[eljefedelrodeodeljefe]

For reference I was working on this week and have a working compiler abstraction here: https://github.com/eljefedelrodeodeljefe/platform-tools

I am able to build C/C++ source files, native addons (haven't pushed win support yet) across all platforms and will head to compiling node itself after the weekend.

Also I disagree that we need a port of GYP or integrating Ninja or GN, since the task of building sources gets overcomplicated and overabstracted in all of those tools, imo, after looking at their sources and having been able to build a prototype really quick. There is a little bit more justification for a build tool on windows, since VS hides a lot of information like include paths in macros, which you have to dig out really. After that a callout to cl.exe is as easy as one to gcc.

[saghul]

@eljefedelrodeodeljefe Thanks for working on this! What I suggested was always based on the assumption that the migration would be slow, and changing one component at a time, trying to limit the risk.

I'm happy to hear you are able to compile addons calling the compiler directly, that's great news! FWIW Python does the same, you just create an instance of Extension, list some sources, call python setup.py build_ext and magic happens.

Keep up the good work!

[indutny]

FWIW, https://github.com/indutny/gyp.js

[indutny]

Update from gyp.js: node.js builds on windows with ninja and gyp.js, it also builds on mac/linux/freebsd as well, and possible on SmartOS with some tweaks to the .gyp file.

Should we consider investing some love into this project and/or using it in node-gyp?

[eljefedelrodeodeljefe]

I hope not. It's good engineering, don't get me wrong, but it is really bad management, sorry. This is 5000+ sloc of a port of a build systems that is by all means mediocre and chromium specific.

I am really unemotional about it, but ...sorry... this .... this is really bad news.

[Fishrock123]

Fwiw, with a JS port of GYP, people will likely no longer require python to build node addons if we use it in node-gyp, could be a big win, there's a good amount of non-compiler python issues that crop up there.

[Fishrock123]

I am really unemotional about it, but ...sorry... this .... this is really bad news.

What's bad news? Supporting an entire existing ecosystem without a another required language dependancy that we can better maintain ourselves? Perhaps not perfect, but that's a lot of wins IMO.

[evanlucas]

I'm pretty +1 on gyp.js. My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?

[eljefedelrodeodeljefe]

That you wanna do something really easy with 5000+ sloc someone needs to maintain and users need to be able to use without the fewest amount of bugs possible. With the rule being: the more sloc the bigger the surface error for bugs. It's specifically bad news, since I don't feel that node has a good track record on rather managerial issues.

[Fishrock123]

My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?

This month. The comments above indicate our ability to build via GYP shouldn't be too difficult to maintain.

That you wanna do something really easy with 5000+ sloc someone needs to maintain and users need to be able to use without the fewest amount of bugs possible.

We need to maintain it anyways for the ecosystem (for now, node-gyp..), so either we maintain Python GYP or JS GYP...

[cjihrig]

I'm +1 to eliminating as much Python as possible (from Node, not Python general).

[eljefedelrodeodeljefe]

no. You are missing out building our own solution, that is then as maintainable as you design it to be. Chromiums reason to build an own full fleshed build systems is rather historic and has mainly to do with the hundreds of repos they have and because they have lots of python people that were the only ones being able to build one back then. I don't know what you expect of a build system, but the way we use gyp we use maybe 5% of it.

It really is as easy as iterating over all .cc files and later link them all. So why not building some 500 sloc own build system that does just that in JS?

[seishun]

This month.

Could you provide a link?

The comments above indicate our ability to build via GYP shouldn't be too difficult to maintain.

You mean building V8 via GYP when V8 drops support for GYP?

We need to maintain it anyways for the ecosystem (for now, node-gyp..), so either we maintain Python GYP or JS GYP...

Ideally GYP support for modules should eventually be phased out in favor of gn. Gyp.js would be useful during the transition period since Node.js users wouldn't need to install Python to install modules that haven't updated to gn yet.

[ofrobots]

My only concern is what will we do once v8 switches to gn?

I don't think gyp as a tool is going away any time soon.

The question of how Node itself is built is a very different one from how native modules are built. I am +1 on reducing dependencies needed to deploy Node.js applications.

[eljefedelrodeodeljefe]

On the addons part:

I can do addon compilations with 300 sloc, doing the same what gyp does with no dependency and in a split of a second, here. Goes for all platforms, all architectures, doesn't need bindings.

The math for gyp-stuff (also ninja and gn) just doesn't add up. Why on earth would you use something big when you can use something lightweight?

  let out = [
    `${process.cwd()}/test/fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/addon_8.cc`,
    `${process.cwd()}/test/fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/myobject_8.cc`
  ]
  pt.compileAddon(out, {output: `addon_8`}, (err) => {
    if (err) {
      console.log(err);
      assert(!err, 'must not call error here')
    }

    let addon = require('../fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/addon.js')

    var obj1 = addon.createObject(10);
    var obj2 = addon.createObject(20);
    var result = addon.add(obj1, obj2);

    assert.equal(result, 30);

  })

[indutny]

@eljefedelrodeodeljefe your point is certainly valid. I'm quite sure that we need to consider alternatives, but what if instead of finding them - we could just cut out all that "is not needed" from gyp.js? I have removed tons of features already, and the ninja generator is written from scratch as just a bare minimum for addons and node.

[indutny]

@eljefedelrodeodeljefe regarding your code sample:

• Does it support defines, escaping defines contents?
• What about include dirs? Dependencies, dependencies includes?
• Actions? Can it copy the output to some directory or run any script?
• Shared libraries?
• How does it support long command line arguments for the cl.exe MSVS compiler?

There are lots of stuff that I have discovered in GYP that we didn't bother ourself much with, because it "just worked".

[bnoordhuis]

either we maintain Python GYP or JS GYP

Short/middle term, I don't think it's either/or. gyp.js undoubtedly has a bunch of features it's missing and bugs that need ironing out. We'll need to maintain both until we're confident it's a drop-in replacement.

[indutny]

@bnoordhuis this could be just one LTS cycle for ironing out differences.

[eljefedelrodeodeljefe]

@indutny that would be possible of course. Can you summarize, what specific gyp or ninja features we use? What the "just worked" components are?

I like ninja over gyp of course, but still I don't find the justification of it (but please change my mind, of course)

re sample:

• Defines: not just yet, but it's a 6 sloc change. Exactly at the central component passing include dirs
• -I, -L etc., yes, all of it, and escaped. Under the hood the above calls two functions .compile(), .link() that, in the .compileAddon case set all the flags necessary for addon compilation in an options object.
• in the addon-case what we compile is a shared library(no?), otherwise yes. You can set, but there is no feature for it just yet.
• long lines: didn't know that was a problem, but this case should be avoided by calling cl.exe /c && cl.exe /link similarly to gcc -c && gcc that would call ld under the hood.

For node that can be tons of flags more, but it's already a lot. The above would result in a spawn callout to the below. (waiting for parallels to spawn)

What we could do is having a build tool like node-gyp and step by step phasing out gyp components first to gyp.js then to none at all, imo

[indutny]

@eljefedelrodeodeljefe GYP over ninja is not a choice, I'm afraid. As ninja builds using pretty raw platform-dependent build files, and GYP is the thing that can generate them.

Relevant to the command-line length:

• https://msdn.microsoft.com/en-us/library/ms404301.aspx

Quote from ninjas manual:

rspfile, rspfile_content
if present (both), Ninja will use a response file for the given command, i.e. write the selected string (rspfile_content) to the given file (rspfile) before calling the command and delete the file after successful execution of the command.

This is particularly useful on Windows OS, where the maximal length of a command line is limited and response files must be used instead.

[indutny]

@eljefedelrodeodeljefe for building node we use all of the features of ninja (it doesn't have many), and some of the GYP features:

• Actions: there are lots of scripts that are run during node.js build, many of them are python scripts, some of them use binaries that were just compiled
• Tons and tons of defines, lots of platform-dependent/flag-dependent conditions
• Of course, dependency graph should be just big enough to support differential rebuilds

Perhaps there are other things, but I can't remember them right now.

[eljefedelrodeodeljefe]

@indutny the node case is a more complex case, but looking at the verbose output, I don't see what the above thing couldn't handle. Except for the scripts that call binaries during, I can't tell, I didn't know. I wanted to make an attempt to build node w/o dependency and would start by just compile and linking the last few build steps.

My problem with make, gyp and ninja is that the amount of boilerplate Makefiles introduce, is that it quickly becomes writing in it as if it were a decent programming language. So you end up having python, js and a pretty complex make scripts that only a few people understand as a whole. Whereas porting the Makefile (or the underlying generated makefiles) to js probably looks ugly initially, but can be read by people.

Let alone there is one big advantage for developers (disadvantage for maintainers): make and msbuild, hide a lot of flags and logic. And, imo, that makes it really complicated to debug and to port stuff. For example, when you call cl.exe directly, it pretty much looks like a call to gcc. But you won't notice, b/c msbuild has tons of path include build in. Including the most basic C headers like errno.h making this visible seems more maintainable, then running gyp that generates a VS project, that generates a msbuild call that generates a call cl.exe

A callout looks like this. Note: this is a reduced flag set, which doesn't for example inlining souces into assembly code and build memory maps, things I think are necessary only for chromium and produce bloat, especially in file sizes of a compilation's secondary files

PLATFORM_TOOLS 1812: "%programfiles(x86)%\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe" [ '/nologo',
  '/I"C:\\Users\\jefe-win\\repos\\platform-tools-2/build/deps/6.2.1/headers/include/node"',
  '/I"C:\\Users\\jefe-win\\repos\\platform-tools-2/node_modules/nan"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.10240.0\\ucrt"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x86"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x64"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x86"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x64"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\lib"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\lib\\amd64"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/Zi',
  '/W3',
  '/WX-',
  '/Ox',
  '/Ob2',
  '/Oi',
  '/Ot',
  '/Oy',
  '/GL',
  '/D',
  'WIN32',
  '/D',
  '_CRT_SECURE_NO_DEPRECATE',
  '/D',
  '_CRT_NONSTDC_NO_DEPRECATE',
  '/D',
  '_HAS_EXCEPTIONS=0',
  '/D',
  'BUILDING_V8_SHARED=1',
  '/D',
  'BUILDING_UV_SHARED=1',
  '/D',
  'BUILDING_NODE_EXTENSION',
  '/D',
  '_WINDLL',
  '/GF',
  '/Gm-',
  '/MT',
  '/GS',
  '/Gy',
  '/fp:precise',
  '/Zc:wchar_t',
  '/Zc:forScope',
  '/Zc:inline',
  '/GR-',
  '/Gd',
  '/TP',
  '/wd4351',
  '/wd4355',
  '/wd4800',
  '/wd4251',
  '/errorReport:queue',
  '/MP',
  '/c',
  '/FoC:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.o',
  'C:\\Users\\jefe-win\\repos\\platform-tools-2/test/fixtures/sources/addons/node-addon-examples/4_object_factory/nan/addon_4.cc' ]

PLATFORM_TOOLS 1812: "%programfiles(x86)%\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe" [ 'C:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.o',
  '/link',
  '/LIBPATH:"C:\\Users\\jefe-win\\repos\\platform-tools-2/build/Release"',
  '/ERRORREPORT:QUEUE',
  '/INCREMENTAL:NO',
  'kernel32.lib',
  'user32.lib',
  'gdi32.lib',
  'winspool.lib ',
  'comdlg32.lib',
  'advapi32.lib',
  'shell32.lib',
  'ole32.lib',
  'oleaut32.lib',
  'uuid.lib',
  'odbc32.lib',
  'C:\\Users\\jefe-win\\repos\\platform-tools-2\\build\\deps\\node\\6.2.1\\node.lib',
  '/MAP',
  '/MAPINFO:EXPORTS',
  '/OPT:REF',
  '/OPT:ICF',
  '/LTCG',
  '/TLBID:1',
  '/DYNAMICBASE',
  '/NXCOMPAT',
  '/MACHINE:X64',
  '/ignore:4199',
  '/DLL',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x86"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 11.0\\VC\\bin"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.10240.0\\ucrt\\x64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Lib\\winv6.3\\um\\x64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.10240.0\\ucr"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/LIBPATH:"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/OUT:C:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.node' ]

[Qard]

@bnoordhuis That's fair. I'm just thinking what the ideal situation would be, but certainly the reality may not be close enough. I haven't tried cross-compiling node before, so I wouldn't know.

Do you feel the gyp format is too limited to adequately express our needs for cross-compiling, or is it just that the current state of our gyp files doesn't show enough consideration of it?

[bnoordhuis]

The latter. It has never been a priority so far.

[indutny]

There is still a possibility of using gyp.js with duktape for extending platform support. Yes, it will be slow, but for doing the first build it should be fine.

[Qard]

That's an interesting idea. It might also be worth working with the native API stuff to try and make a sort of reference Node.js build on duktape. 🤔

[piranna]

It might also be worth working with the native API stuff to try and make a sort of reference Node.js build on duktape

I would love that idea! :-D

[ofrobots]

A few things I have learnt recently:

• GN can produce JSON output. This can be useful in building transpilers.
• Here's a project that uses said JSON output to convert GN to CMake: https://skia.googlesource.com/skia/+/master/gn/gn_to_cmake.py

[dpranke]

Hi all,

I am one of the maintainers of both GYP and GN (and I am part of the Chromium project).

If there are specific questions I could answer for you, do let me know :).

[hashseed]

We came up with a plan forward, described here.

TL;DR: V8 will implement a GN wrapper so that both the legacy GYP config and the new GN wrapper can be used to build the v8 build target. V8 will then stop maintaining the legacy GYP config, which Node.js will inherit.

[seishun]

In a second step, the legacy GYP config will be moved into Node.js’ repository. Keeping it in sync with GN build config becomes Node.js’ responsibility, which has to be done every time Node master updates to a newer V8 version

Has this already been decided? Because it seems... suboptimal in the long term.

[eljefedelrodeodeljefe]

I am also rather unpleased. Even though that was rather clear from the beginning, it now sounds good for V8 but 0-improvement for Node.

[hashseed]

Maintenance of V8's GYP configs will stop by the end of year. Node.js has been the only reason it is still being maintained so far, after Chrome has switched to GN more than half a year ago. Maintaining two build systems long-term doesn't sound particularly appealing.

The proposed plan decouples V8's GYP deprecation from Node.js. Obviously, if Node.js wants to move away from GYP, work has to be invested, regardless of whether Node.js wants to adopt GN or something else. I understand that this does not please everyone, but there are no viable and better alternatives on the horizon to my knowledge.

[eljefedelrodeodeljefe]

There easily could be one if @nodejs/ctc would create a policy for one. I wouldn't have expected something creative from your (Google's) corner...which is alright and no offense.

[refi64]

FWIW, I may be alone here, but I like the plan. Seems like it would have the minimal amount of potential breakage and workflow issues.

[Trott]

There easily could be one if @nodejs/ctc would create a policy for one.

Could you elaborate on this? I suspect I'm misunderstanding the meaning because that doesn't make any sense to me at all. (EDIT: What I mean is: I don't understand how a policy "easily" creates options that aren't there now. But there's a lot I could be missing...)

[fhinkel]

In a second step, the legacy GYP config will be moved into Node.js’ repository. Keeping it in sync with GN build config becomes Node.js’ responsibility, which has to be done every time Node master updates to a newer V8 version

Has this already been decided? Because it seems... suboptimal in the long term.

The gyp config is already in Node.js' repository as part of deps/v8. The difference is, that changes to it must be copied and pasted (or slightly modified) from V8's GN config files instead of from V8's gyp config files.

[hashseed]

Previously mistakenly dumped the meeting notes for the meeting between CTC and V8 team here. Moved them to where they belong here.

Sorry for the confusion.

[gibfahn]

@hashseed

Dumping meeting notes here.

Edited to @ everyone who was mentioned

[indutny]

Did I mention dukgyp already? Needs some love from Windows, otherwise is able to bootstrap node.

[seishun]

@trevnorris: A lot of work would be involved here.

Could someone describe what kind of work it would involve? Anything other than rewriting node.gyp, node.gypi and common.gypi and modifying Makefile and vcbuild.bat?

[eljefedelrodeodeljefe]

@Trott what I meant is that we are not leveraging the possibility of writing something better than exiting solutions. Gyp, make, CMake and all the likes are often bound to specific environments, like the dev environment at Google, and/ or overcomplicate things tremendously, because they turn DSLs. A viable option would just to make our own, suited for the dev environment of the Node.js ecosystem and being more simple, namely not requiring anything, but JS and a compiler. Other big software projects utilise authoring sub projects very well by having a plan / policy / vision someone can grab, setting out to spend a couple of months just programming what has been previously discussed as requirement.

[xml]

Please forgive an outsider to this project. But... it sounds like you have a great idea @eljefedelrodeodeljefe, and nobody seems to dispute the value of the ambition. I think people simply have lots of good experience with the gap between ambition and reality, and all the intervening edge cases. Until there's a tangible, viable alternative (at least a nascent one) that people could examine and rally around, the risk of implementation and adoption can't even be assessed, let alone solved. A strong community are going to quite rightly emphasize and invest in an MVP migration path until that's the case. (An MVP migration path might be a necessary parallel/backup option in any case.)

If you could show a baseline project and say 'let's use this, or something like it', it's a much stronger case. You mentioned you had a working build. Can you package that, name it, document it with a To-Do list, and share it to give people a sense of what would be involved? Once people can count lines, examine interface complexity, react to what's supported and what's not supported (especially around Windows)... it becomes a live option.

Until then, folks are almost certainly going to revert to the known possible.

[refack]

[edited to be less opinionated]

what I meant is that we are not leveraging the possibility of writing something better than exiting solutions

@eljefedelrodeodeljefe let me quote the scripture:

---

---

But seriously, I think a node-native platform-tools/autotools solution is great (wink wink), and perfectly suited for small/simple addons, and I'm willing to pitch in, if you're willing to commit to your platform-tools without apriori support.
If we have a PoC that shows it can compile 'n' arbitrary existing addons (CITGM style), we could convince the community.

As for building the whole node tree, I feel we're stuck with .gyp files for the time being (speaking from experience nodejs/node#12218 which is a spin off of nodejs/node#11217 and took ~2man-weeks, just to deobfuscate one half of one rather small .gyp file).
IMHO as a first step we should concentrate on the addons/node-gyp use case.

[hashseed]

As previously mentioned, V8 team is going to stop supporting the GYP build by the end of 2017. Here's the public announcement.

[Trott]

This issue has been inactive for a while and this repository is now obsolete. I'm going to close this, but feel free to open another issue in a relevant active repository (TSC perhaps?) and include a link back to this issue if this is a subject that should receive continued attention.