Suggestion: New Python & GYP strategic initiative or working group

[rvagg]

Python 2

https://www.python.org/dev/peps/pep-0373/

Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support.

The Python 2 & 3 debacle is increasingly painful for us and is getting worse as each month goes by. I'm not a Python person so I don't understand the landscape, culture or much else, I just know it hurts Node and we're not taking a strategic approach to fixing our Python related problems.

GYP

GYP is a mess too, Google, as it does, has moved on to shinier things and Node made the mistake of embracing it as a foundational tool for our entire ecosystem. A large portion of the regular stream of issues on nodejs/node-gyp are related to GYP itself or the Python 2 & 3 challenge. The maintenance burden is entirely on us and we're not up to the task because few of us want to have to care about this stuff. We obviously haven't embraced the ownership of GYP, it's now just a huge cost for us.

So many questions, so little strategy

• Is it time to make a proper switch to Python 3 across the board? (Makefile, vcbuild.bat, tools/, test utilities, build infra, node-gyp) Or should we migrate off Python in some places .. somehow? Support for Python 3 node-gyp#193
• How do we handle historical version pain with the 2 / 3 division, do we continue to test with 2 on older versions but invoke 3 with newer versions and if so, how do we make that work? Can 'make lint-py PYTHON=python3' be a manditory Jenkins test? build#1631 (comment)
• Should we have a nodejs/gyp repo for properly maintaining it?
• Do we have enough people to even maintain GYP?
• Should we switch to gn? (or go whole-hog with depot_tools while we're at it? HAH. Seriously though, maybe removing deps/v8 and pulling it in on build is a possibility?) Build node with GN node#21410
• Should we prioritise getting CMake working? [request] cmake build job build#1003
• If we go CMake for core, do we deprecate node-gyp and provide a migration path for addon authors to move easily to CMake or some other utility?
• What's the relationship and flow of code between GYP, node-gyp, npm, and nodejs/node/tools/gyp and is it working? tools: remove gyp from tools directory node#22343
• How do we sustainably manage the flow of node-gyp issues, some of which require fixes and some just need a response ("go blame the package author").

I'm not the person to lead this, but I'd be happy to help with infrastructure changes where possible. What I'd like to see is some folks put up their hands to lead some discussions on putting us into a more strategic frame to deal with these things. Just having a group that we can go to with these kinds of questions would be helpful! On the infra side I'd love to be able to say something like "well, the Python working group says that we're off Python 2 by Node 12 so ...".

I don't think it'd be constructive to use this issue to answer any of the specifics above, those are just examples of the kinds of things we need answering in a holistic way. I'm more interested in establishing a strategic framework for generating those answers.

CC @nodejs/build @nodejs/gyp @nodejs/node-gyp @nodejs/python

[richardlau]

* Should we have a nodejs/gyp repo for properly maintaining it?

nodejs/admin#247

[cclauss]

https://github.com/chromium/gyp is already syntax compatible with Python 3
nodejs/build#1631 (comment)

[JamesMGreene]

https://github.com/chromium/gyp is already syntax compatible with Python 3

Note that this is only as of ~1 week ago and it is still largely untested.

[mhdawson]

I think its a good suggestion to form a strategic initiative or working group.

[mcollina]

I think gyp is a technical debt of the Node.js community. Let’s divide the issue:

a) support Gyp in the ecosystem
b) use Gyp for building Node.js

IMHO, core and the ecosystem might require two different answer. However it will take 3+ years to get rid of gyp (if at all) in the ecosystem.

+1 for kicking off a strategic initiative about build tools for Node core and the ecosystem, not just gyp-focused.

[rvagg]

I forgot to add https://github.com/indutny/gyp.js to the list. There was an attempt to get a test version of node-gyp out with this replacing the native gyp, but that never happened and Fedor has moved on (last I saw he was happy for anyone else to take it over). That should at least be on the table for consideration by any working group or strategic initiative.

We just need some folks to put up their hands to lead this thing.

[thefourtheye]

I have very little knowledge about gyp, but I would like to help driving this from Python's front.

[rvagg]

Here's a first pass at an outline for a new Working Group. I'd love someone, preferably from the TSC, to put up their hand to lead this. There's plenty of others that could get involved too I think but we need good leadership.

Python and GYP Working Group

Purpose

To provide strategic guidance to the Node.js project regarding its use of Python and the GYP tool as used within Node.js itself and throughout the Node.js ecosystem via the node-gyp tool.

Goals

This working group will cease to operate as an active concern when each of these goals have been completed and deliverables have been produced.

1. Python and the Node.js Codebase

Form a strategy for the maintenance of Python code contained within the nodejs/node project, with regard to the end of support for Python 2. Such a strategy may involve a staged approach to achieving an ideal end-state if necessary. The scope of this strategy should include all currently supported and future release lines. Stability for downstream consumers (binary and source) of Node.js should be a priority.

2. Python and Node.js Infrastructure

Form recommendations for the Build Working Group regarding its use of Python throughout the project's infrastructure, with regard to the end of support for Python 2. These recommendations are likely to relate primarily to the interaction with code from nodejs/node during test and release builds and versions of Python available and executed on the variety of computers in the test and release clusters.

3. GYP in the Node.js Codebase

Form a strategy for the reliance on GYP and Python in the nodejs/node project, with regard to the end of support for Python 2, the end of upstream support for GYP, the deprecation (and likely future removal) of a GYP build option in V8, and other relevant concerns. Such a strategy should outline an ideal end-state, along with explanations regarding why this end-state is ideal; a staged approach to achieving this end-state may be proposed if necessary. The scope of this strategy should include all currently supported and future release lines. Stability for downstream consumers of Node.js (binary and source) should be a priority.

4. Maintenance of node-gyp

Form a strategy for the maintenance of nodejs/node-gyp and GYP as it is bundled in npm in currently supported release lines of Node.js, with regard to the end of support for Python 2, the end of upstream support for GYP and other relevant concerns. The scope of this strategy should include all currently supported and future release lines. Stability for the Node.js package ecosystem (publishers and consumers) should be a priority.

5. The Future of node-gyp or an Alternative Package Toolchain

Form a strategy for the future development of nodejs/node-gyp or other default or recommended build toolchain for Node.js packages with compiled components (commonly referred to as "native addons"), with regard to the end of support for Python 2, the end of upstream support for GYP and other relevant concerns. Future-proofing and ease of migration to any new proposed toolchain should be priorities.

[ofrobots]

I'd be happy to collaborate in the WG, but I can't lead this.

[cclauss]

@rvagg Makes good sense to me.

I think it useful to add that Node dependencies v8 and inspector_protocol are not yet Python 3 compatible nodejs/node#24512 because v8 will require some effort.

[thefourtheye]

I'd love someone, preferably from the TSC, to put up their hand to lead this.

I would love to do this if there are no objections.

[mhdawson]

@thefourtheye thanks !

[refack]

FYI: the chromium team, @cclauss, and I have been working on getting python3 compatibility to GYP. Also work has been done to get compatibility to node's build toolchain.
The Google GYP repo (https://github.com/chromium/gyp) and the version in https://github.com/nodejs/node-gyp should be ready, but as mentioned above those don't get CI testing with python3.
https://github.com/refack/GYP is a work-in-progress on a GYP fork that does CI testing using both python2 & python3.

[mhdawson]

@thefourtheye given that the holiday season is over might be time to get rolling, PR an entry into the strategic initiatives with you as the lead etc.

[thefourtheye]

@mhdawson Thanks, got a PR up now #657.

[mhdawson]

Related: nodejs/node-addon-api#445

[Yzrsah]

GYP->GN Conversion Cookbook

The build system

[cclauss]

GN’s bootstrap.py, last_commit_position.py and bin scripts are largely written in legacy Python. Small modifications required to run on Python.

[mhdawson]

There is a strategic initiative in https://github.com/nodejs/TSC/blob/master/Strategic-Initiatives.md being led by @thefourtheye so I think we can close this issue. @rvagg is that correct?

[hayd]

Will updates still be made to this thread, or is there somewhere else to watch/offer help?

[richardlau]

According to https://changelog.travis-ci.com/upcoming-python-default-version-update-96873

On April 16th 2019, the default Python version used to run your builds will be updated from Python 2.7 to Python 3.6

Raised nodejs/node#27166 to keep our Travis builds working because I don't think we'll be Python 3 compatible by April 16th (happy to be proven wrong).

[rvagg]

Sadly we failed to get Node 12 to properly support Python 3.

Now we have Homebrew having problems with Node as they look at removing Python 2 entirely: nodejs/node-gyp#1337 (comment)

Someone's going to have to come up with a plan for what to do with Node 12 and Python 2/3—hopefully it doesn't have to involve breaking changes.

[mhdawson]

Issue which tracks the ongoing work: nodejs/node#25789