Suggestion: New Python & GYP strategic initiative or working group #642
Description
Python 2
https://www.python.org/dev/peps/pep-0373/
Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support.
The Python 2 & 3 debacle is increasingly painful for us and is getting worse as each month goes by. I'm not a Python person so I don't understand the landscape, culture or much else, I just know it hurts Node and we're not taking a strategic approach to fixing our Python related problems.
GYP
GYP is a mess too, Google, as it does, has moved on to shinier things and Node made the mistake of embracing it as a foundational tool for our entire ecosystem. A large portion of the regular stream of issues on nodejs/node-gyp are related to GYP itself or the Python 2 & 3 challenge. The maintenance burden is entirely on us and we're not up to the task because few of us want to have to care about this stuff. We obviously haven't embraced the ownership of GYP, it's now just a huge cost for us.
So many questions, so little strategy
- Is it time to make a proper switch to Python 3 across the board? (Makefile, vcbuild.bat, tools/, test utilities, build infra, node-gyp) Or should we migrate off Python in some places .. somehow? Support for Python 3 node-gyp#193
- How do we handle historical version pain with the 2 / 3 division, do we continue to test with 2 on older versions but invoke 3 with newer versions and if so, how do we make that work? Can 'make lint-py PYTHON=python3' be a manditory Jenkins test? build#1631 (comment)
- Should we have a nodejs/gyp repo for properly maintaining it?
- Do we have enough people to even maintain GYP?
- Should we switch to gn? (or go whole-hog with depot_tools while we're at it? HAH. Seriously though, maybe removing deps/v8 and pulling it in on build is a possibility?) Build node with GN node#21410
- Should we prioritise getting CMake working? [request] cmake build job build#1003
- If we go CMake for core, do we deprecate node-gyp and provide a migration path for addon authors to move easily to CMake or some other utility?
- What's the relationship and flow of code between GYP, node-gyp, npm, and nodejs/node/tools/gyp and is it working? tools: remove gyp from tools directory node#22343
- How do we sustainably manage the flow of node-gyp issues, some of which require fixes and some just need a response ("go blame the package author").
I'm not the person to lead this, but I'd be happy to help with infrastructure changes where possible. What I'd like to see is some folks put up their hands to lead some discussions on putting us into a more strategic frame to deal with these things. Just having a group that we can go to with these kinds of questions would be helpful! On the infra side I'd love to be able to say something like "well, the Python working group says that we're off Python 2 by Node 12 so ...".
I don't think it'd be constructive to use this issue to answer any of the specifics above, those are just examples of the kinds of things we need answering in a holistic way. I'm more interested in establishing a strategic framework for generating those answers.
CC @nodejs/build @nodejs/gyp @nodejs/node-gyp @nodejs/python
Activity
richardlau commentedon Dec 18, 2018
nodejs/admin#247
cclauss commentedon Dec 18, 2018
https://github.com/chromium/gyp is already syntax compatible with Python 3
nodejs/build#1631 (comment)
JamesMGreene commentedon Dec 18, 2018
Note that this is only as of ~1 week ago and it is still largely untested.
mhdawson commentedon Dec 18, 2018
I think its a good suggestion to form a strategic initiative or working group.
mcollina commentedon Dec 18, 2018
I think gyp is a technical debt of the Node.js community. Let’s divide the issue:
a) support Gyp in the ecosystem
b) use Gyp for building Node.js
IMHO, core and the ecosystem might require two different answer. However it will take 3+ years to get rid of gyp (if at all) in the ecosystem.
+1 for kicking off a strategic initiative about build tools for Node core and the ecosystem, not just gyp-focused.
rvagg commentedon Dec 18, 2018
I forgot to add https://github.com/indutny/gyp.js to the list. There was an attempt to get a test version of node-gyp out with this replacing the native gyp, but that never happened and Fedor has moved on (last I saw he was happy for anyone else to take it over). That should at least be on the table for consideration by any working group or strategic initiative.
We just need some folks to put up their hands to lead this thing.
thefourtheye commentedon Dec 19, 2018
I have very little knowledge about gyp, but I would like to help driving this from Python's front.
rvagg commentedon Dec 19, 2018
Here's a first pass at an outline for a new Working Group. I'd love someone, preferably from the TSC, to put up their hand to lead this. There's plenty of others that could get involved too I think but we need good leadership.
Python and GYP Working Group
Purpose
To provide strategic guidance to the Node.js project regarding its use of Python and the GYP tool as used within Node.js itself and throughout the Node.js ecosystem via the node-gyp tool.
Goals
This working group will cease to operate as an active concern when each of these goals have been completed and deliverables have been produced.
1. Python and the Node.js Codebase
Form a strategy for the maintenance of Python code contained within the nodejs/node project, with regard to the end of support for Python 2. Such a strategy may involve a staged approach to achieving an ideal end-state if necessary. The scope of this strategy should include all currently supported and future release lines. Stability for downstream consumers (binary and source) of Node.js should be a priority.
2. Python and Node.js Infrastructure
Form recommendations for the Build Working Group regarding its use of Python throughout the project's infrastructure, with regard to the end of support for Python 2. These recommendations are likely to relate primarily to the interaction with code from nodejs/node during test and release builds and versions of Python available and executed on the variety of computers in the test and release clusters.
3. GYP in the Node.js Codebase
Form a strategy for the reliance on GYP and Python in the nodejs/node project, with regard to the end of support for Python 2, the end of upstream support for GYP, the deprecation (and likely future removal) of a GYP build option in V8, and other relevant concerns. Such a strategy should outline an ideal end-state, along with explanations regarding why this end-state is ideal; a staged approach to achieving this end-state may be proposed if necessary. The scope of this strategy should include all currently supported and future release lines. Stability for downstream consumers of Node.js (binary and source) should be a priority.
4. Maintenance of node-gyp
Form a strategy for the maintenance of nodejs/node-gyp and GYP as it is bundled in npm in currently supported release lines of Node.js, with regard to the end of support for Python 2, the end of upstream support for GYP and other relevant concerns. The scope of this strategy should include all currently supported and future release lines. Stability for the Node.js package ecosystem (publishers and consumers) should be a priority.
5. The Future of node-gyp or an Alternative Package Toolchain
Form a strategy for the future development of nodejs/node-gyp or other default or recommended build toolchain for Node.js packages with compiled components (commonly referred to as "native addons"), with regard to the end of support for Python 2, the end of upstream support for GYP and other relevant concerns. Future-proofing and ease of migration to any new proposed toolchain should be priorities.
ofrobots commentedon Dec 19, 2018
I'd be happy to collaborate in the WG, but I can't lead this.
cclauss commentedon Dec 19, 2018
@rvagg Makes good sense to me.
I think it useful to add that Node dependencies v8 and inspector_protocol are not yet Python 3 compatible nodejs/node#24512 because v8 will require some effort.
thefourtheye commentedon Dec 20, 2018
I would love to do this if there are no objections.
mhdawson commentedon Dec 20, 2018
@thefourtheye thanks !
refack commentedon Jan 20, 2019
FYI: the chromium team, @cclauss, and I have been working on getting python3 compatibility to GYP. Also work has been done to get compatibility to node's build toolchain.
The Google GYP repo (https://github.com/chromium/gyp) and the version in https://github.com/nodejs/node-gyp should be ready, but as mentioned above those don't get CI testing with python3.
https://github.com/refack/GYP is a work-in-progress on a GYP fork that does CI testing using both python2 & python3.
mhdawson commentedon Jan 21, 2019
@thefourtheye given that the holiday season is over might be time to get rolling, PR an entry into the strategic initiatives with you as the lead etc.
thefourtheye commentedon Jan 23, 2019
@mhdawson Thanks, got a PR up now #657.
mhdawson commentedon Feb 11, 2019
Related: nodejs/node-addon-api#445
Yzrsah commentedon Feb 13, 2019
GYP->GN Conversion Cookbook
The build system
cclauss commentedon Feb 13, 2019
GN’s bootstrap.py, last_commit_position.py and bin scripts are largely written in legacy Python. Small modifications required to run on Python.
mhdawson commentedon Apr 3, 2019
There is a strategic initiative in https://github.com/nodejs/TSC/blob/master/Strategic-Initiatives.md being led by @thefourtheye so I think we can close this issue. @rvagg is that correct?
hayd commentedon Apr 3, 2019
Will updates still be made to this thread, or is there somewhere else to watch/offer help?
richardlau commentedon Apr 10, 2019
According to https://changelog.travis-ci.com/upcoming-python-default-version-update-96873
Raised nodejs/node#27166 to keep our Travis builds working because I don't think we'll be Python 3 compatible by April 16th (happy to be proven wrong).
rvagg commentedon Apr 24, 2019
Sadly we failed to get Node 12 to properly support Python 3.
Now we have Homebrew having problems with Node as they look at removing Python 2 entirely: nodejs/node-gyp#1337 (comment)
Someone's going to have to come up with a plan for what to do with Node 12 and Python 2/3—hopefully it doesn't have to involve breaking changes.
mhdawson commentedon Aug 15, 2019
Issue which tracks the ongoing work: nodejs/node#25789