Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move minimum supported gcc to gcc 12 for Node.js 23 #3806

Open
richardlau opened this issue Jul 5, 2024 · 17 comments
Open

move minimum supported gcc to gcc 12 for Node.js 23 #3806

richardlau opened this issue Jul 5, 2024 · 17 comments

Comments

@richardlau
Copy link
Member

gcc no longer support gcc 10. I think it's time we plan to move up.

Due to compiler issues, we've already had to move Linux on s390x (LinuxONE) builds to use gcc 12: #3630
(FWIW the Red Hat team maintaining the V8 ports to Power and z also use gcc 12.)

My proposal is to move to gcc 12 as not all platforms that we support that we use gcc to build on have gcc 13 yet (e.g. AIX, IBM i). It would be easier to have the same baseline across all supported platforms, although it is not strictly necessarily (in the past we have had the minimum vary by platform, but in practice that forces Node.js to be compatible with the lowest out of the set).

Compilation of main is currently broken on gcc 12.2 (although not on RHEL 8) but would be fixed by landing nodejs/node#53728.

For releases on Linux we'd continue to use RHEL 8 (now in maintenance but good through to 2029) and use gcc-toolset-12 instead of gcc-toolset-10. This would mean we keep compatibility with glibc 2.28 and libstdc++ 6.0.25 (the base in RHEL 8) and Node.js 23 would continue to run in the same places that Node.js 22, 20 and 18 does. The exception to this is likely to be 32-bit arm (armv7l) which we use a custom cross-compiler for (as RHEL doesn't support armv7l). We would either need to generate a new gcc 12 cross-compiler (and check what that would mean in terms of run-time compatibility) or perhaps discuss whether armv7l is still a platform we continue to supply official binaries for.

@targos
Copy link
Member

targos commented Jul 28, 2024

I opened nodejs/node#54081 to change the warning threshold in Node.js.

It includes a temporary commit that makes the build exit with code 1 when the version is too low so we can easily assess the impact on CI:

https://ci.nodejs.org/job/node-test-pull-request/60690/

targos added a commit to targos/nodejs-build that referenced this issue Jul 28, 2024
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Aug 5, 2024
Update the warning threshold for GCC to 12.2 starting from Node.js 23.
Builds can still proceed with earlier versions of GCC, but are not
guaranteed to work.

PR-URL: #54081
Refs: nodejs/build#3806
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
targos added a commit that referenced this issue Aug 6, 2024
`gcc-12` is not available for this version.

Refs: #3806
targos added a commit that referenced this issue Aug 6, 2024
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Aug 16, 2024
Use gcc 12 to build the official binaries for Node.js 23 onwards on
AIX. Note that this will require `libstdc++12`, available from the
AIX toolbox.

PR-URL: #54338
Refs: nodejs/build#3806
Refs: nodejs/build#3858
Refs: #54081
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Ulises Gascón <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Michaël Zasso <[email protected]>
richardlau added a commit that referenced this issue Aug 16, 2024
Add gcc 12 to RHEL/UBI and Ubuntu containers.

Update arm containers from Ubuntu 20.04 to 22.04.

Remove unused CentOS 7 container.

Refs: #3806
Refs: #3846
@targos
Copy link
Member

targos commented Aug 26, 2024

New CI with error if gcc<12: https://ci.nodejs.org/job/node-test-commit/73486/

@targos
Copy link
Member

targos commented Aug 26, 2024

@targos
Copy link
Member

targos commented Aug 26, 2024

smartos: https://ci.nodejs.org/job/node-test-commit-smartos/56305/nodes=smartos20-64/console

More tricky as it doesn't seem to have gcc-12 installed

@targos
Copy link
Member

targos commented Aug 26, 2024

@targos
Copy link
Member

targos commented Aug 26, 2024

arm cross-compiler: https://ci.nodejs.org/job/node-cross-compile/49474/

@richardlau
Copy link
Member Author

arm cross-compiler: https://ci.nodejs.org/job/node-cross-compile/49474/

FWIW I'm trying (so far without success 😞) to update the cross compiler used.

I've generated a new gcc 12 based cross-compiler using crosstools-ng: https://github.com/richardlau/rpi-newer-crosstools/commits/gcc-12.3.0-glibc-2.28/

Unfortunately I'm getting undefined symbol errors when attempting to cross-compile for 32-bit arm in a RHEL 8 x64 container with gcc 12 host compiler (via gcc-toolset-12):

/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: /home/nodejs/node/out/Release/torque: hidden symbol `_ZNSt7__cxx1112basic_stringIwSt11char_traitsIwESaIwEE9_M_mutateEjjPKwj' isn't defined
/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[1]: *** [tools/v8_gypfiles/torque.host.mk:216: /home/nodejs/node/out/Release/torque] Error 1
make[1]: *** Waiting for unfinished jobs....

My suspicion is that there's a mismatch between the version of libstdc++ in the cross-compiler vs in RHEL 8 (for the host compiler). I'm attempting (building at the moment) to see what happens in a RHEL 9 container (which should have a newer libstdc++ but may still be behind the one from the cross-compiler since RHEL 9 is based on gcc 11).

If that doesn't work (or even if it does), other options are:

  • Don't cross-compile and instead "natively" compile 32-bit binaries in an armv7 container running on the arm64 machines.
    • We'd need to convert the arm64 release machine into a Docker host (and potentially have to containerize the Linux arm64 build or find another way to prevent interference between the arm64 and armv7 builds). The test CI currently builds 32-bit arm binaries both cross-compiled (replicating the existing release set up) and in an Ubuntu 22.04 armv7l container.
    • Advantage of this approach is that we could then eventually retire the cross-compile set up after Node.js 22 goes End-of-Life.
  • Stop producing 32-bit arm binaries for Node.js 23. Possibly move it over to unofficial-builds.

@richardlau
Copy link
Member Author

I'm attempting (building at the moment) to see what happens in a RHEL 9 container (which should have a newer libstdc++ but may still be behind the one from the cross-compiler since RHEL 9 is based on gcc 11).

Update: I got a binary using a RHEL 9 x64 container with gcc-toolset-12 as the host compiler and x64-gcc-12.3.0-glibc-2.28 from https://github.com/richardlau/rpi-newer-crosstools/commits/gcc-12.3.0-glibc-2.28/ which looks like a 32-bit arm binary:

bash-5.1$ /opt/rpi-newer-crosstools/x64-gcc-12.3.0-glibc-2.28/arm-rpi-linux-gnueabihf/bin/arm-rpi-linux-gnueabihf-objdump -f out/Release/node

out/Release/node:     file format elf32-littlearm
architecture: arm, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x009c4378

bash-5.1$

I'll do more checks next week (like verifying the binary runs on armv7🙂 and checking compatibility). 🤞

targos added a commit to targos/nodejs-build that referenced this issue Sep 1, 2024
@richardlau
Copy link
Member Author

I'll do more checks next week (like verifying the binary runs on armv7🙂 and checking compatibility). 🤞

The resultant binary works in

and fails (due to libstdc++) in

root@f6c93998c241:/tmp/armv7-test# ./node
./node: /lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ./node)
./node: /lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./node)
root@f6c93998c241:/tmp/armv7-test#

So it looks like we can retain support for 32-bit arm by using a RHEL 9 container with the generated gcc 12 cross-compiler . This looks like the quickest/least effort change:

  • Upstream the compiler into https://github.com/rvagg/rpi-newer-crosstools (I think I have write access)
  • Add a RHEL 9 arm cross container to the existing Docker hosts with the update rpi-newer-crosstools
  • Update cross-compile CI job to add a matrix entry for gcc 12.
  • cc-selector.sh and VersionSelectorScript changes for Node.js 23+.

Alternative, and more work, would be to move to compiling the 32-bit arm binary in a 32-bit arm container. This would mean:

  • Convert existing release-osuosl-rhel8-arm64-1 machine into a Docker host.
  • Add RHEL 8 container for arm64 builds to the converted release machine. We should be able to reuse the Dockerfile for the RHEL 8 arm64 containers used in the test CI.
  • Add Ubuntu 22.04 container for armv7 (32-bit) builds to the converted release machine. Ubuntu 22.04 is on a slightly earlier glibc version (2.35) to Debian 12 (2.36) so we'd need to compile on Ubuntu 22.04 if we want to maintain compatibility with both. Again we are already running armv7 Ubuntu 22.04 containers in the test CI so we should be able to reuse the existing Dockerfile.
  • Update VersionSelectorScript for Node.js 23.

@richardlau
Copy link
Member Author

I've opened #3889 to add Ansible changes to build 32-bit ARM binaries with the updated gcc 12 cross compiler.

richardlau added a commit to richardlau/build that referenced this issue Sep 5, 2024
richardlau added a commit to richardlau/build that referenced this issue Sep 5, 2024
@richardlau
Copy link
Member Author

@targos
Copy link
Member

targos commented Sep 5, 2024

For SmartOS I saw that we have hosts with newer versions but they are not used by the Jenkins job

@richardlau
Copy link
Member Author

richardlau commented Sep 5, 2024

For SmartOS I saw that we have hosts with newer versions but they are not used by the Jenkins job

Being tracked under #3731 as part of the Equinix migration.

@targos
Copy link
Member

targos commented Sep 9, 2024

@targos
Copy link
Member

targos commented Sep 10, 2024

Only SmartOS remains.

@richardlau
Copy link
Member Author

For the new MNX hosted SmartOS machines which are meant to replace the equinix_mnx ones, it looks like SmartOS 23 has gcc 13 by default:

$ ssh test-mnx-smartos23-x64-1 gcc --version
gcc (GCC) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$

while SmartOS 21 is 10:

$ ssh test-mnx-smartos21-x64-1 gcc --version
gcc (GCC) 10.4.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$

So perhaps this will be taken care of when the build issues on the new machines are resolved in #3731? cc @jperkin @ryanaslett

richardlau pushed a commit that referenced this issue Sep 13, 2024
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Sep 20, 2024
Update the list of toolchains used to build the official Node.js
binaries for Node.js 23 onwards.

PR-URL: #54967
Refs: nodejs/build#3806
Refs: #54081
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Michaël Zasso <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants