RHEL7, v0.25.1, "CXXABI_1.3.8 not found"

[jmac105]

Have you ensured the platform and version of Node.js used for npm install is the same as the platform and version of Node.js used at runtime?

Yes

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?

Yes, 0.25.1

If you are installing as a root or sudo user, have you tried with the npm install --unsafe-perm flag?

Tried with & without

If you are using the ignore-scripts feature of npm, have you tried with the npm install --ignore-scripts=false flag?

Not using, but have tried

What is the complete output of running npm install --verbose sharp? Have you checked this output for useful error messages?

Cut out a lot of the noise around other packages etc... no errors

[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm info lifecycle [email protected]~install: [email protected]
[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] 
[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] > [email protected] install /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp
[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] > (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)
[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] 
[12:56:07]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.9.1/libvips-8.9.1-linux-x64.tar.gz
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info begin Prebuild-install version 5.3.3
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info looking for cached prebuild @ /data/teamcity-agent/work/ed14a4d9a022f175/_prebuilds/0a8714-sharp-v0.25.1-napi-v3-linux-x64.tar.gz
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install http request GET https://github.com/lovell/sharp/releases/download/v0.25.1/sharp-v0.25.1-napi-v3-linux-x64.tar.gz
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install http 200 https://github.com/lovell/sharp/releases/download/v0.25.1/sharp-v0.25.1-napi-v3-linux-x64.tar.gz
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info downloading to @ /data/teamcity-agent/work/ed14a4d9a022f175/_prebuilds/0a8714-sharp-v0.25.1-napi-v3-linux-x64.tar.gz.6854-1bb64237a6c75.tmp
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info renaming to @ /data/teamcity-agent/work/ed14a4d9a022f175/_prebuilds/0a8714-sharp-v0.25.1-napi-v3-linux-x64.tar.gz
[12:56:10]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info unpacking @ /data/teamcity-agent/work/ed14a4d9a022f175/_prebuilds/0a8714-sharp-v0.25.1-napi-v3-linux-x64.tar.gz
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info unpack resolved to /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp/build/Release/sharp.node
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] prebuild-install info install Successfully installed prebuilt binary!
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm verb lifecycle [email protected]~install: unsafe-perm in lifecycle true
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm verb lifecycle [email protected]~install: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp/node_modules/.bin:/data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm verb lifecycle [email protected]~install: CWD: /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm timing audit submit Completed in 1074ms
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm http fetch POST 200 https://internaldomain/security/audits/quick 1075ms
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm timing audit body Completed in 0ms
[12:56:11]	[npm install sharp --ignore-scripts=false --verbose --unsafe-perm] npm timing action:install Completed in 3273ms

What is the output of running npx envinfo --binaries --system?

[13:17:15]	[Step 1/19]   System:
[13:17:15]	[Step 1/19]     OS: Linux 3.10 CentOS Linux 7 (Core)
[13:17:15]	[Step 1/19]     CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
[13:17:15]	[Step 1/19]     Memory: 331.74 MB / 3.52 GB
[13:17:15]	[Step 1/19]     Container: Yes
[13:17:15]	[Step 1/19]     Shell: 4.2.46 - /bin/bash
[13:17:15]	[Step 1/19]   Binaries:
[13:17:15]	[Step 1/19]     Node: 12.15.0 - /usr/bin/node
[13:17:15]	[Step 1/19]     npm: 6.13.7 - /usr/bin/npm
[13:17:15]	[Step 1/19] 
[13:17:15]	[Step 1/19] Process exited with code 0

Output of yum info glibc

[12:56:01]	[Step 1/19] Installed Packages
[12:56:01]	[Step 1/19] Name        : glibc
[12:56:01]	[Step 1/19] Arch        : x86_64
[12:56:01]	[Step 1/19] Version     : 2.17
[12:56:01]	[Step 1/19] Release     : 292.el7
[12:56:01]	[Step 1/19] Size        : 13 M
[12:56:01]	[Step 1/19] Repo        : installed
[12:56:01]	[Step 1/19] From repo   : base
[12:56:01]	[Step 1/19] Summary     : The GNU libc libraries
[12:56:01]	[Step 1/19] URL         : http://www.gnu.org/software/glibc/
[12:56:01]	[Step 1/19] License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
[12:56:01]	[Step 1/19] Description : The glibc package contains standard libraries which are used by
[12:56:01]	[Step 1/19]             : multiple programs on the system. In order to save disk space and
[12:56:01]	[Step 1/19]             : memory, as well as to make upgrading easier, common system code is
[12:56:01]	[Step 1/19]             : kept in one place and shared between programs. This particular
[12:56:01]	[Step 1/19]             : package contains the most important sets of shared libraries: the
[12:56:01]	[Step 1/19]             : standard C library and the standard math library. Without these
[12:56:01]	[Step 1/19]             : two libraries, a Linux system will not function.

Environment:

Teamcity agent (CI system) running on CentOS 7

The NPM install of sharp and all other packages is working just fine. But as soon as the applications requires sharp we get the following error. This has only started happening since updating from 0.24.x, application will not work with version 0.25.1 of sharp but is fine with 0.24.1

[12:56:48]	[Step 7/19] Error requiring file: /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/test/manifest.js
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] Something went wrong installing the "sharp" module
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp/build/Release/sharp.node)
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] - Remove the "node_modules/sharp" directory then run
[12:56:48]	[Step 7/19]   "npm install --ignore-scripts=false --verbose" and look for errors
[12:56:48]	[Step 7/19] - Consult the installation documentation at https://sharp.pixelplumbing.com/install
[12:56:48]	[Step 7/19] - Search for this error at https://github.com/lovell/sharp/issues
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] Error: 
[12:56:48]	[Step 7/19] Something went wrong installing the "sharp" module
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp/build/Release/sharp.node)
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19] - Remove the "node_modules/sharp" directory then run
[12:56:48]	[Step 7/19]   "npm install --ignore-scripts=false --verbose" and look for errors
[12:56:48]	[Step 7/19] - Consult the installation documentation at https://sharp.pixelplumbing.com/install
[12:56:48]	[Step 7/19] - Search for this error at https://github.com/lovell/sharp/issues
[12:56:48]	[Step 7/19] 
[12:56:48]	[Step 7/19]     at Object.<anonymous> (/data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/sharp/lib/constructor.js:34:9)
[12:56:48]	[Step 7/19]     at Module._compile (internal/modules/cjs/loader.js:955:30)
[12:56:48]	[Step 7/19]     at Object.require.extensions.<computed> [as .js] (/data/teamcity-agent/work/ed14a4d9a022f175/sourceCode/node_modules/@hapi/lab/lib/coverage.js:127:28)

[lovell]

Hi, sharp defines _GLIBCXX_USE_CXX11_ABI=0 during compilation to prevent this problem from occurring:

sharp/binding.gyp

Lines 127 to 130 in 4894c10

	['OS == "linux"', {
	'defines': [
	'_GLIBCXX_USE_CXX11_ABI=0'
	],

I suspect this relates to the internal N-API migration #1282 included from v0.25.0.

When I compile locally I see the expected '-D_GLIBCXX_USE_CXX11_ABI=0' having been defined, so I'm unsure where this might be coming from.

[jmac105]

Thanks for looking at it. Let me know if you need any other info to help

[lovell]

It looks like migrating to N-API has placed a new dependency on the __cxa_throw_bad_array_new_length symbol, which is unavailable in the version of the C++ runtime provided by RHEL7 and its variants.

$ nm build/Release/sharp.node | grep CXXABI_1.3.8
U __cxa_throw_bad_array_new_length@@CXXABI_1.3.8

This might relate to exception handling in N-API and the use of throw Napi::Error::New(...) but more research is required.

A possible workaround on CentOS 7 is to use the npm install --build-from-source flag.

https://sharp.pixelplumbing.com/install#building-from-source

[mattpen]

@lovell - I have encountered the same problem today (RHEL 7, sharp v0.25.1, message "CXXABI_1.3.8 not found" on require('sharp')).

I can confirm that npm install --build-from-source is a viable work around. However, our server process needs to frequently run npm operations, so this isn't an ideal long term solution in our case. Do you expect there to be a fix that avoids this work around? Thanks!

[lovell]

The __cxa_throw_bad_array_new_length symbol is used where the C++ source has new Class[n] notation, which is occurring within the node-addon-api layer of Node.js itself here:

https://github.com/nodejs/node-addon-api/blob/d43da6ac2b4bf8519afcad08b59a9223257e6979/napi-inl.h#L2715

RHEL provides gcc v4.8.5 but __cxa_throw_bad_array_new_length was added in v4.9. Setting _GLIBCXX_USE_CXX11_ABI=0 doesn't appear to prevent newer versions of gcc from outputting this symbol.

The "fix" for this is probably going to be to generate the prebuilt binaries on RHEL/CentOS 7 and gcc v4.8.5 instead of Ubuntu Xenial and gcc v5.3.1 as is currently the case.

[lovell]

Commit d8426b1 switches the Linux x64 glibc CI environment to use CentOS 7, so the prebuilt binaries provided by the future v0.25.2 release will be compatible again with RHEL7.

[lovell]

v0.25.2 is now available, using Centos 7 for the prebuilt binaries.

[lovell]

For anyone coming here with a similar error using sharp v0.33.0 or later, that's because the prebuilt binaries no longer support CentOS 7.

The supported glibc and musl libc versions are documented at https://sharp.pixelplumbing.com/install#prebuilt-binaries

There's some background at lovell/sharp-libvips#176 but the summary is that CentOS is essentially EOL and you'll experience better performance on the same hardware by switching to a more modern Linux.