Random build failures on Docker + Google Cloud Builder

[wolever]

I've noticed that the build will randomly fail when running on google cloud build running inside a node:9.3.0 Docker container.

In theory it should be possible to reproduce with docker run -it --rm node:9.3.0 yarn add sha3, but after ~30 builds I haven't seen the failure there.

In production, this seems to happen 1/10 or 1/20 builds, and re-running the build consistently fixes it.

This is the error:

Step #0: Exit code: 1
Step #0: Command: node-gyp rebuild
Step #0: Arguments: 
Step #0: Directory: /home/node/app/node_modules/sha3
Step #0: Output:
Step #0: gyp info it worked if it ends with ok
Step #0: gyp info using [email protected]
Step #0: gyp info using [email protected] | linux | x64
Step #0: gyp http GET https://nodejs.org/download/release/v9.3.0/node-v9.3.0-headers.tar.gz
Step #0: gyp http 200 https://nodejs.org/download/release/v9.3.0/node-v9.3.0-headers.tar.gz
Step #0: gyp http GET https://nodejs.org/download/release/v9.3.0/SHASUMS256.txt
Step #0: gyp http 200 https://nodejs.org/download/release/v9.3.0/SHASUMS256.txt
Step #0: gyp info spawn /usr/bin/python2
Step #0: gyp info spawn args [ '/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
Step #0: gyp info spawn args 'binding.gyp',
Step #0: gyp info spawn args '-f',
Step #0: gyp info spawn args 'make',
Step #0: gyp info spawn args '-I',
Step #0: gyp info spawn args '/home/node/app/node_modules/sha3/build/config.gypi',
Step #0: gyp info spawn args '-I',
Step #0: gyp info spawn args '/usr/local/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
Step #0: gyp info spawn args '-I',
Step #0: gyp info spawn args '/root/.node-gyp/9.3.0/include/node/common.gypi',
Step #0: gyp info spawn args '-Dlibrary=shared_library',
Step #0: gyp info spawn args '-Dvisibility=default',
Step #0: gyp info spawn args '-Dnode_root_dir=/root/.node-gyp/9.3.0',
Step #0: gyp info spawn args '-Dnode_gyp_dir=/usr/local/lib/node_modules/npm/node_modules/node-gyp',
Step #0: gyp info spawn args '-Dnode_lib_file=/root/.node-gyp/9.3.0/<(target_arch)/node.lib',
Step #0: gyp info spawn args '-Dmodule_root_dir=/home/node/app/node_modules/sha3',
Step #0: gyp info spawn args '-Dnode_engine=v8',
Step #0: gyp info spawn args '--depth=.',
Step #0: gyp info spawn args '--no-parallel',
Step #0: gyp info spawn args '--generator-output',
Step #0: gyp info spawn args 'build',
Step #0: gyp info spawn args '-Goutput_dir=.' ]
Step #0: make: Entering directory '/home/node/app/node_modules/sha3/build'
Step #0: gyp info spawn make
Step #0: gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
Step #0: CXX(target) Release/obj.target/sha3/src/addon.o
Step #0: In file included from /root/.node-gyp/9.3.0/include/node/node.h:63:0,
Step #0: from ../src/addon.cpp:1:
Step #0: /root/.node-gyp/9.3.0/include/node/v8.h:15:0: error: unterminated #ifndef
Step #0: #ifndef INCLUDE_V8_H_
Step #0: ^
Step #0: In file included from ../src/addon.cpp:4:0:
Step #0: ../../nan/nan.h:135:0: warning: "TYPE_CHECK" redefined
Step #0: #define TYPE_CHECK(T, S) \
Step #0: ^
Step #0: In file included from /root/.node-gyp/9.3.0/include/node/node.h:63:0,
Step #0: from ../src/addon.cpp:1:
Step #0: /root/.node-gyp/9.3.0/include/node/v8.h:160:0: note: this is the location of the previous definition
Step #0: #define TYPE_CHECK(T, S) \
Step #0: ^
Step #0: In file included from /root/.node-gyp/9.3.0/include/node/node.h:64:0,
Step #0: from ../src/addon.cpp:1:
Step #0: /root/.node-gyp/9.3.0/include/node/v8-platform.h:13:1: error: expected identifier before 'namespace'
Step #0: namespace v8 {
Step #0: ^
Step #0: /root/.node-gyp/9.3.0/include/node/v8-platform.h:13:1: error: expected '}' before 'namespace'
Step #0: /root/.node-gyp/9.3.0/include/node/v8-platform.h:13:1: error: expected unqualified-id before 'namespace'
Step #0: In file included from /usr/lib/gcc/x86_64-linux-gnu/4.9/include/x86intrin.h:27:0,
Step #0: from /usr/include/x86_64-linux-gnu/c++/4.9/bits/opt_random.h:33,
Step #0: from /usr/include/c++/4.9/random:50,
Step #0: from /usr/include/c++/4.9/bits/stl_algo.h:66,
Step #0: from /usr/include/c++/4.9/algorithm:62,
Step #0: from ../../nan/nan.h:54,
Step #0: from ../src/addon.cpp:4:
Step #0: /usr/lib/gcc/x86_64-linux-gnu/4.9/include/ia32intrin.h:53:25: error: expected '}' before end of line
Step #0: #pragma GCC push_options
Step #0: ^
Step #0: /usr/lib/gcc/x86_64-linux-gnu/4.9/include/ia32intrin.h:53:25: error: expected unqualified-id before end of line
Step #0: /usr/lib/gcc/x86_64-linux-gnu/4.9/include/ia32intrin.h:53:25: error: expected '}' before end of line
Step #0: /usr/lib/gcc/x86_64-linux-gnu/4.9/include/ia32intrin.h:53:25: error: expected declaration before end of line
Step #0: sha3.target.mk:99: recipe for target 'Release/obj.target/sha3/src/addon.o' failed
Step #0: make: Leaving directory '/home/node/app/node_modules/sha3/build'
Step #0: make: *** [Release/obj.target/sha3/src/addon.o] Error 1
Step #0: gyp ERR! build error 
Step #0: gyp ERR! stack Error: `make` failed with exit code: 2
Step #0: gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:258:23)
Step #0: gyp ERR! stack at ChildProcess.emit (events.js:159:13)
Step #0: gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
Step #0: gyp ERR! System Linux 4.4.0-124-generic
Step #0: gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
Step #0: gyp ERR! cwd /home/node/app/node_modules/sha3
Step #0: gyp ERR! node -v v9.3.0
Step #0: gyp ERR! node-gyp -v v3.6.2
Step #0: gyp ERR! not ok

[canterberry]

Judging from the stack trace, this appears to be an error with the node-gyp toolchain. I can see in the logs that the headers and SHA256SUMS are being downloaded, but I don't see where their integrity is being verified. Because this issue is happening intermittently, it seems possible that, occasionally, the downloaded headers are corrupted or incomplete. This would explain some of the errors, like the unterminated ifndef we see as the first error.

It's possible that this is a known issue with node-gyp, npm, and/or node, which may be fixed in a more recent version. If this is the case, then upgrading could remedy the flakiness.

[canterberry]

💡 We've seen some reports that this issue can be solved by installing the build-essential package (Linux) or the Mac equivalent, Xcode Command-Line Tools.

[canterberry]

Considering this as a duplicate of #27.

[wolever]

I don't believe (although please correct my if I'm wrong) this is a duplicate of #27, as it seems like that issue is related to the total lack of build-essential, where this is a sporadic error that only shows up occasionally.

It seems like [email protected] doesn't require a build step, though, so upgrading to that may solve the issue.

I'll give that a try, although since I haven't been able to reproduce this error with any regularity, I won't be able to report back until things haven't broken for some time.

[canterberry]

@wolever Thanks for reporting back. Yeah, it's definitely caused by the post-install native code compilation steps in v1.x and the undeclared/implicit dependencies on build-essential. Upgrading to v2.x should solve that, for sure, since it doesn't have native code compilation steps or dependencies on build-essential. 👌