Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIX build failure #23962

Closed
richardlau opened this issue Oct 29, 2018 · 13 comments
Closed

AIX build failure #23962

richardlau opened this issue Oct 29, 2018 · 13 comments
Assignees
Labels
aix Issues and PRs related to the AIX platform.

Comments

@richardlau
Copy link
Member

Looking into this. Opening for investigation/tracking.

Refs: #22237 (comment)

https://ci.nodejs.org/job/node-test-commit-aix/18405/nodes=aix61-ppc64/consoleFull

First few errors in build log:

01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:27:22: error: expected type-specifier
01:12:48            typename = std::enable_if_t<std::is_scalar<NativeT>::value>>
01:12:48                       ^
01:12:48 ../src/aliased_buffer.h:27:22: error: expected '>'
01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:65:31: error: template argument 3 is invalid
01:12:48                  v8::Uint8Array>& backing_buffer)
01:12:48                                ^
01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:123:41: error: template argument 3 is invalid
01:12:48      Reference(AliasedBuffer<NativeT, V8T>* aliased_buffer, size_t index)
01:12:48                                          ^
01:12:48 ../src/aliased_buffer.h:163:31: error: template argument 3 is invalid
01:12:48      AliasedBuffer<NativeT, V8T>* aliased_buffer_;
01:12:48 

cc @nodejs/platform-aix

@richardlau richardlau added the aix Issues and PRs related to the AIX platform. label Oct 29, 2018
@richardlau richardlau self-assigned this Oct 29, 2018
@addaleax
Copy link
Member

Looks like this comes from #23808, on which CI was run but not on AIX?

@richardlau
Copy link
Member Author

#23808 was the most recent change to aliased_buffer.h.

@richardlau
Copy link
Member Author

Looks like this comes from #23808, on which CI was run but not on AIX?

@refack @nodejs/build-infra Any ideas why the CI for #23808 (comment) (https://ci.nodejs.org/job/node-test-pull-request/18121/) didn't run on AIX?

@richardlau
Copy link
Member Author

Also from the build log https://ci.nodejs.org/job/node-test-commit-aix/18405/nodes=aix61-ppc64/consoleFull it looks like gcc 4.8.5 is being used, which doesn't look right:

00:44:45 + . ./build/jenkins/scripts/select-compiler.sh
00:44:45 + [  != DONT ]
00:44:45 + SELECT_ARCH=AIXPPC
00:44:45 + [ -z ]
00:44:45 /tmp/jenkins6046268851601114322.sh[29]: test: 0403-004 Specify a parameter with this command.
00:44:45 + [ AIXPPC = PPC64LE ]
00:44:45 + [ AIXPPC = S390X ]
00:44:45 + [ AIXPPC = AIXPPC ]
00:44:45 + echo Setting compiler for Node version  on AIX
00:44:45 Setting compiler for Node version  on AIX
00:44:45 + [  -gt 9 ]
00:44:45 + which gcc
00:44:45 + which g++
00:44:45 + which g++
00:44:45 + export CC=ccache /opt/freeware/bin/ccache/gcc CXX=ccache /opt/freeware/bin/ccache/g++ CXX_host=ccache /opt/freeware/bin/ccache/g++
00:44:45 + echo Compiler set to default at 4.8.5
00:44:45 Compiler set to default at 4.8.5

@richardlau
Copy link
Member Author

I think this error suggests the script is being run with the default korn shell instead of bash?

/tmp/jenkins6046268851601114322.sh[29]: test: 0403-004 Specify a parameter with this command.

@nodejs/build-infra Can someone check this on the AIX CI machines?

@richardlau
Copy link
Member Author

This is what the equivalent select-compiler.sh output on AIX is supposed to look like in a recent working build off master:

https://ci.nodejs.org/job/node-test-commit-aix/18415/nodes=aix61-ppc64/consoleFull

06:23:05 + . ./build/jenkins/scripts/select-compiler.sh
06:23:05 ++ '[' '' '!=' DONT ']'
06:23:05 ++ case $NODE_NAME in
06:23:05 ++ SELECT_ARCH=AIXPPC
06:23:05 ++ '[' -z ']'
06:23:05 +++ python tools/getnodeversion.py
06:23:05 ++ NODE_VERSION=12.0.0
06:23:05 +++ echo 12.0.0
06:23:05 +++ cut -d . -f 1
06:23:05 ++ NODEJS_MAJOR_VERSION=12
06:23:05 ++ '[' AIXPPC = PPC64LE ']'
06:23:05 ++ '[' AIXPPC = S390X ']'
06:23:05 ++ '[' AIXPPC = AIXPPC ']'
06:23:05 ++ echo 'Setting compiler for Node version 12 on AIX'
06:23:05 Setting compiler for Node version 12 on AIX
06:23:05 ++ '[' 12 -gt 9 ']'
06:23:05 ++ export LIBPATH=/home/iojs/gmake/opt/freeware/lib:/home/iojs/gcc-6.3.0-1/opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/lib
06:23:05 ++ LIBPATH=/home/iojs/gmake/opt/freeware/lib:/home/iojs/gcc-6.3.0-1/opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/lib
06:23:05 ++ export PATH=/home/iojs/gcc-6.3.0-1/opt/freeware/bin:/opt/freeware/bin/ccache:/home/iojs/jdk8/jre/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
06:23:05 ++ PATH=/home/iojs/gcc-6.3.0-1/opt/freeware/bin:/opt/freeware/bin/ccache:/home/iojs/jdk8/jre/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
06:23:05 +++ which gcc
06:23:05 +++ which g++
06:23:05 +++ which g++
06:23:05 ++ export 'CC=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/gcc' 'CXX=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++' 'CXX_host=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ CC='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/gcc'
06:23:05 ++ CXX='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ CXX_host='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ echo 'Compiler set to 6.3'
06:23:05 Compiler set to 6.3

We should be using gcc 6.3 on AIX for master (https://github.com/nodejs/node/blob/master/BUILDING.md#aix).

@refack
Copy link
Contributor

refack commented Oct 29, 2018

I fixed a bug in the build script around 4AM EDT at @joyeecheung's ping on IRC
image

tl;dr the default shell on AIX is not bash so some of the tests in select-compiler.sh weren't evaluated correctly, and the job ended up compiling with gcc4.8.

I'm closing this as resolved, but if anyone sees one of those fails here is the explanation.

@refack refack closed this as completed Oct 29, 2018
@refack
Copy link
Contributor

refack commented Oct 29, 2018

@refack Any ideas why the CI for #23808 (comment) (ci.nodejs.org/job/node-test-pull-request/18121) didn't run on AIX?

There were infra issues, so the AIX job was disabled for a few days. I turned it back on last night.

@richardlau
Copy link
Member Author

@refack Any ideas why the CI for #23808 (comment) (ci.nodejs.org/job/node-test-pull-request/18121) didn't run on AIX?

There were infra issues, so the AIX job was disabled for a few days. I turned it back on last night.

Is that logged anywhere? That means that for the few days it was disabled none of the CI's were running on AIX.

@refack
Copy link
Contributor

refack commented Oct 29, 2018

Is that logged anywhere? That means that for the few days it was disabled none of the CI's were running on AIX.

I'm not sure if I sent out a notification on IRC...
There was a notice that one machines whet offline for manual testing https://github.com/nodejs/build/projects/2

P.S. Anyway as policy whenever we (the Build WG) take a platform offline, we take responsibility to chase down regressions afterwards, if any arise.

@mhdawson
Copy link
Member

mhdawson commented Oct 29, 2018

@refack I don't quite understand, the last update to https://github.com/nodejs/build/commits/master/jenkins/scripts/select-compiler.sh was 27 days ago
based on the history?

@refack
Copy link
Contributor

refack commented Oct 29, 2018

@mhdawson AFAICT this is what happened:

  1. select-compiler.sh has always been silently failing on AIX and picking up gcc4.8. My assumption is that it's because it uses bash features, but the the aix Jenkins job was calling the default sh which is not bash.
  2. During the last few days we landed several PRs with code that actually requires gcc4.9.
  3. The AIX job was offline during the last few days
  4. When I brought it back online (2018-10-28 ~10PM EDT) it started surfacing the issue with the wrong gcc version.
  5. At 2018-10-29 ~4AM EDT Joyee pinged the build IRC channel
  6. I made the following change; (adding an explicit bash shebang #!/usr/bin/bash -xe)
  7. Everything works (that is select-compiler.sh picks gcc6 and the code in master compiles and passed the test suite).

@mhdawson
Copy link
Member

Ok thanks, got it. I suspect it was just using the wrong compiler since the last update I made. I had checked the release jobs carefully, but I guess I needed to check all of them (one of the reasons I was cautious about changing the script).

Separately, I think we should open an issue in the build repo any time we remove a platform from one of the jobs so that it gets maximum visibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aix Issues and PRs related to the AIX platform.
Projects
None yet
Development

No branches or pull requests

4 participants