Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: intermittent failure running add-on tests on AIX #8239

Closed
mhdawson opened this issue Aug 23, 2016 · 13 comments
Closed

test: intermittent failure running add-on tests on AIX #8239

mhdawson opened this issue Aug 23, 2016 · 13 comments
Labels
test Issues and PRs related to the tests.

Comments

@mhdawson
Copy link
Member

mhdawson commented Aug 23, 2016 •
edited
Loading

  • Version: master
  • Platform: AIX
  • Subsystem: addons

Looks like the AIX jobs are failing intermittently due to a problem compiling add-ons.

It looks to me like the problem is that the step of creating the exp file has not completed before the add-on tests start to run.

For example in a failing run: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/429/consoleFull

I see

 SOLINK_MODULE(target) Release/obj.target/addon.node
  LD_LIBRARY_PATH=/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/lib.host:/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release; sh tools/create_expfile.sh "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target" "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node.exp"
Searching /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target to write out expfile to /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node.exp
ld: 0711-317 ERROR: Undefined symbol: .v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::String::NewStringType, int)
ld: 0711-317 ERROR: Undefined symbol: .v8::Function::Call(v8::Local<v8::Value>, int, v8::Local<v8::Value>*)
ld: 0711-317 ERROR: Undefined symbol: .v8::Isolate::GetCurrent()
ld: 0711-317 ERROR: Undefined symbol: .v8::HandleScope::HandleScope(v8::Isolate*)
ld: 0711-317 ERROR: Undefined symbol: .v8::FunctionTemplate::New(v8::Isolate*, void (*)(v8::FunctionCallbackInfo<v8::Value> const&), v8::Local<v8::Value>, v8::Local<v8::Signature>, int)
ld: 0711-317 ERROR: Undefined symbol: .v8::FunctionTemplate::GetFunction()
ld: 0711-317 ERROR: Undefined symbol: .v8::Function::SetName(v8::Local<v8::String>)
ld: 0711-317 ERROR: Undefined symbol: .v8::Object::Set(v8::Local<v8::Value>, v8::Local<v8::Value>)
ld: 0711-317 ERROR: Undefined symbol: .v8::HandleScope::~HandleScope()
ld: 0711-317 ERROR: Undefined symbol: .node_module_register
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
collect2: error: ld returned 8 exit status
gmake[2]: *** [Release/obj.target/addon.node] Error 1

Were as in a successful run I see the line:

Searching /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target to write out expfile to /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node.exp

before the add-on tests start to run. For example:

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/431/consoleFull
@mhdawson mhdawson added the test Issues and PRs related to the tests. label Aug 23, 2016
@mhdawson
Copy link
Member Author

The target for AIX is what triggers the building of the exp file but don't see why tests would start before it was complete

'targets': [
        {
          'target_name': 'node',
          'type': 'executable',
          'dependencies': ['<(node_core_target_name)', 'node_exp'],

          'include_dirs': [
            'src',
            'deps/v8/include',
          ],

          'sources': [
            'src/node_main.cc',
            '<@(library_files)',
            # node.gyp is added to the project by default.
            'common.gypi',
          ],

          'ldflags': ['-Wl,-bbigtoc,-bE:<(PRODUCT_DIR)/node.exp'],
        },
        {
          'target_name': 'node_exp',
          'type': 'none',
          'dependencies': [
            '<(node_core_target_name)',
          ],
          'actions': [
            {
              'action_name': 'expfile',
              'inputs': [
                '<(OBJ_DIR)'
              ],
              'outputs': [
                '<(PRODUCT_DIR)/node.exp'
              ],
              'action': [
                'sh', 'tools/create_expfile.sh',
                      '<@(_inputs)', '<@(_outputs)'
              ],
            }
          ]
        }

@richardlau
Copy link
Member

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/429/consoleFull has two occurrences of

Searching /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target to write out expfile to /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node.exp

which suggests the node_exp target is being run twice.

@mhdawson
Copy link
Member Author

@bnoordhuis any chance you are familiar enough with gyp to see if my thought that its a dependency problem makes sense and what it might be ?

@bnoordhuis
Copy link
Member

Does adding 'process_outputs_as_sources': 1 to the node_exp target help? I suspect the problem is that gyp can't divine it should wait for the target to finish before moving on to the link phase.

@Trott Trott mentioned this issue Aug 25, 2016
Closed
@mhdawson
Copy link
Member Author

Thanks @bnoordhuis I'll give that a try.

@mhdawson
Copy link
Member Author

Did 6 runs with that change and the add-on tests all ran ok in those cases.

PR with the fix: #8272

@mhdawson mhdawson mentioned this issue Aug 25, 2016
Closed
2 tasks
@mhdawson
Copy link
Member Author

Reopening to investigate as I think I'm still seeing failures in the builds despite having gotten 6 successes in a row when testing it out.

@mhdawson
Copy link
Member Author

mhdawson commented Aug 26, 2016 •
edited
Loading

Since it still seems to be occurring, I've temporarily pulled it out of node-test-commit. Guess I was just really unlucky that those 6 runs passed earlier.

@mhdawson
Copy link
Member Author

mhdawson commented Aug 26, 2016 •
edited
Loading

Looks like there is something special going on with respect to dependencies for the addon tests.

# .buildstamp and .docbuildstamp need $(NODE_EXE) but cannot depend on it
# directly because it calls make recursively.  The parent make cannot know
# if the subprocess touched anything so it pessimistically assumes that
# .buildstamp and .docbuildstamp are out of date and need a rebuild.
# Just goes to show that recursive make really is harmful...
# TODO(bnoordhuis) Force rebuild after gyp update.
build-addons: $(NODE_EXE) test/addons/.buildstamp

Now need to see if this is the root cause of the problem.

@mhdawson mhdawson reopened this Aug 26, 2016
@mhdawson
Copy link
Member Author

Testing out change in this branch https://github.com/mhdawson/io.js/tree/aixaddon2

@mhdawson
Copy link
Member Author

mhdawson commented Aug 26, 2016 •
edited
Loading

Ok so far promising, launching 10 runs on that branch to see how it goes

Also broader run to validate on other platforms: https://ci.nodejs.org/job/node-test-commit/4784/

@mhdawson
Copy link
Member Author

Ok another PR which I believe fixes it #8285

@mhdawson
Copy link
Member Author

Believe this is fixed now. AIX is back in since yesterday and I don't see any failures. I'll keep an eye on it for the next few days but closing this issue.

Fishrock123 pushed a commit to Fishrock123/node that referenced this issue Sep 8, 2016
@mhdawson @Fishrock123
build: fix dependencies on AIX
fbb59b2 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: nodejs#8239
PR-URL: nodejs#8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
Fishrock123 pushed a commit that referenced this issue Sep 9, 2016
@mhdawson @Fishrock123
build: fix dependencies on AIX
bc9d2fb 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: #8239
PR-URL: #8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
MylesBorins pushed a commit that referenced this issue Sep 30, 2016
@mhdawson
build: fix dependencies on AIX
92dc1f3 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: #8239
PR-URL: #8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
MylesBorins pushed a commit that referenced this issue Oct 10, 2016
@mhdawson
build: fix dependencies on AIX
3345aab 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: #8239
PR-URL: #8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
rvagg pushed a commit that referenced this issue Oct 18, 2016
@mhdawson @rvagg
build: fix dependencies on AIX
d9457a7 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: #8239
PR-URL: #8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
MylesBorins pushed a commit that referenced this issue Oct 26, 2016
@mhdawson @MylesBorins
build: fix dependencies on AIX
8dfab3a 
Addon tests were starting to run before the node exp file
creation was complete. Add process_outputs_as_sources to avoid
this.

Fixes: #8239
PR-URL: #8272
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bnoordhuis @richardlau @mhdawson