test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) #14206

refack · 2017-07-12T21:18:36Z

Version: master
Platform: multiple
Subsystem: test,V8

Moving nodejs/build#789 to core
The test has intermitently failed on:

ppcbe-ubuntu1404
armv7-wheezy
centos5-64
docker_alpine34-64

One of the more interesting outputs (from docker_alpine34-64):

[----------] 3 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (384 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
[       OK ] EnvironmentTest.AtExitWithArgument (388 ms)
[ RUN      ] EnvironmentTest.MultipleEnvironmentsPerIsolate


#
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#
Received signal 4 ILL_ILLOPN 55c3c9aafe69

The more common output:

[----------] 3 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (362 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
[       OK ] EnvironmentTest.AtExitWithArgument (354 ms)
[ RUN      ] EnvironmentTest.MultipleEnvironmentsPerIsolate
Build was aborted

This is the test code

TEST_F(EnvironmentTest, MultipleEnvironmentsPerIsolate) {
const v8::HandleScope handle_scope(isolate_);
const Argv argv;
Env env1 {handle_scope, isolate_, argv};
Env env2 {handle_scope, isolate_, argv};

AtExit(*env1, at_exit_callback1);
AtExit(*env2, at_exit_callback2);
RunAtExit(*env1);
EXPECT_TRUE(called_cb_1);
EXPECT_FALSE(called_cb_2);

RunAtExit(*env2);
EXPECT_TRUE(called_cb_2);
}

From
test/cctest/test_environment.cc

The text was updated successfully, but these errors were encountered:

addaleax · 2017-07-12T21:22:17Z

Quoting #13861 (comment):

Running the cctest locally makes valgrind complain in a way that points to us not cleaning up libuv handles properly in the multi-Environment tests, similar to the problem fixed by d5db4d2. If I’m right (and valgrind complaining should be a good sign for that), this is not a problem that’s going to effect regular users, and more likely to be a problem with the test itself.

I’ve also previously asked whether it’s possible to get a core dump from that machine to make sure; I think you might be the best person to have an answer?

I think all we need is somebody who has the time to dig into the valgrind logs to figure this out.

Trott · 2017-07-13T14:55:45Z

Still happening, of course: https://ci.nodejs.org/job/node-test-commit-plinux/10186/nodes=ppcbe-ubuntu1404/console

Is anyone on this? Anyone especially sensible to loop in on it?

refack · 2017-07-13T15:07:29Z

I'll try on Windows, but !@#$@%#$%^ MMMV (it might not even be a problem here 🤷‍♂️ )

refack · 2017-07-13T15:09:56Z

We can try to call @indutny to the flag (it's partly his fault, JK nothing but ❤️ )

mhdawson · 2017-07-13T15:49:51Z

Created this issue to request access to the BE machine were it recreates most easily for @jBarz : nodejs/build#795

He's tried to reproduce locally with no luck.

jBarz · 2017-07-13T17:38:05Z

I can reproduce on build machine. However, it is intermittent (about once in 5 tries)

mhdawson · 2017-07-13T22:58:34Z

I wondering if we should exclude this test until we can figure out what is going on. It seems to happen quire regularly for PPC and alpine.

jBarz · 2017-07-14T01:22:17Z

Continuing to investigate. So far I have found it to hang at the following location in api.cc from v8 when initializing the isolate.

Isolate::Scope isolate_scope(v8_isolate);
 if (params.entry_hook || !i::Snapshot::Initialize(isolate)) {
   isolate->Init(NULL);  // <--- Hangs here
 }

refack · 2017-07-14T02:49:06Z

I wondering if we should exclude this test until we can figure out what is going on. It seems to happen quire regularly for PPC and alpine.

If we have a way to make sure it doesn't get neglected, I'm +1

mhdawson · 2017-07-14T19:04:45Z

Any objects to excluding the test ? If I don't see anybody and nobody beats me too it I'll look at a PR for excluding on Monday.

Trott · 2017-07-15T02:14:35Z

Any objects to excluding the test ? If I don't see anybody and nobody beats me too it I'll look at a PR for excluding on Monday.

We have Code + Learn in Shanghai in about 30 hours. Hard to say how things will or won't work (lots of variables at play here that we haven't dealt with before), but I would expect at least 25 and probably closer to 50 PRs to result from it. A green CI would be terribly useful. So I'm going to go ahead and remove the affected hosts from CI before Code + Learn if no one objects.

/cc @nodejs/build

Ref: nodejs#14206

jBarz · 2017-07-15T21:10:20Z

It seems like it hangs on trying to acquire the mutex to dispatch a job to the CompilerDispatcher class.

#define CODE_EVENT_DISPATCH(code)              \
  base::LockGuard<base::Mutex> guard(&mutex_); \
  for (auto it = listeners_.begin(); it != listeners_.end(); ++it) (*it)->code

Also, I cannot reproduce the error when I run the test in isolation:

out/Release/cctest --gtest_filter=EnvironmentTest.MultipleEnvironmentsPerIsolate

But I can reproduce it when I also run the test prior to it:

out/Release/cctest --gtest_filter=EnvironmentTest.AtExitWithArgument:EnvironmentTest.MultipleEnvironmentsPerIsolate

Trott · 2017-07-16T01:37:53Z

But I can reproduce it when I also run the test prior to it:

@jBarz The side effects leaking between tests is probably what @addaleax describes in #14206 (comment), yes?

This is a temporary measure until the issue is fixed. PR-URL: #14246 Refs: #14206 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

jBarz · 2017-07-16T15:42:08Z

I agree that there is a side effect from the preceding test case. However, not sure if the source of the leak is libuv or v8 itself.

This is a temporary measure until the issue is fixed. PR-URL: #14246 Refs: #14206 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

refack · 2017-07-19T20:16:21Z

@jBarz any news?

refack · 2017-07-22T20:25:40Z

This seems to creep up and actually segfaults AtExitWithArgument:

[----------] 2 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (483 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
Makefile:354: recipe for target 'test-ci' failed
make[1]: *** [test-ci] Segmentation fault (core dumped)
Makefile:529: recipe for target 'run-ci' failed
make: *** [run-ci] Error 2

https://ci.nodejs.org/job/node-test-commit-linux/11359/nodes=ubuntu1604-32/

refack · 2017-07-30T23:23:45Z

@jBarz @mhdawson any news? (I promised not to let this fall between the cracks)

jBarz · 2017-07-31T17:22:24Z

Hi @refack , I was on vacation for a week. I am going to look at this either today/tomorrow and give an update asap.

mhdawson · 2017-08-01T21:58:50Z

@refack was just looking at the issue to see where we were, thanks for keeping an eye on it.

refack · 2017-08-07T20:46:37Z

@jBarz @mhdawson
a ping a week
and performance is peak
(sorry not a poet)

jBarz · 2017-08-08T14:53:49Z

hey @refack Last week was extremely busy with other things.
I am going to try to get this resolved this week.

The `Environment::destroy_ids_timer_handle` should be cleaned up by `Environment::CleanupHandles()`. Fix that by adding it to the list. This partially fixes a cctest. Ref: nodejs#14206

The `IsolateData` instance is created before the `Environment` instance, so free in reverse order. Fixes: nodejs#14206

This reverts commit 75bf8a9. Ref: nodejs#14206 Ref: nodejs#14317

This reverts commit 95ab966. Ref: nodejs#14206 Ref: nodejs#14246

addaleax · 2017-08-10T18:57:13Z

Fix is in #14749

The `Environment::destroy_ids_timer_handle` should be cleaned up by `Environment::CleanupHandles()`. Fix that by adding it to the list. This partially fixes a cctest. Ref: #14206 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

This reverts commit 75bf8a9. Ref: #14206 Ref: #14317 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

This reverts commit 95ab966. Ref: #14206 Ref: #14246 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

The `Environment::destroy_ids_timer_handle` should be cleaned up by `Environment::CleanupHandles()`. Fix that by adding it to the list. This partially fixes a cctest. Ref: #14206 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

The `IsolateData` instance is created before the `Environment` instance, so free in reverse order. Fixes: #14206 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

This reverts commit 75bf8a9. Ref: #14206 Ref: #14317 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

This reverts commit 95ab966. Ref: #14206 Ref: #14246 PR-URL: #14749 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>

refack mentioned this issue Jul 12, 2017

ppcbe-ubuntu1404 jobs are not finishing nodejs/build#789

Closed

mhdawson mentioned this issue Jul 13, 2017

Temprorary access for @jBarz nodejs/build#795

Closed

This was referenced Jul 14, 2017

util: remove redundant declaration #14199

Closed

cluster: overriding inspector port #14140

Merged

refack added a commit to refack/node that referenced this issue Jul 15, 2017

test: temporarily disable MultipleEnvironmentsPerIsolate

df461d2

Ref: nodejs#14206

refack mentioned this issue Jul 15, 2017

test: temporarily disable MultipleEnvironmentsPerIsolate #14246

Closed

3 tasks

Trott pushed a commit that referenced this issue Jul 16, 2017

test: disable MultipleEnvironmentsPerIsolate

95ab966

This is a temporary measure until the issue is fixed. PR-URL: #14246 Refs: #14206 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Gibson Fahnestock <[email protected]>

refack mentioned this issue Jul 18, 2017

v8.2.0 proposal #13744

Merged

refack changed the title ~~test: cctest:EnvironmentTest.MultipleEnvironmentsPerIsolate flakes (post snapshot)~~ test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) Jul 22, 2017

refack mentioned this issue Jul 22, 2017

[v8.x backport] assert: make assert.fail doc compliant #14428

Closed

4 tasks

refack mentioned this issue Aug 5, 2017

src: properly manage timer in cares ChannelWrap #14634

Closed

3 tasks

Trott mentioned this issue Aug 8, 2017

nodejs Makefile:196: recipe for target 'test' failed make: *** [test] Error 1 #14654

Closed

addaleax added a commit to addaleax/node that referenced this issue Aug 10, 2017

test: properly order freeing resources in cctest

682a951

The `IsolateData` instance is created before the `Environment` instance, so free in reverse order. Fixes: nodejs#14206

addaleax added a commit to addaleax/node that referenced this issue Aug 10, 2017

Revert "test: add DISABLED_ prefix to commented out test"

3f95be9

This reverts commit 75bf8a9. Ref: nodejs#14206 Ref: nodejs#14317

addaleax added a commit to addaleax/node that referenced this issue Aug 10, 2017

Revert "test: disable MultipleEnvironmentsPerIsolate"

557eb7b

This reverts commit 95ab966. Ref: nodejs#14206 Ref: nodejs#14246

addaleax mentioned this issue Aug 10, 2017

Fix cctest #14749

Closed

3 tasks

addaleax closed this as completed in 93f25f4 Aug 10, 2017

mhdawson mentioned this issue Dec 6, 2017

Access to PPC machine for @jBarz nodejs/build#1032

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) #14206

test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) #14206

refack commented Jul 12, 2017

addaleax commented Jul 12, 2017

Trott commented Jul 13, 2017

refack commented Jul 13, 2017

refack commented Jul 13, 2017

mhdawson commented Jul 13, 2017 •

edited

Loading

jBarz commented Jul 13, 2017

mhdawson commented Jul 13, 2017

jBarz commented Jul 14, 2017 •

edited by gibfahn

Loading

refack commented Jul 14, 2017

mhdawson commented Jul 14, 2017

Trott commented Jul 15, 2017

jBarz commented Jul 15, 2017 •

edited

Loading

Trott commented Jul 16, 2017

jBarz commented Jul 16, 2017

refack commented Jul 19, 2017

refack commented Jul 22, 2017 •

edited

Loading

refack commented Jul 30, 2017

jBarz commented Jul 31, 2017

mhdawson commented Aug 1, 2017

refack commented Aug 7, 2017

jBarz commented Aug 8, 2017

addaleax commented Aug 10, 2017

test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) #14206

test: cctest MultipleEnvironmentsPerIsolate & AtExitWithArgument are flaky (no-snapshot) #14206

Comments

refack commented Jul 12, 2017

addaleax commented Jul 12, 2017

Trott commented Jul 13, 2017

refack commented Jul 13, 2017

refack commented Jul 13, 2017

mhdawson commented Jul 13, 2017 • edited Loading

jBarz commented Jul 13, 2017

mhdawson commented Jul 13, 2017

jBarz commented Jul 14, 2017 • edited by gibfahn Loading

refack commented Jul 14, 2017

mhdawson commented Jul 14, 2017

Trott commented Jul 15, 2017

jBarz commented Jul 15, 2017 • edited Loading

Trott commented Jul 16, 2017

jBarz commented Jul 16, 2017

refack commented Jul 19, 2017

refack commented Jul 22, 2017 • edited Loading

refack commented Jul 30, 2017

jBarz commented Jul 31, 2017

mhdawson commented Aug 1, 2017

refack commented Aug 7, 2017

jBarz commented Aug 8, 2017

addaleax commented Aug 10, 2017

mhdawson commented Jul 13, 2017 •

edited

Loading

jBarz commented Jul 14, 2017 •

edited by gibfahn

Loading

jBarz commented Jul 15, 2017 •

edited

Loading

refack commented Jul 22, 2017 •

edited

Loading