Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

investigate flaky tests on fedora 24 #17589

Closed
Trott opened this issue Dec 10, 2017 · 7 comments
Closed

investigate flaky tests on fedora 24 #17589

Trott opened this issue Dec 10, 2017 · 7 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.

Comments

@Trott
Copy link
Member

Trott commented Dec 10, 2017

  • Version: 10.0.0-pre
  • Platform: fedora24
  • Subsystem: test

sequential/test-child-process-pass-fd and sequential/test-inspector-port-cluster sometimes fail as as a pair on fedora 24.

Recent examples:

https://ci.nodejs.org/job/node-test-commit-linux/nodes=fedora24/14848/console 
not ok 2001 sequential/test-child-process-pass-fd
  ---
  duration_ms: 2.823
  severity: fail
  stack: |-
    node[26803]: pthread_create: Resource temporarily unavailable
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
...
not ok 2046 sequential/test-inspector-port-cluster
  ---
  duration_ms: 0.661
  severity: fail
  stack: |-
    Debugger listening on ws://127.0.0.1:12346/5b474053-46e9-47a3-a22d-12fdf214dcdc
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:9229/01dcd483-2268-4194-86dd-fd6da35c083d
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12356/088df66b-35aa-48df-95cf-9ceb550b522d
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12351/1a086704-73e0-43db-87a7-fee4cf2518ef
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12366/ac22826d-1f0c-47ca-a745-d52362749d52
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://0.0.0.0:12361/86a78680-52b3-4a47-9250-529eaa308a6b
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://[::1]:12376/744a25a8-b1ad-4351-948b-38d5cd959926
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12386/56a7d647-bcd1-4b7d-87e9-0c6374a26fe1
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12381/5083d73a-b46c-4b2e-8c67-58c86f51fdb9
    For help see https://nodejs.org/en/docs/inspector
    node[27553]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27553]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: Debugger listening on ws://127.0.0.1:12391/339d35cf-af3c-4332-85cf-23c05b2cc96d
    For help see https://nodejs.org/en/docs/inspector
    node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27543]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27543]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    Debugger listening on ws://[::]:12371/a06cbab2-38e2-49d7-a1e0-752d437970b8
    For help see https://nodejs.org/en/docs/inspector
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    Debugger listening on ws://127.0.0.1:12406/4db10d1c-17a0-4134-a981-a2aa98f7e9ff
    For help see https://nodejs.org/en/docs/inspector
    node[27562]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27562]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    Debugger listening on ws://127.0.0.1:12396/2adf877b-8a1a-4558-acaa-9511aae1b852
    For help see https://nodejs.org/en/docs/inspector
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    node[27597]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27597]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27599]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27599]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27596]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27596]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    Debugger listening on ws://127.0.0.1:12390/f5b2e26d-f286-465d-ac9b-f58bc2ced564
    For help see https://nodejs.org/en/docs/inspector
    assert.js:43
      throw new errors.AssertionError({
      ^
    
    AssertionError [ERR_ASSERTION]: null strictEqual 0
        at Worker.checkExitCode (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-inspector-port-cluster.js:329:10)
        at Worker.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/common/index.js:522:15)
        at Worker.emit (events.js:126:13)
        at ChildProcess.worker.process.once (internal/cluster/master.js:184:12)
        at Object.onceWrapper (events.js:218:13)
        at ChildProcess.emit (events.js:126:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
 https://ci.nodejs.org/job/node-test-commit-linux/nodes=fedora24/14835/console 
not ok 2001 sequential/test-child-process-pass-fd
  ---
  duration_ms: 2.823
  severity: fail
  stack: |-
    node[26803]: pthread_create: Resource temporarily unavailable
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
        at process.target.send (internal/child_process.js:606:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:37:15)
        at Object.onceWrapper (events.js:218:13)
        at Socket.emit (events.js:126:13)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1194:10)
  ...
not ok 2046 sequential/test-inspector-port-cluster
  ---
  duration_ms: 0.661
  severity: fail
  stack: |-
    Debugger listening on ws://127.0.0.1:12346/5b474053-46e9-47a3-a22d-12fdf214dcdc
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:9229/01dcd483-2268-4194-86dd-fd6da35c083d
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12356/088df66b-35aa-48df-95cf-9ceb550b522d
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12351/1a086704-73e0-43db-87a7-fee4cf2518ef
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12366/ac22826d-1f0c-47ca-a745-d52362749d52
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://0.0.0.0:12361/86a78680-52b3-4a47-9250-529eaa308a6b
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://[::1]:12376/744a25a8-b1ad-4351-948b-38d5cd959926
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12386/56a7d647-bcd1-4b7d-87e9-0c6374a26fe1
    For help see https://nodejs.org/en/docs/inspector
    Debugger listening on ws://127.0.0.1:12381/5083d73a-b46c-4b2e-8c67-58c86f51fdb9
    For help see https://nodejs.org/en/docs/inspector
    node[27553]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27553]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: Debugger listening on ws://127.0.0.1:12391/339d35cf-af3c-4332-85cf-23c05b2cc96d
    For help see https://nodejs.org/en/docs/inspector
    node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27543]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27543]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    Debugger listening on ws://[::]:12371/a06cbab2-38e2-49d7-a1e0-752d437970b8
    For help see https://nodejs.org/en/docs/inspector
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    Debugger listening on ws://127.0.0.1:12406/4db10d1c-17a0-4134-a981-a2aa98f7e9ff
    For help see https://nodejs.org/en/docs/inspector
    node[27562]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27562]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    Debugger listening on ws://127.0.0.1:12396/2adf877b-8a1a-4558-acaa-9511aae1b852
    For help see https://nodejs.org/en/docs/inspector
    events.js:111
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at _errnoException (util.js:1013:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
        at onErrorNT (internal/child_process.js:389:16)
        at process._tickCallback (internal/process/next_tick.js:152:19)
        at Function.Module.runMain (module.js:703:11)
        at startup (bootstrap_node.js:195:16)
        at bootstrap_node.js:646:3
    node[27597]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27597]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27599]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27599]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    node[27596]: pthread_create: Resource temporarily unavailable
    /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node[27596]: ../src/inspector_io.cc:207:bool node::inspector::InspectorIo::Start(): Assertion `(uv_thread_create(&thread_, InspectorIo::ThreadMain, this)) == (0)' failed.
     1: node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     2: 0x1234545 [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     3: 0x12db98a [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     4: node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     5: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     6: node::Start(int, char**) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
     7: __libc_start_main [/lib64/libc.so.6]
     8: _start [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node]
    Debugger listening on ws://127.0.0.1:12390/f5b2e26d-f286-465d-ac9b-f58bc2ced564
    For help see https://nodejs.org/en/docs/inspector
    assert.js:43
      throw new errors.AssertionError({
      ^
    
    AssertionError [ERR_ASSERTION]: null strictEqual 0
        at Worker.checkExitCode (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-inspector-port-cluster.js:329:10)
        at Worker.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/common/index.js:522:15)
        at Worker.emit (events.js:126:13)
        at ChildProcess.worker.process.once (internal/cluster/master.js:184:12)
        at Object.onceWrapper (events.js:218:13)
        at ChildProcess.emit (events.js:126:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
@Trott Trott added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Dec 10, 2017
@Trott
Copy link
Member Author

Trott commented Dec 10, 2017

Commonality is that both tests launch a lot of child processes and end up getting "Resource temporarily unavailable". Obvious things to try:

  • If the tests are spawning lots of processes asynchronously, maybe the tests can be rewritten to spawn one at a time and only spawn the next one when the test has finished for the first one.

  • Or maybe the tests can be split into multiple test files to reduce the number of spawned processes.

@Trott
Copy link
Member Author

Trott commented Dec 10, 2017

@nodejs/testing @nodejs/build

@Trott
Copy link
Member Author

Trott commented Dec 11, 2017

The bullet point suggested solutions above won't work for test-child-process-pass-fd because it requires the large number of processes to test for the bug that it was written for.

Options might be:

  • Move it to pummel, which would be a shame because it would rarely be run.
  • Have it check that it's on macOS because that's the only (known) place where the bug was found. That's OK, but it probably ought be run on other platforms too.
  • Find some way to check for resource requirements that will cause the test to be skipped on our Fedora 24 setup but not on macOS and elsewhere. Downside here is that whatever we come up with might be brittle in that it might cause the test to be skipped on future macOS hosts in CI, which we really don't want.

@Trott
Copy link
Member Author

Trott commented Dec 11, 2017

Another possibility: Rewrite the test to accommodate "Resource temporarily unavailable".

@Trott
Copy link
Member Author

Trott commented Dec 11, 2017

CI stress test on master for sequential/test-child-process-pass-fd that will hopefully show it being flaky: https://ci.nodejs.org/job/node-stress-single-test/1563/nodes=fedora24/

@Trott
Copy link
Member Author

Trott commented Dec 11, 2017

Oh, neat-o, I guess...once the stress test fails, it then fails again and again and again. Stress test for master is red, so that at least confirms the problem.

Trott added a commit to Trott/io.js that referenced this issue Dec 11, 2017
@Trott
test: fix flaky test-child-process-pass-fd
529f000 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

Fixes: nodejs#17589
@Trott Trott mentioned this issue Dec 11, 2017
Closed
2 tasks
@Trott
Copy link
Member Author

Trott commented Dec 12, 2017

Proposed fix in #17598

@Trott Trott closed this as completed in d64b0a8 Dec 13, 2017
MylesBorins pushed a commit that referenced this issue Jan 8, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
6cb4cc2 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
MylesBorins pushed a commit that referenced this issue Jan 22, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
a304206 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
MylesBorins pushed a commit that referenced this issue Jan 22, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
fcc5b99 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
MylesBorins pushed a commit that referenced this issue Feb 11, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
be8c371 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
MylesBorins pushed a commit that referenced this issue Feb 12, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
22bc6a1 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
MylesBorins pushed a commit that referenced this issue Feb 13, 2018
@Trott @MylesBorins
test: fix flaky test-child-process-pass-fd
fb0bd8a 
test-child-process-pass-fd needs to launch many processes
simultaneously. On Fedora 24, this can result in EAGAIN "Resource
temporarily unavailable" errors. When this occurs, simply try to launch
a worker again.

PR-URL: #17598
Fixes: #17589
Reviewed-By: Santiago Gimeno <[email protected]>
Reviewed-By: Gibson Fahnestock <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: fix flaky test-child-process-pass-fd on Fedora 24 Trott/io.js
1 participant
@Trott