Investigate flaky parallel/test-performance #19197

apapirovski · 2018-03-07T10:10:58Z

The recently changed parallel/test-performance appears to unfortunately be flaky now. Recently failed on aix61-ppc64. More info:

assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)

/cc @TimothyGu

The text was updated successfully, but these errors were encountered:

apapirovski · 2018-03-08T09:43:32Z

Failed again recently: https://ci.nodejs.org/job/node-test-commit-linux-containered/2768/nodes=ubuntu1604_sharedlibs_debug_x64/tapResults/

Trott · 2018-03-08T13:50:59Z

The timing stuff sure suggests to me that the test ought to be in sequential rather than parallel.

Trott · 2018-03-08T14:17:27Z

Oh, yes, definitely can fail due to system load.

$ tools/test.py --repeat 192 -j 96 test/parallel/test-performance.js 
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
[00:11|% 100|+ 189|-   3]: Done                     
$

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: nodejs#19197

Trott · 2018-03-08T14:20:28Z

PR to address this: #19228

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: #19197 PR-URL: #19228 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Minwoo Jung <[email protected]> Reviewed-By: Jon Moss <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: nodejs#19197 PR-URL: nodejs#19228 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Minwoo Jung <[email protected]> Reviewed-By: Jon Moss <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>

joyeecheung · 2018-08-17T20:27:49Z

This happened again in sequential: nodejs/reliability#14

Reason	`sequential/test-performance`
Type	JS_TEST_FAILURE
Failed PR	2 (#22318, #22288)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-7, test-requireio_andineck-debian9-armv6l_pi1p-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16459/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16462/

Example

not ok 2289 sequential/test-performance
  ---
  duration_ms: 4.558
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:338
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(Math.abs(delta) < 1000)
    
        at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/test/sequential/test-performance.js:64:7)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/test/sequential/test-performance.js:73:1)
        at Module._compile (internal/modules/cjs/loader.js:689:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
        at Module.load (internal/modules/cjs/loader.js:599:32)
        at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
        at Function.Module._load (internal/modules/cjs/loader.js:530:3)
        at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
        at startup (internal/bootstrap/node.js:266:19)
        at bootstrapNodeJSCore (internal/bootstrap/node.js:611:3)
  ...

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: nodejs#19197 PR-URL: nodejs#19228 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Minwoo Jung <[email protected]> Reviewed-By: Jon Moss <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>

1) This adds a better error logging so we are able to address further failures easier. 2) It adds a extra epsilon so the test runs into less issues in case the machine is under heavy load. 3) The epsilon in increased if the CPU is under heavy load. 4) The total startup epsilon was reduced due to recent startup time improvements. PR-URL: #22404 Fixes: #19197 Refs: nodejs/reliability#14 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: #19197 Backport-PR-URL: #22380 PR-URL: #19228 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Minwoo Jung <[email protected]> Reviewed-By: Jon Moss <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>

apapirovski added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Mar 7, 2018

Trott added a commit to Trott/io.js that referenced this issue Mar 8, 2018

test: address unreliable test-performance

de2e35a

test-performance can fail due to resource constraints. Move it from parallel to sequential so it does not compete with other tests for resources. Fixes: nodejs#19197

Trott mentioned this issue Mar 8, 2018

test: address unreliable test-performance #19228

Closed

2 tasks

jasnell closed this as completed in ed8fa81 Mar 9, 2018

joyeecheung mentioned this issue Aug 17, 2018

CI Failures: 20180817 nodejs/reliability#14

Closed

joyeecheung reopened this Aug 17, 2018

BridgeAR mentioned this issue Aug 19, 2018

test: harden sequential/test-performance #22404

Closed

4 tasks

BridgeAR closed this as completed in c35ce56 Aug 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate flaky parallel/test-performance #19197

Investigate flaky parallel/test-performance #19197

apapirovski commented Mar 7, 2018

apapirovski commented Mar 8, 2018

Trott commented Mar 8, 2018

Trott commented Mar 8, 2018

Trott commented Mar 8, 2018

joyeecheung commented Aug 17, 2018

Investigate flaky parallel/test-performance #19197

Investigate flaky parallel/test-performance #19197

Comments

apapirovski commented Mar 7, 2018

apapirovski commented Mar 8, 2018

Trott commented Mar 8, 2018

Trott commented Mar 8, 2018

Trott commented Mar 8, 2018

joyeecheung commented Aug 17, 2018