Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revert: benchmark: test refactoring #31722

Closed
wants to merge 4 commits into from

Conversation

addaleax
Copy link
Member

@addaleax addaleax commented Feb 10, 2020

This fixes running benchmark tests. Please 👍 this comment to approve fast-tracking.

Example failure:

=== release test-benchmark-async-hooks ===                   
Path: benchmark/test-benchmark-async-hooks
Error: wrk failed with 1. Output: Usage: wrk <options> <url>                            
  Options:                                            
    -c, --connections <N>  Connections to keep open   
    -d, --duration    <T>  Duration of test           
    -t, --threads     <N>  Number of threads to use   
                                                      
    -s, --script      <S>  Load Lua script file       
    -H, --header      <H>  Add header to request      
        --latency          Print latency statistics   
        --timeout     <T>  Socket/request timeout     
    -v, --version          Print version details      
                                                      
  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h)

    at ChildProcess.<anonymous> (/home/sqrt/src/node/master/benchmark/_http-benchmarkers.js:235:16)
    at Object.onceWrapper (events.js:428:26)
    at ChildProcess.emit (events.js:321:20)
    at maybeClose (internal/child_process.js:1051:16)
    at Socket.<anonymous> (internal/child_process.js:442:11)
    at Socket.emit (events.js:321:20)
    at Pipe.<anonymous> (net.js:673:12)
assert.js:102
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

1 !== 0

    at ChildProcess.<anonymous> (/home/sqrt/src/node/master/test/common/benchmark.js:30:12)
    at ChildProcess.emit (events.js:321:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 1,
  expected: 0,
  operator: 'strictEqual'
}
Command: out/Release/node /home/sqrt/src/node/master/test/benchmark/test-benchmark-async-hooks.js
[00:00|% 100|+   0|-   1]: Done                              

(This is back to working after the revert.)

Refs: #31396

Fyi @BridgeAR

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines

@addaleax addaleax added benchmark Issues and PRs related to the benchmark subsystem. fast-track PRs that do not need to wait for 48 hours to land. labels Feb 10, 2020
@nodejs-github-bot nodejs-github-bot added assert Issues and PRs related to the assert subsystem. async_hooks Issues and PRs related to the async hooks subsystem. buffer Issues and PRs related to the buffer subsystem. crypto Issues and PRs related to the crypto subsystem. http Issues or PRs related to the http subsystem. v8 engine Issues and PRs related to the V8 dependency. labels Feb 10, 2020
@addaleax addaleax removed v8 engine Issues and PRs related to the V8 dependency. assert Issues and PRs related to the assert subsystem. async_hooks Issues and PRs related to the async hooks subsystem. buffer Issues and PRs related to the buffer subsystem. crypto Issues and PRs related to the crypto subsystem. http Issues or PRs related to the http subsystem. labels Feb 10, 2020
@nodejs-github-bot
Copy link
Collaborator

Copy link
Contributor

@cjihrig cjihrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Out of curiosity, are there any small fixes that could done in place of a full revert?

@addaleax
Copy link
Member Author

@cjihrig I tried to debug it, but gave up after half an hour. Somewhere, the duration parameter gets set to a non-integer value, and wrk doesn’t accept that, but that’s all that I could find out.

addaleax added a commit that referenced this pull request Feb 10, 2020
This reverts commit 357230f.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
addaleax added a commit that referenced this pull request Feb 10, 2020
This reverts commit 78aa348.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
addaleax added a commit that referenced this pull request Feb 10, 2020
This reverts commit dac5795.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
addaleax added a commit that referenced this pull request Feb 10, 2020
This reverts commit b70741e.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@addaleax
Copy link
Member Author

Landed in 2abf0af...5b0308c, thanks for the fast reviews!

@addaleax addaleax closed this Feb 10, 2020
@addaleax addaleax deleted the revert-31396 branch February 10, 2020 21:36
@BridgeAR
Copy link
Member

I just had a look at this. It's due to setting the duration environment variable which is used in TestDoubleBenchmarker. The env was spread into the envs together with duration set when using wrk. That way the string value from the env replaced the original duration set by the options and the benchmark failed.

A fix for this is pretty straight forward by only setting the env value in TestDoubleBenchmarker.

@BridgeAR
Copy link
Member

There might be more ongoing though. I'll have a look at it later on again.

codebytere pushed a commit that referenced this pull request Feb 17, 2020
This reverts commit 357230f.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
codebytere pushed a commit that referenced this pull request Feb 17, 2020
This reverts commit 78aa348.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
codebytere pushed a commit that referenced this pull request Feb 17, 2020
This reverts commit dac5795.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
codebytere pushed a commit that referenced this pull request Feb 17, 2020
This reverts commit b70741e.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@codebytere codebytere mentioned this pull request Feb 17, 2020
@targos targos added backport-blocked-v12.x and removed dont-land-on-v12.x fast-track PRs that do not need to wait for 48 hours to land. labels Apr 20, 2020
targos pushed a commit to targos/node that referenced this pull request Apr 25, 2020
This reverts commit 357230f.

Refs: nodejs#31396
PR-URL: nodejs#31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
targos pushed a commit to targos/node that referenced this pull request Apr 25, 2020
This reverts commit b70741e.

Refs: nodejs#31396
PR-URL: nodejs#31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
targos pushed a commit that referenced this pull request Apr 28, 2020
This reverts commit 357230f.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
targos pushed a commit that referenced this pull request Apr 28, 2020
This reverts commit b70741e.

Refs: #31396
PR-URL: #31722
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Gus Caplan <[email protected]>
Reviewed-By: Vladimir de Turckheim <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Issues and PRs related to the benchmark subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants