benchmark, test: test the HTTP benchmark with a dummy benchmarker #12121

joyeecheung · 2017-03-29T16:28:52Z

This is a retake of #12025, it basically just goes with the last option in #12025 (comment), makes a dummy http benchmarker to connect to the benchmarked http server once, and sets all the numeric benchmark configurations to the minimal.

The first commit also refactors _http-benchmarks a bit to use ES6 classes.
The second commit also moves http/_http_simple.js to fixtures/simple-http-server.js since it is not a benchmark itself

This adds about 10s to the test running time on my mac (i7-4770HQ CPU @ 2.20GHz) (as comparison, test-benchmark-net.js runs for about 5s)

Refs: #12068

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

test, benchmark

joyeecheung · 2017-03-29T16:33:04Z

/cc @Trott @nodejs/benchmarking @nodejs/build

CI: https://ci.nodejs.org/job/node-test-pull-request/7089/

joyeecheung · 2017-03-29T17:05:42Z

It's timing out on pi1-raspbian-wheezy..should we skip this test on these slower devices, or set a larger timeout for them? (not sure how to do this)

joyeecheung · 2017-03-29T17:12:49Z

There is also another way to fix the timeout: test the benchmarks one by one in multiple tests using --filter of benchmark/run.js instead of checking them all in one test.

Trott · 2017-03-29T19:36:37Z

Love it! I'd be OK with "fixing" the Pi timing out problem by not running this test on Pi. Since it's just the Pi 1 that is failing you can do this:

if (!common.enoughTestMem) {
  common.skip('Insufficient memory for HTTP benchmark test');
  return;
}

jasnell · 2017-03-29T20:05:37Z

benchmark/_dummy-benchmarker.js

@@ -0,0 +1,7 @@
+'use strict';


not intending to be pedantic... but can we use _noop-benchmarker (noop for non-operational) instead of _dummy-....

I don't like noop because noop means "no operator". A noop should do nothing whatsoever. But that's not the case here.

In this case, this is a test double, so _test-double-benchmarker maybe? Sometimes people call certain types of test doubles mocks and other types stubs, so mock-benchmarker or stub-benchmarker might also be appropriate. (I stick to test double so that I don't have to worry about whether or not something is a mock or a stub.)

Also, I'm fine with whatever name because this is an internal thing and we can always change it later. So if @jasnell or someone else disagrees with my suggestions,
¯\(ツ)/¯.

Yeah it's not really a noop since it does connects to the server (a real noop-benchmarker would just skip the http.get call and write to stdout straight away). I think _test-double-benchmarker is better :D

jasnell · 2017-03-29T20:06:51Z

benchmark/http/chunked.js

-  num: [1, 4, 8, 16],
-  size: [1, 64, 256],
+  n: [1, 4, 8, 16],
+  len: [1, 64, 256],


why change these?

So test-benchmark-http.js can set the parameters for it (otherwise it will need to add --set num=1 --set size=1...also I think it's better to keep the configuration names consistent)

Also if we want to do #12068, doing this to other benchmarks too will make it eaiser..

joyeecheung · 2017-04-01T10:42:08Z

Renamed dummy-benchmarker to test-trouble-benchmarker and skip test on Pi 1..PTAL, thanks @jasnell @Trott

New CI: https://ci.nodejs.org/job/node-test-pull-request/7150/

joyeecheung · 2017-04-01T16:30:41Z

node-test-commit-osx took 2h30m to run..although it doesn' seem to be related to this PR since https://ci.nodejs.org/job/node-test-commit-osx/8734/ and https://ci.nodejs.org/job/node-test-commit/8819/ took more than 2 hours too?

joyeecheung · 2017-04-01T16:32:13Z

Launch a osx CI against this one: https://ci.nodejs.org/job/node-test-commit-osx/8737/
and against master: https://ci.nodejs.org/job/node-test-commit-osx/8736/

Trott · 2017-04-01T16:46:03Z

I think you meant to rename it test-double/TestDouble rather than test-trouble/testTrouble...

joyeecheung · 2017-04-01T17:37:06Z

It's double now, thanks for catching that @vsemozhetbyt @Trott :D
new CI: https://ci.nodejs.org/job/node-test-pull-request/7152/

Trott

LGTM if CI is green

Trott · 2017-04-05T23:59:35Z

@jasnell Is @joyeecheung's answer to your question "why change the config property names" sufficient? If so can you update your review so this can be unblocked?

@joyeecheung Can you rebase?

(I'd really like to see this land. :-D )

jasnell · 2017-04-06T00:04:22Z

Yes. It's fine. I can't clear my review from the mobile view. This lgtm

No longer accurate

Refactor benchmark/_http-benchmarkers.js and add a dummy HTTP benchmarker for testing.

joyeecheung · 2017-04-08T06:03:31Z

Rebased. New CI: https://ci.nodejs.org/job/node-test-pull-request/7277/

Trott · 2017-04-08T16:18:07Z

CI failures are unrelated. LGTM.

joyeecheung · 2017-04-09T10:21:19Z

Landed in c095394...2d3d4cc, thanks!

Refactor benchmark/_http-benchmarkers.js and add a test double HTTP benchmarker for testing. PR-URL: #12121 Refs: #12068 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #12121 Refs: #12068 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: James M Snell <[email protected]>

italoacasas · 2017-04-10T22:16:09Z

cc @joyeecheung

joyeecheung · 2017-04-11T02:13:20Z

Some of the changes depend on #10558, but should still be backportable. Doing a backport now...

Refactor benchmark/_http-benchmarkers.js and add a test double HTTP benchmarker for testing. PR-URL: nodejs#12121 Refs: nodejs#12068

PR-URL: nodejs#12121 Refs: nodejs#12068

gibfahn · 2017-06-08T11:21:55Z

@joyeecheung do you think it makes sense to backport this to Node 6?

If it is backported it should land with #13109 and #13390

joyeecheung · 2017-06-10T02:43:25Z

@gibfahn The new benchmark suite does not exist in 6.x so it can not be backported.

nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests. labels Mar 29, 2017

vsemozhetbyt added the http Issues or PRs related to the http subsystem. label Mar 29, 2017

jasnell previously requested changes Mar 29, 2017

View reviewed changes

joyeecheung force-pushed the dummy-http branch from 7a41687 to 2e0d5b8 Compare April 1, 2017 10:40

Trott approved these changes Apr 1, 2017

View reviewed changes

joyeecheung added 4 commits April 8, 2017 13:56

benchmark: add dummy HTTP benchmarker

ce42c03

Refactor benchmark/_http-benchmarkers.js and add a dummy HTTP benchmarker for testing.

benchmark: control HTTP benchmarks run time

4303a5e

test: add http benchmark test

14b27c6

s/trouble/double

21c48b1

joyeecheung force-pushed the dummy-http branch from 98832cf to 21c48b1 Compare April 8, 2017 06:03

joyeecheung closed this Apr 9, 2017

joyeecheung added a commit that referenced this pull request Apr 9, 2017

benchmark: control HTTP benchmarks run time

3e3414f

PR-URL: #12121 Refs: #12068 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: James M Snell <[email protected]>

joyeecheung added a commit that referenced this pull request Apr 9, 2017

test: add http benchmark test

2d3d4cc

PR-URL: #12121 Refs: #12068 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: James M Snell <[email protected]>

Trott mentioned this pull request Apr 9, 2017

test,benchmark: enable minimal CI for http bench #12025

Closed

3 tasks

italoacasas added the backport-to-v7.x label Apr 10, 2017

joyeecheung added a commit to joyeecheung/node that referenced this pull request Apr 11, 2017

benchmark: add test double HTTP benchmarker

9432cae

Refactor benchmark/_http-benchmarkers.js and add a test double HTTP benchmarker for testing. PR-URL: nodejs#12121 Refs: nodejs#12068

joyeecheung added a commit to joyeecheung/node that referenced this pull request Apr 11, 2017

benchmark: control HTTP benchmarks run time

b59c461

PR-URL: nodejs#12121 Refs: nodejs#12068

joyeecheung added a commit to joyeecheung/node that referenced this pull request Apr 11, 2017

test: add http benchmark test

d60fa5b

PR-URL: nodejs#12121 Refs: nodejs#12068

joyeecheung mentioned this pull request Apr 11, 2017

(v7.x-backport): test the HTTP benchmark with a dummy benchmarker #12323

Closed

3 tasks

addaleax added backported-to-v7.x and removed backport-requested-v7.x labels Apr 13, 2017

refack mentioned this pull request May 4, 2017

test the benchmarks at one iteration #12068

Closed

jasnell mentioned this pull request May 11, 2017

8.0.0 Release Proposal #12220

Closed

joyeecheung added the dont-land-on-v6.x label Jun 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark, test: test the HTTP benchmark with a dummy benchmarker #12121

benchmark, test: test the HTTP benchmark with a dummy benchmarker #12121

joyeecheung commented Mar 29, 2017 •

edited

Loading

joyeecheung commented Mar 29, 2017

joyeecheung commented Mar 29, 2017

joyeecheung commented Mar 29, 2017 •

edited

Loading

Trott commented Mar 29, 2017 •

edited

Loading

jasnell Mar 29, 2017

Trott Mar 29, 2017 •

edited

Loading

Trott Mar 29, 2017 •

edited

Loading

joyeecheung Mar 30, 2017

jasnell Mar 29, 2017

joyeecheung Mar 30, 2017

joyeecheung Mar 30, 2017

joyeecheung commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

Trott commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

Trott left a comment

Trott commented Apr 5, 2017

jasnell commented Apr 6, 2017

joyeecheung commented Apr 8, 2017

Trott commented Apr 8, 2017

joyeecheung commented Apr 9, 2017 •

edited

Loading

italoacasas commented Apr 10, 2017

joyeecheung commented Apr 11, 2017

gibfahn commented Jun 8, 2017

joyeecheung commented Jun 10, 2017

benchmark, test: test the HTTP benchmark with a dummy benchmarker #12121

benchmark, test: test the HTTP benchmark with a dummy benchmarker #12121

Conversation

joyeecheung commented Mar 29, 2017 • edited Loading

Checklist

Affected core subsystem(s)

joyeecheung commented Mar 29, 2017

joyeecheung commented Mar 29, 2017

joyeecheung commented Mar 29, 2017 • edited Loading

Trott commented Mar 29, 2017 • edited Loading

jasnell Mar 29, 2017

Choose a reason for hiding this comment

Trott Mar 29, 2017 • edited Loading

Choose a reason for hiding this comment

Trott Mar 29, 2017 • edited Loading

Choose a reason for hiding this comment

joyeecheung Mar 30, 2017

Choose a reason for hiding this comment

jasnell Mar 29, 2017

Choose a reason for hiding this comment

joyeecheung Mar 30, 2017

Choose a reason for hiding this comment

joyeecheung Mar 30, 2017

Choose a reason for hiding this comment

joyeecheung commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

Trott commented Apr 1, 2017

joyeecheung commented Apr 1, 2017

Trott left a comment

Choose a reason for hiding this comment

Trott commented Apr 5, 2017

jasnell commented Apr 6, 2017

joyeecheung commented Apr 8, 2017

Trott commented Apr 8, 2017

joyeecheung commented Apr 9, 2017 • edited Loading

italoacasas commented Apr 10, 2017

joyeecheung commented Apr 11, 2017

gibfahn commented Jun 8, 2017

joyeecheung commented Jun 10, 2017

joyeecheung commented Mar 29, 2017 •

edited

Loading

joyeecheung commented Mar 29, 2017 •

edited

Loading

Trott commented Mar 29, 2017 •

edited

Loading

Trott Mar 29, 2017 •

edited

Loading

Trott Mar 29, 2017 •

edited

Loading

joyeecheung commented Apr 9, 2017 •

edited

Loading