Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

benchmark: add benchmark for coverage #36972

Closed
wants to merge 1 commit into from
Closed

benchmark: add benchmark for coverage #36972

wants to merge 1 commit into from

Conversation

bcoe
Copy link
Contributor

@bcoe bcoe commented Jan 17, 2021 •
edited
Loading

Working on #36956, I became a bit worried that as we extend code coverage to support more types of branching, performance might start to become unacceptable.

I created this benchmark to experiment:

Node 14.15.4, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -18.94 % ±0.77% ±1.02% ±1.30%

Node 15.6.0, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -17.95 % ±1.08% ±1.43% ±1.83%

Coverage with #36956, vs., no coverage:

process/coverage.js n=100000 *** -19.37 % ±1.36% ±1.79% ±2.30%

To Summarize

In the current release of 14/15, we see that running Node with coverage decreases performance by about 18%.

With the introduction of optional chaining in this benchmark, we see a 1 - 2% additional hit to performance.

Other interesting facts

  • I benchmarked coverage.js using Istanbul, vs., NODE_V8_COVERAGE, it was 37.46% slower than no coverage.
  • I benchmarked deps: V8: cherry-pick fe191e8d05cc #36956 with no coverage, vs., 15.6.0 with no coverage; there did not appear to be a significant difference in performance (0.37 % ±1.14% ±1.50% ±1.92%).

Conclusions

  • My gut is that it's worth taking the 1 - 2% performance hit to support a new type of branching in our coverage reporting.
  • As new constructs like this are added to JavaScript, we need to be mindful that the cost of branch coverage will continue to increase.
  • I was excited to see that v8's coverage offers a significant performance benefit over pre-instrumenting the JavaScript code.

CC: @schuay

@bcoe bcoe requested review from Trott, targos and devsnek January 17, 2021 03:43
@nodejs-github-bot nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. process Issues and PRs related to the process subsystem. labels Jan 17, 2021
@bcoe
Copy link
Contributor Author

bcoe commented Jan 17, 2021

@joyeecheung I was thinking this benchmark would be a really good use of v8.startCoverage, because it would let us turn on coverage collection programmatically, at which point it would be easier to use this benchmark to detect regressions in coverage performance in the future.

@Trott
Copy link
Member

Trott commented Jan 17, 2021

Subsystem in commit message should be benchmark rather than test.

@bcoe bcoe force-pushed the benchmark-coverage branch from d206807 to 1d8a88b Compare January 18, 2021 18:57
@bcoe bcoe changed the title test: add benchmark for coverage benchmark: add benchmark for coverage Jan 18, 2021
@bcoe bcoe force-pushed the benchmark-coverage branch 2 times, most recently from c5507d4 to b5eadb0 Compare January 18, 2021 19:03
@bcoe bcoe force-pushed the benchmark-coverage branch from b5eadb0 to efea2d6 Compare January 18, 2021 19:16
@bcoe
Copy link
Contributor Author

bcoe commented Jan 18, 2021

@Trott refactored so that the benchmark is easier to use, it now spawns a subprocess with coverage enabled (_so no need to create a separate bin) 👍

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/35626/

joyeecheung
joyeecheung approved these changes Jan 19, 2021
View reviewed changes
benchmark/process/coverage.js Outdated Show resolved Hide resolved
@bcoe bcoe force-pushed the benchmark-coverage branch from efea2d6 to ed4e294 Compare January 19, 2021 17:23
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@bcoe bcoe force-pushed the benchmark-coverage branch from ed4e294 to 682f4e9 Compare January 22, 2021 17:17
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@bcoe bcoe force-pushed the benchmark-coverage branch from 682f4e9 to ae141a5 Compare January 22, 2021 19:45
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@bcoe bcoe force-pushed the benchmark-coverage branch from ae141a5 to 2823b83 Compare January 22, 2021 19:58
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/35683/

@bcoe bcoe force-pushed the benchmark-coverage branch from 2823b83 to 56cf8b5 Compare January 22, 2021 23:25
@bcoe bcoe requested a review from richardlau January 22, 2021 23:25
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@nodejs-github-bot
Copy link
Collaborator

nodejs-github-bot commented Jan 25, 2021 •
edited by bcoe
Loading

CI: https://ci.nodejs.org/job/node-test-pull-request/35719/ ✅

bcoe added a commit that referenced this pull request Jan 27, 2021
@bcoe
benchmark: add benchmark for NODE_V8_COVERAGE
8c9dc4e 
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@bcoe
Copy link
Contributor Author

bcoe commented Jan 27, 2021

Landed in 8c9dc4e

@bcoe bcoe closed this Jan 27, 2021
targos pushed a commit that referenced this pull request Feb 2, 2021
@bcoe @targos
benchmark: add benchmark for NODE_V8_COVERAGE
dcd34b0 
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@targos targos mentioned this pull request Feb 2, 2021
Merged
targos pushed a commit that referenced this pull request May 1, 2021
@bcoe @targos
benchmark: add benchmark for NODE_V8_COVERAGE
d2770a5 
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@danielleadams danielleadams mentioned this pull request May 3, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardlau richardlau richardlau approved these changes

@joyeecheung joyeecheung joyeecheung approved these changes

@benjamingr benjamingr benjamingr approved these changes

@Trott Trott Trott approved these changes

@jasnell jasnell jasnell approved these changes

@targos targos Awaiting requested review from targos

@devsnek devsnek Awaiting requested review from devsnek

Assignees
No one assigned
Labels
benchmark Issues and PRs related to the benchmark subsystem. process Issues and PRs related to the process subsystem.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bcoe @Trott @nodejs-github-bot @jasnell @benjamingr @joyeecheung @richardlau