Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: add benchmark for coverage #36972

Closed
wants to merge 1 commit into from

Conversation

bcoe
Copy link
Contributor

@bcoe bcoe commented Jan 17, 2021

Working on #36956, I became a bit worried that as we extend code coverage to support more types of branching, performance might start to become unacceptable.

I created this benchmark to experiment:

Node 14.15.4, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -18.94 % ±0.77% ±1.02% ±1.30%

Node 15.6.0, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -17.95 % ±1.08% ±1.43% ±1.83%

Coverage with #36956, vs., no coverage:

process/coverage.js n=100000 *** -19.37 % ±1.36% ±1.79% ±2.30%

To Summarize

In the current release of 14/15, we see that running Node with coverage decreases performance by about 18%.

With the introduction of optional chaining in this benchmark, we see a 1 - 2% additional hit to performance.

Other interesting facts

  • I benchmarked coverage.js using Istanbul, vs., NODE_V8_COVERAGE, it was 37.46% slower than no coverage.
  • I benchmarked deps: V8: cherry-pick fe191e8d05cc #36956 with no coverage, vs., 15.6.0 with no coverage; there did not appear to be a significant difference in performance (0.37 % ±1.14% ±1.50% ±1.92%).

Conclusions

  • My gut is that it's worth taking the 1 - 2% performance hit to support a new type of branching in our coverage reporting.
  • As new constructs like this are added to JavaScript, we need to be mindful that the cost of branch coverage will continue to increase.
  • I was excited to see that v8's coverage offers a significant performance benefit over pre-instrumenting the JavaScript code.

CC: @schuay

@bcoe bcoe requested review from Trott, targos and devsnek January 17, 2021 03:43
@nodejs-github-bot nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. process Issues and PRs related to the process subsystem. labels Jan 17, 2021
@bcoe
Copy link
Contributor Author

bcoe commented Jan 17, 2021

@joyeecheung I was thinking this benchmark would be a really good use of v8.startCoverage, because it would let us turn on coverage collection programmatically, at which point it would be easier to use this benchmark to detect regressions in coverage performance in the future.

@Trott
Copy link
Member

Trott commented Jan 17, 2021

Subsystem in commit message should be benchmark rather than test.

@bcoe bcoe changed the title test: add benchmark for coverage benchmark: add benchmark for coverage Jan 18, 2021
@bcoe bcoe force-pushed the benchmark-coverage branch 2 times, most recently from c5507d4 to b5eadb0 Compare January 18, 2021 19:03
@bcoe
Copy link
Contributor Author

bcoe commented Jan 18, 2021

@Trott refactored so that the benchmark is easier to use, it now spawns a subprocess with coverage enabled (_so no need to create a separate bin) 👍

@nodejs-github-bot
Copy link
Collaborator

benchmark/process/coverage.js Outdated Show resolved Hide resolved
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot
Copy link
Collaborator

@bcoe bcoe requested a review from richardlau January 22, 2021 23:25
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot
Copy link
Collaborator

nodejs-github-bot commented Jan 25, 2021

bcoe added a commit that referenced this pull request Jan 27, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@bcoe
Copy link
Contributor Author

bcoe commented Jan 27, 2021

Landed in 8c9dc4e

@bcoe bcoe closed this Jan 27, 2021
targos pushed a commit that referenced this pull request Feb 2, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@targos targos mentioned this pull request Feb 2, 2021
targos pushed a commit that referenced this pull request May 1, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
@danielleadams danielleadams mentioned this pull request May 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Issues and PRs related to the benchmark subsystem. process Issues and PRs related to the process subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants