test_runner: add coverage support to run function #53937

atlowChemi · 2024-07-19T08:00:17Z

nodejs-github-bot · 2024-07-19T08:00:21Z

Review requested:

@nodejs/test_runner

cjihrig

Left a couple comments regarding a possible longer term strategy, but this is looking good. Thank you for picking this up!

lib/internal/test_runner/harness.js

lib/internal/test_runner/runner.js

doc/api/test.md

cjihrig

I think these changes are moving in the right direction, but I think this should be split up into multiple PRs.

lib/internal/main/test_runner.js

lib/internal/test_runner/harness.js

lib/internal/test_runner/test.js

test/parallel/test-runner-run.mjs

lib/internal/test_runner/test.js

lib/internal/test_runner/coverage.js

cjihrig · 2024-08-22T19:05:03Z

@atlowChemi just a heads up if you come back to this PR - there will likely be new flags to account for from #54429. A lot of internal refactoring also happened, so the diff here should be a lot simpler.

atlowChemi · 2024-08-25T13:44:54Z

@atlowChemi just a heads up if you come back to this PR - there will likely be new flags to account for from #54429. A lot of internal refactoring also happened, so the diff here should be a lot simpler.

@cjihrig Thanks for the heads up!

I am trying yo get back to this, been pretty busy in work & personal life lately, hoping to get back to this soon 🙏🏽

cjihrig

Other than docs, this looks pretty much done!

test/parallel/test-runner-run-coverage.mjs

cjihrig · 2024-08-30T15:23:25Z

I have some issues with the tests

Do @RedYetiDev's comments make a difference? If they still fail with those changes, I can take a look.

atlowChemi · 2024-09-05T04:54:58Z

I have some issues with the tests

Do @RedYetiDev's comments make a difference? If they still fail with those changes, I can take a look.

They do! Thanks @RedYetiDev 🙂
I actually noticed it just after I wrote here seeking for help, but took some time to get to it..

jasnell · 2024-09-08T20:58:21Z

lib/internal/test_runner/runner.js

+    lineCoverage,
+    branchCoverage,
+    functionCoverage,


Suggested change

lineCoverage,

branchCoverage,

functionCoverage,

lineCoverage = 0,

branchCoverage = 0,

functionCoverage = 0,

lib/internal/test_runner/runner.js

atlowChemi · 2024-09-14T19:26:33Z

@atlowChemi I don't think this should land until #54851, given that the coverage thresholds are currently modifying the exit code

@RedYetiDev I don't think that this PR should depend on #54851 as it is semver-major, and we can make such changes later to avoid backporting issues.
@cjihrig WDYT?

cjihrig · 2024-09-14T19:28:32Z

We don't need to wait on #54851. It's OK to change the process exit code for now. We'll have to change that behavior in a semver major anyway.

RedYetiDev · 2024-09-14T20:38:51Z

🎉 Looks good imo!

nodejs-github-bot · 2024-09-14T20:39:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62439/

nodejs-github-bot · 2024-09-15T04:51:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62447/

nodejs-github-bot · 2024-09-15T11:23:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62452/

mcollina

lgtm

nodejs-github-bot · 2024-09-16T06:55:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62469/

nodejs-github-bot · 2024-09-16T17:52:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62482/

nodejs-github-bot · 2024-09-17T18:11:38Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62507/

RedYetiDev · 2024-09-18T17:10:04Z

@atlowChemi

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/30322/RUN_SUBSET=3,nodes=win2019-COMPILED_BY-vs2022/testReport/junit/(root)/parallel/test_runner_run_coverage/ seems related:

00:02:15.600 not ok 686 parallel/test-runner-run-coverage
00:02:15.600   ---
00:02:15.600   duration_ms: 4266.02000
00:02:15.600   severity: fail
00:02:15.600   exitcode: 1
00:02:15.600   stack: |-
00:02:15.600     â–¶ require('node:test').run coverage settings
00:02:15.601       â–¶ validation
00:02:15.601         âœ” should only allow boolean in options.coverage (63.9542ms)
00:02:15.601         âœ” should only allow string|string[] in options.coverageExcludeGlobs (60.1064ms)
00:02:15.601         âœ” should only allow string|string[] in options.coverageIncludeGlobs (54.7812ms)
00:02:15.601         âœ” should only allow an int within range in options.lineCoverage (51.6314ms)
00:02:15.601         âœ” should only allow an int within range in options.branchCoverage (47.7735ms)
00:02:15.601         âœ” should only allow an int within range in options.functionCoverage (45.1432ms)
00:02:15.601       â–¶ validation (66.871ms)
00:02:15.601       â–¶ run with coverage
00:02:15.601         âœ” should run with coverage (868.6549ms)
00:02:15.601         âœ” should run with coverage and exclude by glob (1350.5099ms)
00:02:15.601         âœ– should run with coverage and include by glob (600.2586ms)
00:02:15.602           AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.602           
00:02:15.602           false !== true
00:02:15.602           
00:02:15.602               at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:131:16)
00:02:15.602               at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.602               at TestsStream.emit (node:events:519:28)
00:02:15.602               at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.602               at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.602               at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.602               at postRun (node:internal/test_runner/runner:700:28)
00:02:15.602               at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.602             generatedMessage: true,
00:02:15.602             code: 'ERR_ASSERTION',
00:02:15.603             actual: false,
00:02:15.603             expected: true,
00:02:15.603             operator: 'strictEqual'
00:02:15.603           }
00:02:15.603     
00:02:15.603         âœ– should run while including and excluding globs (412.3843ms)
00:02:15.603           AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.603           
00:02:15.604           false !== true
00:02:15.604           
00:02:15.604               at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:149:16)
00:02:15.604               at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.604               at TestsStream.emit (node:events:519:28)
00:02:15.604               at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.604               at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.605               at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.605               at postRun (node:internal/test_runner/runner:700:28)
00:02:15.605               at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.605             generatedMessage: true,
00:02:15.605             code: 'ERR_ASSERTION',
00:02:15.605             actual: false,
00:02:15.605             expected: true,
00:02:15.605             operator: 'strictEqual'
00:02:15.606           }
00:02:15.606     
00:02:15.606         âœ– should run with coverage and fail when below line threshold (1.1888ms)
00:02:15.606           AssertionError [ERR_ASSERTION]: Expected "actual" to be strictly unequal to: 1
00:02:15.606               at TestContext.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:158:14)
00:02:15.606               at Test.runInAsyncScope (node:async_hooks:211:14)
00:02:15.606               at Test.run (node:internal/test_runner/test:930:25)
00:02:15.606               at Suite.processPendingSubtests (node:internal/test_runner/test:629:18)
00:02:15.606               at Test.postRun (node:internal/test_runner/test:1026:19)
00:02:15.606               at Test.run (node:internal/test_runner/test:969:12)
00:02:15.607               at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
00:02:15.607               at async Suite.processPendingSubtests (node:internal/test_runner/test:629:7) {
00:02:15.607             generatedMessage: true,
00:02:15.607             code: 'ERR_ASSERTION',
00:02:15.607             actual: 1,
00:02:15.607             expected: 1,
00:02:15.607             operator: 'notStrictEqual'
00:02:15.608           }
00:02:15.608     
00:02:15.608       â–¶ run with coverage (3266.002ms)
00:02:15.608     â–¶ require('node:test').run coverage settings (3268.4502ms)
00:02:15.608     â„¹ tests 11
00:02:15.608     â„¹ suites 3
00:02:15.608     â„¹ pass 8
00:02:15.608     â„¹ fail 3
00:02:15.608     â„¹ cancelled 0
00:02:15.608     â„¹ skipped 0
00:02:15.609     â„¹ todo 0
00:02:15.609     â„¹ duration_ms 3288.5163
00:02:15.609     
00:02:15.609     âœ– failing tests:
00:02:15.609     
00:02:15.609     test at test\parallel\test-runner-run-coverage.mjs:121:11
00:02:15.609     âœ– should run with coverage and include by glob (600.2586ms)
00:02:15.609       AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.609       
00:02:15.609       false !== true
00:02:15.609       
00:02:15.610           at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:131:16)
00:02:15.610           at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.610           at TestsStream.emit (node:events:519:28)
00:02:15.610           at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.610           at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.610           at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.610           at postRun (node:internal/test_runner/runner:700:28)
00:02:15.610           at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.610         generatedMessage: true,
00:02:15.610         code: 'ERR_ASSERTION',
00:02:15.610         actual: false,
00:02:15.610         expected: true,
00:02:15.611         operator: 'strictEqual'
00:02:15.611       }
00:02:15.611     
00:02:15.611     test at test\parallel\test-runner-run-coverage.mjs:137:11
00:02:15.611     âœ– should run while including and excluding globs (412.3843ms)
00:02:15.611       AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.611       
00:02:15.612       false !== true
00:02:15.612       
00:02:15.612           at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:149:16)
00:02:15.612           at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.612           at TestsStream.emit (node:events:519:28)
00:02:15.612           at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.612           at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.612           at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.612           at postRun (node:internal/test_runner/runner:700:28)
00:02:15.612           at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.612         generatedMessage: true,
00:02:15.612         code: 'ERR_ASSERTION',
00:02:15.612         actual: false,
00:02:15.612         expected: true,
00:02:15.612         operator: 'strictEqual'
00:02:15.612       }
00:02:15.612     
00:02:15.613     test at test\parallel\test-runner-run-coverage.mjs:155:11
00:02:15.613     âœ– should run with coverage and fail when below line threshold (1.1888ms)
00:02:15.613       AssertionError [ERR_ASSERTION]: Expected "actual" to be strictly unequal to: 1
00:02:15.613           at TestContext.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:158:14)
00:02:15.613           at Test.runInAsyncScope (node:async_hooks:211:14)
00:02:15.613           at Test.run (node:internal/test_runner/test:930:25)
00:02:15.613           at Suite.processPendingSubtests (node:internal/test_runner/test:629:18)
00:02:15.613           at Test.postRun (node:internal/test_runner/test:1026:19)
00:02:15.613           at Test.run (node:internal/test_runner/test:969:12)
00:02:15.613           at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
00:02:15.613           at async Suite.processPendingSubtests (node:internal/test_runner/test:629:7) {
00:02:15.613         generatedMessage: true,
00:02:15.613         code: 'ERR_ASSERTION',
00:02:15.614         actual: 1,
00:02:15.614         expected: 1,
00:02:15.614         operator: 'notStrictEqual'
00:02:15.614       }
00:02:15.614     (node:4360) ExperimentalWarning: glob is an experimental feature and might change at any time
00:02:15.615     (Use `node --trace-warnings ...` to show where the warning was created)

nodejs-github-bot · 2024-09-20T03:25:02Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62583/

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jul 19, 2024

cjihrig reviewed Jul 19, 2024

View reviewed changes

lib/internal/test_runner/harness.js Outdated Show resolved Hide resolved

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

cjihrig mentioned this pull request Jul 20, 2024

test_runner: refactor coverage to pass in config options #53931

Merged

atlowChemi force-pushed the coverage-support-via-run branch 3 times, most recently from e9d7a22 to 05f4e39 Compare July 22, 2024 05:53

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

atlowChemi force-pushed the coverage-support-via-run branch 3 times, most recently from 2907841 to 7ef36f2 Compare July 29, 2024 18:35

cjihrig reviewed Jul 31, 2024

View reviewed changes

RedYetiDev mentioned this pull request Aug 12, 2024

enable --experimental-test-coverage using run method #54343

Open

RedYetiDev reviewed Aug 13, 2024

View reviewed changes

lib/internal/test_runner/coverage.js Outdated Show resolved Hide resolved

RedYetiDev added wip Issues and PRs that are still a work in progress. coverage Issues and PRs related to native coverage support. labels Aug 22, 2024

atlowChemi force-pushed the coverage-support-via-run branch from 7ef36f2 to 9a9c295 Compare August 29, 2024 07:41

cjihrig reviewed Aug 29, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

RedYetiDev reviewed Aug 29, 2024

View reviewed changes

test/parallel/test-runner-run-coverage.mjs Outdated Show resolved Hide resolved

test/parallel/test-runner-run-coverage.mjs Outdated Show resolved Hide resolved

atlowChemi force-pushed the coverage-support-via-run branch from 4ab6491 to ffe1e56 Compare September 5, 2024 05:03

atlowChemi marked this pull request as ready for review September 5, 2024 05:03

jasnell reviewed Sep 8, 2024

View reviewed changes

RedYetiDev reviewed Sep 8, 2024

View reviewed changes

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

atlowChemi force-pushed the coverage-support-via-run branch from 23a7582 to 7f4608c Compare September 14, 2024 20:26

cjihrig approved these changes Sep 14, 2024

View reviewed changes

atlowChemi requested review from cjihrig, jasnell and RedYetiDev September 14, 2024 20:28

cjihrig approved these changes Sep 14, 2024

View reviewed changes

atlowChemi added semver-minor PRs that contain new features and should be released in the next minor version. request-ci Add this label to start a Jenkins CI on a PR. labels Sep 14, 2024

RedYetiDev removed their request for review September 14, 2024 20:38

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 14, 2024

mcollina approved these changes Sep 15, 2024

View reviewed changes

MoLow approved these changes Sep 15, 2024

View reviewed changes

RedYetiDev removed the wip Issues and PRs that are still a work in progress. label Sep 17, 2024

test_runner: add support for coverage via run()

db37754

atlowChemi force-pushed the coverage-support-via-run branch from 7f4608c to db37754 Compare September 20, 2024 03:21

atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 20, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_runner: add coverage support to run function #53937

test_runner: add coverage support to run function #53937

atlowChemi commented Jul 19, 2024 •

edited

Loading

nodejs-github-bot commented Jul 19, 2024

cjihrig left a comment

cjihrig left a comment

cjihrig commented Aug 22, 2024

atlowChemi commented Aug 25, 2024

cjihrig left a comment

This comment was marked as resolved.

cjihrig commented Aug 30, 2024

atlowChemi commented Sep 5, 2024

jasnell Sep 8, 2024 •

edited

Loading

atlowChemi commented Sep 14, 2024

cjihrig commented Sep 14, 2024

RedYetiDev commented Sep 14, 2024 •

edited

Loading

nodejs-github-bot commented Sep 14, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 15, 2024

mcollina left a comment

nodejs-github-bot commented Sep 16, 2024

nodejs-github-bot commented Sep 16, 2024

nodejs-github-bot commented Sep 17, 2024

RedYetiDev commented Sep 18, 2024 •

edited

Loading

nodejs-github-bot commented Sep 20, 2024

test_runner: add coverage support to run function #53937

Are you sure you want to change the base?

test_runner: add coverage support to run function #53937

Conversation

atlowChemi commented Jul 19, 2024 • edited Loading

nodejs-github-bot commented Jul 19, 2024

cjihrig left a comment

Choose a reason for hiding this comment

cjihrig left a comment

Choose a reason for hiding this comment

cjihrig commented Aug 22, 2024

atlowChemi commented Aug 25, 2024

cjihrig left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

cjihrig commented Aug 30, 2024

atlowChemi commented Sep 5, 2024

jasnell Sep 8, 2024 • edited Loading

Choose a reason for hiding this comment

atlowChemi commented Sep 14, 2024

cjihrig commented Sep 14, 2024

RedYetiDev commented Sep 14, 2024 • edited Loading

nodejs-github-bot commented Sep 14, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 15, 2024

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Sep 16, 2024

nodejs-github-bot commented Sep 16, 2024

nodejs-github-bot commented Sep 17, 2024

RedYetiDev commented Sep 18, 2024 • edited Loading

nodejs-github-bot commented Sep 20, 2024

atlowChemi commented Jul 19, 2024 •

edited

Loading

jasnell Sep 8, 2024 •

edited

Loading

RedYetiDev commented Sep 14, 2024 •

edited

Loading

RedYetiDev commented Sep 18, 2024 •

edited

Loading