[v18.x backport] test_runner features #46839

MoLow · 2023-02-25T21:26:42Z

since these commits depend on each other and many of them don't land clearly - backporting these in bulk makes more sense

Fixes: nodejs#44612 PR-URL: nodejs#45264 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#45712 Fixes: nodejs#45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

Move the logic for handling --test-reporter out of the general module loader and into the test_runner subsystem. PR-URL: nodejs#45923 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46030 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

This commit updates the test runner to make the built in test reporters internal modules. PR-URL: nodejs#46092 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

nodejs-github-bot · 2023-02-25T21:26:46Z

Review requested:

@nodejs/test_runner

This commit adds code coverage functionality to the node:test module. When node:test is used in conjunction with the new --test-coverage CLI flag, a coverage report is created when the test runner finishes. The coverage summary is forwarded to any test runner reporters so that the display can be customized as desired. This new functionality is compatible with the existing NODE_V8_COVERAGE environment variable as well. There are still several limitations, which will be addressed in subsequent pull requests: - Coverage is only reported for a single process. It is possible to merge coverage reports together. Once this is done, the --test flag will be supported as well. - Source maps are not currently supported. - Excluding specific files or directories from the coverage report is not currently supported. Node core modules and node_modules/ are excluded though. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

Add experimental to the name as requested during review. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

PR-URL: nodejs#46311 Fixes: nodejs#45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#46441 Fixes: nodejs#45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46450 Fixes: nodejs#45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46056 Fixes: nodejs#46048 Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: nodejs#46544 Fixes: nodejs#46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46440 Fixes: nodejs#45833 Refs: nodejs#45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#45736 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: nodejs#46651 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: nodejs#46707 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46540 Fixes: nodejs#46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46737 Fixes: nodejs#46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: nodejs#46604 Fixes: nodejs#46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After nodejs#46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: nodejs#46720 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: #46707 Backport-PR-URL: #46839 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46540 Backport-PR-URL: #46839 Fixes: #46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46737 Backport-PR-URL: #46839 Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: #46604 Backport-PR-URL: #46839 Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After #46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: #46720 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol · 2023-03-03T06:50:29Z

Landed in 9434396...e17d857 🎉

Fixes: #44612 PR-URL: #45264 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45712 Backport-PR-URL: #46839 Fixes: #45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

Move the logic for handling --test-reporter out of the general module loader and into the test_runner subsystem. PR-URL: #45923 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46030 Backport-PR-URL: #46839 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

This commit updates the test runner to make the built in test reporters internal modules. PR-URL: #46092 Backport-PR-URL: #46839 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

This commit adds code coverage functionality to the node:test module. When node:test is used in conjunction with the new --test-coverage CLI flag, a coverage report is created when the test runner finishes. The coverage summary is forwarded to any test runner reporters so that the display can be customized as desired. This new functionality is compatible with the existing NODE_V8_COVERAGE environment variable as well. There are still several limitations, which will be addressed in subsequent pull requests: - Coverage is only reported for a single process. It is possible to merge coverage reports together. Once this is done, the --test flag will be supported as well. - Source maps are not currently supported. - Excluding specific files or directories from the coverage report is not currently supported. Node core modules and node_modules/ are excluded though. PR-URL: #46017 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

Add experimental to the name as requested during review. PR-URL: #46017 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

PR-URL: #46311 Backport-PR-URL: #46839 Fixes: #45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #46441 Backport-PR-URL: #46839 Fixes: #45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46450 Backport-PR-URL: #46839 Fixes: #45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46056 Backport-PR-URL: #46839 Fixes: #46048 Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46544 Backport-PR-URL: #46839 Fixes: #46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46440 Backport-PR-URL: #46839 Fixes: #45833 Refs: #45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: #46707 Backport-PR-URL: #46839 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46540 Backport-PR-URL: #46839 Fixes: #46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46737 Backport-PR-URL: #46839 Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: #46604 Backport-PR-URL: #46839 Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After #46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: #46720 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

fossamagna and others added 5 commits February 25, 2023 23:21

test_runner: avoid swallowing of asynchronously thrown errors

9ec0a4e

Fixes: nodejs#44612 PR-URL: nodejs#45264 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: add reporters

7201845

PR-URL: nodejs#45712 Fixes: nodejs#45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: report file in test runner events

b628161

PR-URL: nodejs#46030 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v18.x Issues that can be reproduced on v18.x or PRs targeting the v18.x-staging branch. labels Feb 25, 2023

MoLow mentioned this pull request Feb 25, 2023

test_runner: add initial code coverage support #46017

Merged

MoLow changed the title ~~[v18.x backport] test_runner reporters~~ [v18.x backport] test_runner features Feb 25, 2023

MoLow force-pushed the backport-v18-test-runner-reporters branch from 985157f to dd5e659 Compare February 25, 2023 21:47

cjihrig and others added 14 commits February 25, 2023 23:55

doc,lib,src,test: rename --test-coverage

8ad7668

Add experimental to the name as requested during review. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

test: fix tap escaping with and without --test

95250e1

PR-URL: nodejs#46311 Fixes: nodejs#45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: top-level diagnostics not ommited when running with --test

e1f5a3a

PR-URL: nodejs#46441 Fixes: nodejs#45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

test_runner: fix missing test diagnostics

f4264f0

PR-URL: nodejs#46450 Fixes: nodejs#45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

test: fix tap parser fails if a test logs a number

9c49554

PR-URL: nodejs#46056 Fixes: nodejs#46048 Reviewed-By: Moshe Atlow <[email protected]>

test_runner: allow nesting test within describe

1502aca

PR-URL: nodejs#46544 Fixes: nodejs#46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

test_runner: flatten TAP output when running using --test

f7d8bed

PR-URL: nodejs#46440 Fixes: nodejs#45833 Refs: nodejs#45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: parse non-ascii character correctly

730f11f

PR-URL: nodejs#45736 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

test_runner: display skipped tests in spec reporter output

4a63d24

PR-URL: nodejs#46651 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

test_runner: emit test-only diagnostic warning

b50cfb3

PR-URL: nodejs#46540 Fixes: nodejs#46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

test_runner: bootstrap reporters before running tests

69100c9

PR-URL: nodejs#46737 Fixes: nodejs#46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

test_runner: add describe.only and it.only shorthands

6ef4886

PR-URL: nodejs#46604 Fixes: nodejs#46562 Reviewed-By: Moshe Atlow <[email protected]>

MoLow force-pushed the backport-v18-test-runner-reporters branch from dd5e659 to b7aefcb Compare February 25, 2023 21:55

MoLow added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2023

juanarbol pushed a commit that referenced this pull request Mar 3, 2023

test_runner: parse non-ascii character correctly

8bd3c0f

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023

test_runner: display skipped tests in spec reporter output

638bbe9

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023

test_runner: bootstrap reporters before running tests

faedcde

PR-URL: #46737 Backport-PR-URL: #46839 Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023

test_runner: add describe.only and it.only shorthands

bbe1f7d

PR-URL: #46604 Backport-PR-URL: #46839 Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

juanarbol closed this Mar 3, 2023

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: avoid swallowing of asynchronously thrown errors

5fdf374

Fixes: #44612 PR-URL: #45264 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test: fix tap escaping with and without --test

32b020c

PR-URL: #46311 Backport-PR-URL: #46839 Fixes: #45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: fix missing test diagnostics

ba784e8

PR-URL: #46450 Backport-PR-URL: #46839 Fixes: #45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test: fix tap parser fails if a test logs a number

4f5aff2

PR-URL: #46056 Backport-PR-URL: #46839 Fixes: #46048 Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: parse non-ascii character correctly

894d711

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: display skipped tests in spec reporter output

0898145

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: bootstrap reporters before running tests

28a1317

PR-URL: #46737 Backport-PR-URL: #46839 Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test_runner: add describe.only and it.only shorthands

0a690ef

PR-URL: #46604 Backport-PR-URL: #46839 Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

MoLow deleted the backport-v18-test-runner-reporters branch April 2, 2023 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v18.x backport] test_runner features #46839

[v18.x backport] test_runner features #46839

MoLow commented Feb 25, 2023 •

edited

Loading

nodejs-github-bot commented Feb 25, 2023

juanarbol commented Mar 3, 2023

[v18.x backport] test_runner features #46839

[v18.x backport] test_runner features #46839

Conversation

MoLow commented Feb 25, 2023 • edited Loading

nodejs-github-bot commented Feb 25, 2023

juanarbol commented Mar 3, 2023

MoLow commented Feb 25, 2023 •

edited

Loading