test: add error only reporter for node:test

[cjihrig]

Opening as a draft to determine if this is a desirable change.

This commit introduces a node:test reporter to the common utils. This reporter can be used to silence output other than errors from node:test. This is useful because in Node's own test suite, the output of node:test is included in the output of the Python test runner.

Refs: #49120

A comparison of output before and after this change is shown below. The TAP reporter is shown for the current output. The output is generated by ./tools/test.py test/parallel/test-runner-cli-concurrency.js with an artificial error introduced.

Before:

=== release test-runner-cli-concurrency ===                   
Path: parallel/test-runner-cli-concurrency
TAP version 13
# Subtest: default concurrency
ok 1 - default concurrency
  ---
  duration_ms: 74.002459
  ...
# Subtest: concurrency of one
ok 2 - concurrency of one
  ---
  duration_ms: 190.883208
  ...
# Subtest: concurrency of two
not ok 3 - concurrency of two
  ---
  duration_ms: 125.033667
  location: '/redacted/test/parallel/test-runner-cli-concurrency.js:23:1'
  failureType: 'testCodeFailure'
  error: 'bye'
  code: 'ERR_TEST_FAILURE'
  stack: |-
    TestContext.<anonymous> (/redacted/test/parallel/test-runner-cli-concurrency.js:27:9)
    Test.runInAsyncScope (node:async_hooks:206:9)
    Test.run (node:internal/test_runner/test:702:25)
    Test.processPendingSubtests (node:internal/test_runner/test:439:18)
    Test.postRun (node:internal/test_runner/test:807:19)
    Test.run (node:internal/test_runner/test:751:12)
    async Test.processPendingSubtests (node:internal/test_runner/test:439:7)
  ...
1..3
# tests 3
# suites 0
# pass 2
# fail 1
# cancelled 0
# skipped 0
# todo 0
# duration_ms 396.478041

After:

=== release test-runner-cli-concurrency ===                   
Path: parallel/test-runner-cli-concurrency
Test failure: 'concurrency of two'
Location: test/parallel/test-runner-cli-concurrency.js:23:1
Error: bye
    at TestContext.<anonymous> (/redacted/test/parallel/test-runner-cli-concurrency.js:27:9)
    at Test.runInAsyncScope (node:async_hooks:206:9)
    at Test.run (node:internal/test_runner/test:702:25)
    at Test.processPendingSubtests (node:internal/test_runner/test:439:18)
    at Test.postRun (node:internal/test_runner/test:807:19)
    at Test.run (node:internal/test_runner/test:751:12)
    at async Test.processPendingSubtests (node:internal/test_runner/test:439:7)

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[aduh95]

If you push the artificial error to your branch, we could try running a node-stress-single-test CI with a single run to validate how Jenkins would parse the test failure.

[cjihrig]

@aduh95 pushed

[aduh95]

CI: https://ci.nodejs.org/job/node-stress-single-test/479/

[aduh95]

Unfortunately, the Jenkins output is still not great and doesn't show the actual failure: https://ci.nodejs.org/job/node-stress-single-test/479/nodes=rhel8-x64/testReport/junit/(root)/parallel/test_runner_cli_concurrency/

EDIT: see comments below, that's not actually related to node:test, wrong assumption on my end.

[cjihrig]

Interesting. Do you know how that is rendered.

I was targeting the output in https://ci.nodejs.org/job/node-stress-single-test/479/nodes=rhel8-x64/console, which does look much better. But it would be nice to fix https://ci.nodejs.org/job/node-stress-single-test/479/nodes=rhel8-x64/testReport/junit/(root)/parallel/test_runner_cli_concurrency/.

[aduh95]

It looks like Jenkins cuts the "stacktrace" when it encounters stack: |- – maybe because it thinks it's a substack? I honestly don't know, I can try to look around if they have any documentation around that behavior.
I think we're using JUnit plugin, but it's unclear how it actually parses the test output.

[cjihrig]

Is this just a general problem not related to the use of node:test? Looking at https://ci.nodejs.org/job/node-test-commit-linux-containered/42211/#showFailuresLink, I see 9 failures, and I don't think any of those tests use node:test. All of them look like this:

fail (1)

---
duration_ms: 815.156
exitcode: 1
severity: fail
...

EDIT: This seems to be related to tap2junit I think.

[richardlau]

Unfortunately, the Jenkins output is still not great and doesn't show the actual failure: https://ci.nodejs.org/job/node-stress-single-test/479/nodes=rhel8-x64/testReport/junit/(root)/parallel/test_runner_cli_concurrency/

I think that's a tap2junit issue? Those Jenkins pages used to have the failure output in the "Stacktrace" box.

[atlowChemi]

Overall LGTM, I think this would be a nice addition

[atlowChemi]

Why not yield the values directly, instead of having the output value?

[cjihrig]

Now that nodejs/tap2junit#54 has landed, what are the next steps to seeing it working in Jenkins?

[MoLow]

it needs to be released and the updated in all the ci machines CC @nodejs/build @cclauss

[MoLow]

@cjihrig nodejs/build#3761 is now deployed. should this PR be closed?

[cjihrig]

Unsure. That PR was blocking me from evaluating this PR in the CI to see if it could be beneficial.

[cjihrig]

So this still doesn't seem to work with the view in https://ci.nodejs.org/job/node-stress-single-test/514/nodes=rhel8-x64/testReport/junit/(root)/parallel/test_runner_cli_concurrency/, and I don't know what needs to happen to appease tap2junit (both with and without the change in this PR).

The console output (shown in the OP) is still significantly better IMO.