tools: improve node:test output when running in github actions
#49129
Conversation
|
Review requested:
|
nodejs-github-bot
added
meta
|
@MoLow This seems to have not worked? /opt/hostedtoolcache/Python/3.11.4/x64/bin/python3 tools/test.py -p tap --logfile test.tap \
--mode=release --flaky-tests=keep_retrying \
-p actions --node-args='--test-reporter=spec' --node-args='--test-reporter-destination=stdout' -t 300 --measure-flakiness 9 default pummel addons js-native-api node-api benchmark
Running 4010 tests
.................................................. |
@atlowChemi this will only change the behavior in case of failed tests |
| @@ -94,4 +94,4 @@ jobs: | |||
| - name: Test | |||
| run: | | |||
| cd $TAR_DIR | |||
| make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9" | |||
| make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --node-args='--test-reporter=spec' --measure-flakiness 9" | |||
There was a problem hiding this comment.
This should be an improvement in terms of the amount of output we see from failed tests, which should help with debugging.
There is also the issue of the asan CI job truncating output. Since node:test is never used as the main test runner, a possible next step to address that issue could be to create a custom reporter in tools/ that only outputs the test failures and diagnostic messages. It could basically be a slimmed down version of the spec reporter.
There was a problem hiding this comment.
I guess we can do that in a follow-up. perhaps only in the asan job
| @@ -94,4 +94,4 @@ jobs: | |||
| - name: Test | |||
| run: | | |||
| cd $TAR_DIR | |||
| make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9" | |||
| make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --node-args='--test-reporter=spec' --measure-flakiness 9" | |||
There was a problem hiding this comment.
It looks like the CI ran fine so I guess there is no issue, but are we sure this won't impact any of the tests that expect different reporters?
There was a problem hiding this comment.
no issue. tests that look at reporters (e.g snapshot tests, exit code tests etc) always spawn a child and control the env vars and flags
| def AboutToRun(self, case): | ||
| if not hasattr(case, 'additional_flags'): | ||
| case.additional_flags = [] | ||
| case.additional_flags.append('--test-reporter=./tools/github_reporter/index.js') | ||
| case.additional_flags.append('--test-reporter-destination=stdout') |
There was a problem hiding this comment.
If I understand this correctly, this is adding two additional flags to all the tests that run. This feels worrisome, because as a test author I’d expect the flags I define at the top of the test file to be the only flags that are used; I can imagine issues with process.argv and so on. Could these be defined via environment variables instead?
There was a problem hiding this comment.
this is only the case in github actions, within jenkins or locally the flags wont be added.
the only way to add this via an environment variable is with NODE_OPTIONS
|
Adding the |
done |
MoLow
added
request-ci
github-actions
bot
removed
the
request-ci
MoLow
added
author ready
nodejs-github-bot
removed
the
commit-queue
|
Landed in 802c52f...9cc7327 |
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#49129 Refs: nodejs#49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #49129 Refs: #49120 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
Refs: #49120