Skip to content

OSOE-464: Making dotnet test output go to the build output to aid debugging, as well as printing ITestOutputHelper content #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 43 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
51dfe31
Not writing test outputs to a file, to be able to troubleshoot tests …
Piedone Nov 12, 2022
a997a7b
Diag test verbosity
Piedone Nov 12, 2022
0f13926
Revert Invoke-SolutionTests.ps1
Piedone Nov 12, 2022
1d5c001
Merge branch 'issue/OSOE-466' into issue/OSOE-464
Piedone Nov 12, 2022
d3fa27e
Making dotnet test go to the build output to aid debugging, as well a…
Piedone Nov 12, 2022
e3d3ec7
Removing the rest of test.out-related code
Piedone Nov 12, 2022
e371d76
Merge remote-tracking branch 'origin/dev' into issue/OSOE-464
Piedone Nov 14, 2022
2e6636d
Re-adding process-level grouping
Piedone Nov 14, 2022
c341ef1
Removing unneeded group name separator
Piedone Nov 15, 2022
03008f5
Back to no high-level test grouping, but re-adding output filtering
Piedone Nov 15, 2022
75a9726
Code styling
Piedone Nov 15, 2022
243ff58
Merge remote-tracking branch 'origin/dev' into issue/OSOE-464
Piedone Nov 15, 2022
2ce191d
Using full cmdlet name instead of single-character aliases for better…
Piedone Nov 15, 2022
82a5323
Docs
Piedone Nov 15, 2022
0eeda56
Testing that the basic live-output test execution also causes expecte…
Piedone Nov 15, 2022
a5c1a40
Revert "Testing that the basic live-output test execution also causes…
Piedone Nov 15, 2022
8597152
Pointing build-dotnet to issue branch too
Piedone Nov 15, 2022
659dc73
Leftover hashmark
Piedone Nov 15, 2022
6998a75
Output about the number of tests
Piedone Nov 15, 2022
220d729
Output about test execution start
Piedone Nov 15, 2022
e81ce27
Debug output for workflow cancellation
Piedone Nov 15, 2022
84141ba
Perhaps exit 1 will make the run fail?
Piedone Nov 16, 2022
5a9a831
Removing debug code
Piedone Nov 16, 2022
4912bec
Error output for better understanding when canceling the workflow
Piedone Nov 16, 2022
7827fa4
Exit 1 is of no use
Piedone Nov 16, 2022
9a5b62a
Temporarily disabling .NET setup
Piedone Nov 16, 2022
7f4e3e9
JS action skeleton for post-job workflow cancelation
Piedone Nov 16, 2022
829c298
Trying curious syntax for post-job commands
Piedone Nov 16, 2022
1f56fac
Adding package.json for action
Piedone Nov 16, 2022
96383e0
Revert "Trying curious syntax for post-job commands"
Piedone Nov 16, 2022
187e1d8
Moving package.json to action folder
Piedone Nov 16, 2022
92eb9f7
Back to composite action
Piedone Nov 16, 2022
3825d1e
Trying webiny/action-post-run
Piedone Nov 16, 2022
9fa4d0b
Syntax
Piedone Nov 16, 2022
829fa11
Separate PS file
Piedone Nov 16, 2022
9795fed
Path to script file
Piedone Nov 16, 2022
e30a536
Files don't work, let's try curl
Piedone Nov 16, 2022
4f93303
Windows test
Piedone Nov 16, 2022
acb5f5f
Syntax fix
Piedone Nov 16, 2022
6865a0b
Back to original action with docs
Piedone Nov 16, 2022
0fdd730
Revert machine-type handling
Piedone Nov 16, 2022
2965e67
Re-activating .NET setup step
Piedone Nov 16, 2022
fbdbebb
Revert branch selectors.
sarahelsaig Nov 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 3 additions & 9 deletions .github/actions/test-dotnet/Invoke-SolutionTests.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -43,27 +43,21 @@ foreach ($test in $tests) {
'--no-build',
'--nologo',
'--logger', 'trx;LogFileName=test-results.trx'
# This is for xUnit ITestOutputHelper, see https://xunit.net/docs/capturing-output.
'--logger', 'console;verbosity=detailed'
'--verbosity', $Verbosity
[string]::IsNullOrEmpty($Filter) ? '' : "--filter"
[string]::IsNullOrEmpty($Filter) ? '' : $Filter
$test
)

dotnet test @dotnetTestSwitches 2>&1 >test.out
dotnet test @dotnetTestSwitches

if ($?)
{
Write-Output "Test Successful: $test"
continue
}

$needsGrouping = (Select-String "::group::" test.out).Length -eq 0

if ($needsGrouping) { Write-Output "::group::Test Failed: $test" }

bash -c "cat test.out | grep -v 'Connection refused \[::ffff:127.0.0.1\]' | grep -v 'ChromeDriver was started successfully'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we still filter out these pointless "Connection refused" and "ChromeDriver" log lines on dotnet test @dotnetTestSwitches?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know or have found a way to live-filter a streaming output, though I only have smallish experience with PS.

The goal here is for the output of the tests to show up in the log continuously (to pinpoint what the last thing was before the runner was killed due to timeout). If there's a file at the end of the dotnet test call then you can grep like this, but we can't wait for a file.

We could still redirect to a file and tail it and that can be filtered but going through a file doesn't seem like the best idea, and we also need to somehow know when the stream actually ended and quit the Wait.

So, I don't know. It seems to me that implementing grouping around the test project if it doesn't have its custom group and letting UI tests group themselves, as well as filtering out such messages need a file. If we have a file, however, we won't have a continuous output.

Do you have any suggestions by chance?

BTW I'm also trying to figure out where the annoying log messages come from and disable them in the first place.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dotnet test @dotnetTestSwitches 2>&1 | ? { $_ -notlike '*Connection refused [[]::ffff:127.0.0.1[]]*' -and $_ -notlike '*ChromeDriver was started successfully*' } should filter it line-by-line as it's passed to the pipe. We can live without the outer grouping until they finally add named groups to the runner.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nice, so this works like this!


if ($needsGrouping) { Write-Output "::endgroup::" }

exit 100
}
4 changes: 1 addition & 3 deletions .github/actions/test-dotnet/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,7 @@ runs:
if: success() || failure()
with:
name: ui-test-failure-dump-${{ steps.setup.outputs.friendly-build-directory-name }}-${{ steps.setup.outputs.runner-suffix }}
path: |
${{ inputs.build-directory }}/FailureDumps/
test.out
path: ${{ inputs.build-directory }}/FailureDumps/
if-no-files-found: ignore

- name: Test Report
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-and-test-orchard-core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ jobs:
location: ${{ inputs.build-directory}}

- name: Tests
uses: Lombiq/GitHub-Actions/.github/actions/test-dotnet@issue/OSOE-466
uses: Lombiq/GitHub-Actions/.github/actions/test-dotnet@issue/OSOE-464
with:
build-directory: ${{ inputs.build-directory }}
test-verbosity: ${{ inputs.build-verbosity }}
Expand Down