[9.0-preview.2] Inconsistent messaging of success vs. warnings from dotnet test

[martincostello]

Describe the bug

When tests are intentionally skipped, the .NET SDK reports inconsistent messaging on whether the build/tests succeeded. This behaviour also varies between running locally and in GitHub Actions (interactive vs. not).

.NET 8

  Determining projects to restore...
  Restored /home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj (in 733 ms).
  LondonTravel.Skill.EndToEndTests -> /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll
Test run for /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.27]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [SKIP]
[xUnit.net 00:00:00.28]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [SKIP]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [1 ms]
  Skipped Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [1 ms]
  Skipped Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [1 ms]
[xUnit.net 00:00:00.33]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [SKIP]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [1 ms]

Skipped! - Failed:     0, Passed:     0, Skipped:    10, Total:    10, Duration: 49 ms - LondonTravel.Skill.EndToEndTests.dll (net8.0)

.NET 9 in CI

  Determining projects to restore...
  Restored /home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj (in 948 ms).
/usr/share/dotnet/sdk/9.0.100-preview.2.24157.14/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(314,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj]
  LondonTravel.Skill.EndToEndTests -> /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll

.NET 9 Locally

Restore complete (0.7s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  LondonTravel.Skill.EndToEndTests succeeded (0.4s) → artifacts\bin\LondonTravel.Skill.EndToEndTests\release\LondonTravel.Skill.EndToEndTests.dll
  LondonTravel.Skill.EndToEndTests test succeeded with warnings (2.0s)
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.36]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.41]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]

Build succeeded with warnings in 3.8s
Test run succeeded. Total: 10 Failed: 0 Passed: 0 Skipped: 10, Duration: 2.0s

In CI there's no detail of the skipped tests (or indeed that any tests were even run).

Locally, the messaging is inconsistent (emphasis mine):

• LondonTravel.Skill.EndToEndTests succeeded
• LondonTravel.Skill.EndToEndTests test succeeded with warnings
• Build succeeded with warnings in 3.8s
• Test run succeeded. Total: 10 Failed: 0 Passed: 0 Skipped: 10, Duration: 2.0s

The messaging suggests that build and test both succeeded and succeeded with warning.

Based on my intent at least, in this scenario I would expect the build and the tests to both be successful because no tests failed.

If regardless of my intent skipping should be treated as a warning, then then messaging should be consistent and arguably it should be possible to treat skipped tests as not being at a warning level.

To Reproduce

1. Clone martincostello/alexa-london-travel@1152660
2. Run build.ps1 from the root of the repository.

Further technical details

.NET SDK 9.0.100-preview.2.24157.14

/cc @nohwnd

[martincostello]

The CI vs. not thing might actually be a different issue.

Carrying on with what I was actually doing when I observed the original issue, I find that in CI the tests failing isn't being reported on the console - I just get /usr/share/dotnet/sdk/9.0.100-preview.2.24157.14/Microsoft.TestPlatform.targets(46,5): error MSB6006: "dotnet" exited with code 1. [/home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj].

I can only see why the tests failed with the GitHub Actions test logger.

[nohwnd]

I will fix net9 output when all tests are skipped to say Skipped! rather than Passed!.

The CI output is interesting, you are using the preview2 version which should print errors via msbuild api, so they should show up on console, maybe it is the extra logger the action is providing, but it seems unlikely.

Most of my updates on terminal logger integration go here: microsoft/vstest#4843

[nohwnd]

I've re-read this again, and looked at terminal logger code, and I think it will be for the best if the test summary in terminal logger does not report success or fail, but instead just says: Test summary: followed by test counts. All error that are reported will be accounted for in the result of the build, as well as all warnings, so repeating the status is only asking for inconcistencies.

[martincostello]

The CI output is interesting, you are using the preview2 version which should print errors via msbuild api, so they should show up on console, maybe it is the extra logger the action is providing, but it seems unlikely.

Observing this still with preview.3 - here's an example.

[baronfel]

We couldn't get this into the preview 3 cutoff, but the PR is undergoing approvals and should be in preview 4.