.NET Test Log are inaccurate or are missing some details

[JunTaoLuo]

Environment data

dotnet --info output: 1.0.0-preview2-003118
VS Code version: 1.2.1
C# Extension version: 1.1.6

Skipped tests are marked as passed

Currently tests marked as skipped will show a passed result when run from VS Code. For example:

        Fact(Skip = "Verifying skipping tests")]
        [Trait("Testing", "skip")]
        public async void SkipFactTest()
        {
            ...
        }

outputs:

Running test NETCore.Tests.NETCore10Tests.SkipFactTest...
Test passed 

It would be nice if instead the test indicates that the tests were skipped. It would also be useful if the skip reason is also displayed as is currently done by the console runner. Note that this information can be currently found in the OmniSharp Log but it's noisy and hard to extract the relevant information

Theory tests do not indicate how many tests were run and how many passed or failed

Similar to the previous case, .NET Test Log does not indicate how many iterations of a Theory test is run and how many of them passed or failed. This information is readily available if the test was run with the console runner. For example

        [Theory]
        [InlineData(0)]
        [InlineData(1)]
        [Trait("Testing", "int")]
        public void InlineDataTests(int number)
        {
            Assert.True(number > 0);
        }

outputs:

Running test NETCore.Tests.NETCore10Tests.InlineDataTests...
Test failed 

It is difficult to extract the fact that two tests were run, for inputs 0 and 1, and the input of 0 caused a failure while the input of 1 passed. Even in the OmniSharp Log it is difficult to see this:

[INFORMATION:OmniSharp.DotNetTest.Helpers.DotNetTestManager.DotNetTestManager] read: {"MessageType":"TestExecution.TestStarted","Payload":{"CodeFilePath":null,"DisplayName":"NETCore.Tests.NETCore10Tests.InlineDataTests(number: 1)","FullyQualifiedName":"NETCore.Tests.NETCore10Tests.InlineDataTests(404a12ed4144efffd791150cf180f32f0d6c56d6)","Id":"1345ebaf-613d-1a67-95f3-ec76c96ca195","LineNumber":null,"Properties":{}}}
[INFORMATION:OmniSharp.DotNetTest.Helpers.DotNetTestManager.DotNetTestManager] read: {"MessageType":"TestExecution.TestResult","Payload":{"Test":{"CodeFilePath":null,"DisplayName":"NETCore.Tests.NETCore10Tests.InlineDataTests(number: 1)","FullyQualifiedName":"NETCore.Tests.NETCore10Tests.InlineDataTests(404a12ed4144efffd791150cf180f32f0d6c56d6)","Id":"1345ebaf-613d-1a67-95f3-ec76c96ca195","LineNumber":null,"Properties":{}},"Outcome":1,"ErrorMessage":null,"ErrorStackTrace":null,"DisplayName":null,"Messages":[],"ComputerName":null,"Duration":"00:00:00.0090000","StartTime":"2016-06-20T15:41:05.349957-07:00","EndTime":"2016-06-20T15:41:05.405497-07:00"}}
[INFORMATION:OmniSharp.DotNetTest.Helpers.DotNetTestManager.DotNetTestManager] read: {"MessageType":"TestExecution.TestStarted","Payload":{"CodeFilePath":null,"DisplayName":"NETCore.Tests.NETCore10Tests.InlineDataTests(number: 0)","FullyQualifiedName":"NETCore.Tests.NETCore10Tests.InlineDataTests(62dd32adac6b68b101fd4c6898501e43c9aac5c9)","Id":"d915c95c-c7b9-eae1-de82-749baa9c6399","LineNumber":null,"Properties":{}}}
[INFORMATION:OmniSharp.DotNetTest.Helpers.DotNetTestManager.DotNetTestManager] read: {"MessageType":"TestExecution.TestResult","Payload":{"Test":{"CodeFilePath":null,"DisplayName":"NETCore.Tests.NETCore10Tests.InlineDataTests(number: 0)","FullyQualifiedName":"NETCore.Tests.NETCore10Tests.InlineDataTests(62dd32adac6b68b101fd4c6898501e43c9aac5c9)","Id":"d915c95c-c7b9-eae1-de82-749baa9c6399","LineNumber":null,"Properties":{}},"Outcome":2,"ErrorMessage":"Assert.True() Failure\nExpected: True\nActual:   False","ErrorStackTrace":"   at Xunit.Assert.True(Nullable`1 condition, String userMessage)\n   at NETCore.Tests.NETCore10Tests.InlineDataTests(Int32 number) in /Users/jtluo/Documents/workspace/juntaoluo/tp/DotnetTest/test/NETCore10.Tests/NETCore10Tests.cs:line 41","DisplayName":null,"Messages":[""],"ComputerName":null,"Duration":"00:00:00.0140000","StartTime":"2016-06-20T15:41:05.444392-07:00","EndTime":"2016-06-20T15:41:05.446223-07:00"}}
    NETCore.Tests.NETCore10Tests.InlineDataTests(number: 0) [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /Users/jtluo/Documents/workspace/juntaoluo/tp/DotnetTest/test/NETCore10.Tests/NETCore10Tests.cs(41,0): at NETCore.Tests.NETCore10Tests.InlineDataTests(Int32 number)
  Finished:    NETCore10.Tests

Custom theory tests results report incorrect input parameters

See repro at https://github.com/JunTaoLuo/DotnetTest. Use the test project under test/NETCore10.Tests and run the test NETCore.Tests.NETCore10Tests.SkipTheoryTests. It is strange to see that there are 4 variations that are detected and run. In the OmniSharp Log I see:

    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 0)(number: 0) [SKIP]
      Test must be skipped
    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 0)(number: 1) [SKIP]
      Test must be skipped
    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 1)(number: 0) [SKIP]
      Test must be skipped
    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 1)(number: 1) [SKIP]
      Test must be skipped

When run with the command line runner I see the two variations that I expected:

xUnit.net .NET CLI test runner (64-bit .NET Core osx.10.11-x64)
  Discovering: NETCore10.Tests
  Discovered:  NETCore10.Tests
  Starting:    NETCore10.Tests
    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 0) [SKIP]
      Test must be skipped
    NETCore.Tests.NETCore10Tests.SkipTheoryTests(number: 1) [SKIP]
      Test must be skipped
  Finished:    NETCore10.Tests
=== TEST EXECUTION SUMMARY ===
   NETCore10.Tests  Total: 2, Errors: 0, Failed: 0, Skipped: 2, Time: 0.128s

Minor GUI inconsistency

This is a minor inconvenience but the references, run test and debug test options show up in different orders for different test methods sometimes. It could be annoying since I often mis click when I assume the last option is run test. For example.

Note that these issues were found on VS Code for OSX.
cc @troydai

[JunTaoLuo]

Other improvements that I have not seen filed elsewhere:

• Need to set debugtype to portable on Windows for .NET Core test projects
• Debugging tests does not output test result (failed, passed)
• Test results only show up as passed or failed, exception that caused the test failures are in the Omnisharp output log.
• Cannot specify TFM or Configuration
• Cancel a running or hanging test. Maybe once you click run test, the text can change to cancel test?

[troydai]

Related: #419