Update Microsoft.Net.Test.Sdk version by RikkiGibson · Pull Request #44442 · dotnet/roslyn

RikkiGibson · 2020-05-20T21:26:57Z

Thanks to @nohwnd for following up with me about this 😄

There have been improvements to the test SDK that we have not been utilizing, such as improved behavior around Debug.Assert in .NET Core. Instead of crashing the test process we now just fail the single test. I know you mentioned this recently @gafter.

RikkiGibson · 2020-05-21T17:13:13Z

Found the following netcoreapp3.1 test failures with the new test SDK and not with the older version:

xUnit.net VSTest Adapter v2.4.1-pre.build.4059 (64-bit .NET Core 3.1.4)
  Starting:    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests
    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithRedirectedInputIndicatorAndStandardInputNotRedirected_ReportsCS8782 [FAIL]
      Assert.Contains() Failure
      Not found: 8772
      In value:  
      Stack Trace:
        C:\Users\rikki\src\roslyn\src\Compilers\CSharp\Test\CommandLine\CommandLineTests.cs(5836,0): at Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithRedirectedInputIndicatorAndStandardInputNotRedirected_ReportsCS8782()
    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithSourceCodeRedirectedViaStandardInput_ProducesRunnableProgram [FAIL]
      Compilation error(s) occurred:  A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.
      
      Expected: False
      Actual:   True
      Stack Trace:
        C:\Users\rikki\src\roslyn\src\Compilers\CSharp\Test\CommandLine\CommandLineTests.cs(5753,0): at Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithSourceCodeRedirectedViaStandardInput_ProducesRunnableProgram()
    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithSourceCodeRedirectedViaStandardInput_ProducesLibrary [FAIL]
      Compilation error(s) occurred:  A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.
      
      Expected: False
      Actual:   True
      Stack Trace:
        C:\Users\rikki\src\roslyn\src\Compilers\CSharp\Test\CommandLine\CommandLineTests.cs(5786,0): at Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithSourceCodeRedirectedViaStandardInput_ProducesLibrary()
    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithMultipleStdInOperators_WarnsCS2002 [FAIL]
      Assert.Contains() Failure
      Not found: 2002
      In value:  
      Stack Trace:
        C:\Users\rikki\src\roslyn\src\Compilers\CSharp\Test\CommandLine\CommandLineTests.cs(5864,0): at Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CscCompile_WithMultipleStdInOperators_WarnsCS2002()
    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CsiScript_WithSourceCodeRedirectedViaStandardInput_ExecutesNonInteractively [FAIL]
      Compilation error(s) occurred:  A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.
      
      Expected: False
      Actual:   True
      Stack Trace:
        C:\Users\rikki\src\roslyn\src\Compilers\CSharp\Test\CommandLine\CommandLineTests.cs(5808,0): at Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests.CommandLineTests.CsiScript_WithSourceCodeRedirectedViaStandardInput_ExecutesNonInteractively()
  Finished:    Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests

I see the following test failures in net472:

xUnit.net VSTest Adapter v2.4.1-pre.build.4059 (32-bit Desktop .NET 4.0.30319.42000)
  Starting:    Microsoft.CodeAnalysis.UnitTests
    Microsoft.CodeAnalysis.UnitTests.ModuleMetadataTests.CreateFromFile [FAIL]
      Assert.Throws() Failure
      Expected: typeof(System.IO.IOException)
      Actual:   typeof(System.ArgumentException): URI formats are not supported.
      ---- System.ArgumentException : URI formats are not supported.
      Stack Trace:
           at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
           at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
           at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength)
           at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
           at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
           at System.IO.File.OpenRead(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\Portable\FileSystem\FileUtilities.cs(412,0): at Roslyn.Utilities.FileUtilities.OpenFileStream(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\Portable\MetadataReference\ModuleMetadata.cs(200,0): at Microsoft.CodeAnalysis.ModuleMetadata.CreateFromFile(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\CodeAnalysisTest\MetadataReferences\ModuleMetadataTests.cs(92,0): at Microsoft.CodeAnalysis.UnitTests.ModuleMetadataTests.<>c.<CreateFromFile>b__5_3()
        ----- Inner Stack Trace -----
           at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
           at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
           at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength)
           at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
           at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
           at System.IO.File.OpenRead(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\Portable\FileSystem\FileUtilities.cs(412,0): at Roslyn.Utilities.FileUtilities.OpenFileStream(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\Portable\MetadataReference\ModuleMetadata.cs(200,0): at Microsoft.CodeAnalysis.ModuleMetadata.CreateFromFile(String path)
        C:\Users\rikki\src\roslyn\src\Compilers\Core\CodeAnalysisTest\MetadataReferences\ModuleMetadataTests.cs(92,0): at Microsoft.CodeAnalysis.UnitTests.ModuleMetadataTests.<>c.<CreateFromFile>b__5_3()
  Finished:    Microsoft.CodeAnalysis.UnitTests

The net472 failure seems simple to fix, but not sure about the netcoreapp3.1 failures. Also, why would the tests have different behavior in the VS test runner compared to Test.cmd or in CI? Wondering if you have any thoughts on where to start investigating this difference @shyamnamboodiripad @nohwnd?

shyamnamboodiripad · 2020-05-21T18:26:04Z

@RikkiGibson Thanks for the ping. I am not sure why those would only start repro-ing after the SDK update. @nohwnd would probably know. Also tagging @AbhitejJohn to see if he has insights.

AbhitejJohn · 2020-05-21T19:04:38Z

Those could be related to the presence of a laucnhsettings.json here. Do we still need that if we we are running these tests with vstest.console/TP?

jaredpar · 2020-05-21T19:24:13Z

The net472 failure seems simple to fix, but not sure about the netcoreapp3.1 failures

True it's easy to fix but I want to understand what changed here? How can updating this package cause our tests to fail?

nohwnd · 2020-06-05T06:27:52Z

Missed this notification, did you manage to fix it? When test host fails to locate hostpolicy it is usually because the runtime is not on the system. 16.3.0 started running tests in their correct bitness and this uncovered missing dependencies, because often the host system is missing x86 runtime.

But the error message mentions 'C:\Program Files\dotnet' so missing 32-bit was probably not the problem.

RikkiGibson · 2020-06-05T17:29:02Z

Not yet, will revisit this within a few weeks. It still feels wrong that the tests pass on command line but fail in the IDE. Surely the console runner is running the tests with correct bitness.

RikkiGibson · 2020-06-17T16:21:09Z

@nohwnd Have been looking into this more and found that the tests in question work in 16.2.0 but break in 16.3.0.

RikkiGibson · 2020-06-22T17:13:19Z

I think the description of the problem here was a little unclear because I did not specify that we use the Xunit runner in CI. We get the test failures when using 'dotnet test' as well as in Test Explorer.

nohwnd · 2020-09-01T09:13:27Z

We started multitargetting .NET Framework in 16.8 you might want to update to that version 16.8.0-preview-20200812-03 https://www.nuget.org/packages/Microsoft.NET.Test.Sdk/16.8.0-preview-20200812-03

As for the original error with netcoreapp3.1 is because in between 16.2.0 and 16.3.0 there was a big change which changed how tests are executed. Before that it would not respect the architecture (x86/x64) and would simply use the dotnet you were currently running. There were also few bugs regarding the resolve of dotnet SDK, which in this case ended up resolving to C:\Program Files\dotnet, where you don't have any runtime (based on the message), while before it would resolve to C:\Program Files (x86)\dotnet\ based on the path of the 32-bit dotnet.exe that was running the tests.

sharwell · 2020-09-01T16:47:44Z

⚠️ We should verify that we are not impacted by microsoft/vstest#2192 and microsoft/vstest#2067, which permanently broke this package after 16.1.1 for dotnet/roslyn-sdk.

jaredpar · 2020-09-01T17:39:23Z

@sharwell can u elaborate a bit on what broke and why?

RikkiGibson · 2020-09-01T18:37:12Z

I wasn't able to run all tests successfully using dotnet test or VS 16.8-preview2. The failures are different in each place. I will spin up on this a little later and try to report more specifics once I can verify that my build environment is clean, etc.

sharwell · 2020-09-01T19:41:05Z

@jaredpar support for netstandard1.x and netcoreapp1.x was completely removed

nohwnd · 2020-09-02T14:27:14Z

@RikkiGibson could you share the errors? Maybe I saw them somewhere already.

RikkiGibson · 2020-11-03T18:03:12Z

Superseded by #48817

Update Microsoft.Net.Test.Sdk version

f50c795

RikkiGibson added the Area-Infrastructure label May 20, 2020

RikkiGibson requested a review from a team as a code owner May 20, 2020 21:26

nohwnd mentioned this pull request Jun 5, 2020

Incorrect result is produced by DataFlowAnalysis.DataFlowsOut when local functions are involved #41600

Open

Merge branch 'master' of github.com:dotnet/roslyn into update-test-sdk

458c124

RikkiGibson added 2 commits September 1, 2020 09:40

Merge branch 'master' into update-test-sdk

184ce7f

Update Versions.props

4ee1400

RikkiGibson closed this Nov 3, 2020

RikkiGibson deleted the update-test-sdk branch November 3, 2020 18:03

Uh oh!

Conversation

RikkiGibson commented May 20, 2020

Uh oh!

RikkiGibson commented May 21, 2020

Uh oh!

shyamnamboodiripad commented May 21, 2020

Uh oh!

AbhitejJohn commented May 21, 2020

Uh oh!

jaredpar commented May 21, 2020

Uh oh!

nohwnd commented Jun 5, 2020

Uh oh!

RikkiGibson commented Jun 5, 2020

Uh oh!

RikkiGibson commented Jun 17, 2020

Uh oh!

RikkiGibson commented Jun 22, 2020

Uh oh!

nohwnd commented Sep 1, 2020

Uh oh!

sharwell commented Sep 1, 2020

Uh oh!

jaredpar commented Sep 1, 2020

Uh oh!

RikkiGibson commented Sep 1, 2020

Uh oh!

sharwell commented Sep 1, 2020

Uh oh!

nohwnd commented Sep 2, 2020

Uh oh!

RikkiGibson commented Nov 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants