`dotnet test --blame-crash` collects crash DMP even after successful test runs

[AArnott]

⚠ This is a regression in the .NET 5 SDK and blocks adoption of the .NET 5 SDK by this template repo and others.

When running dotnet test --blame-crash, a dmp file is produced at the conclusion of each test run even though no crash or timeout occurred.

Repro steps

git clone https://github.com/aarnott/library.template
cd library.template
git checkout c234c68d0b45e5cc621db26063c199b51b4a45fd
dotnet test --blame-crash

Observed output is below. Notice lines such as:

D:\git\lib.template\test\Library.Tests\TestResults\5d81aeb2-f632-461f-b151-7f96f9b8005f\testhost_22312_20201117T083038_crashdump.dmp

 dotnet test --blame-crash
  Determining projects to restore...
  All projects are up-to-date for restore.
  Library -> D:\git\lib.template\bin\Library\Debug\netstandard2.0\Library.dll
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
  Library.Tests -> D:\git\lib.template\bin\Library.Tests\Debug\net472\Library.Tests.dll
  Library.Tests -> D:\git\lib.template\bin\Library.Tests\Debug\netcoreapp3.1\Library.Tests.dll
  Library.Tests -> D:\git\lib.template\bin\Library.Tests\Debug\netcoreapp2.1\Library.Tests.dll
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collector 'Blame' message: All tests finished running, Sequence file will not be generated.

Attachments:
  D:\git\lib.template\test\Library.Tests\TestResults\98a4200c-be91-4880-917b-9633e603c62f\testhost_35340_20201117T083036_crashdump.dmp
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 3 ms - Library.Tests.dll (netcoreapp3.1)
Test run for D:\git\lib.template\bin\Library.Tests\Debug\netcoreapp2.1\Library.Tests.dll (.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collector 'Blame' message: All tests finished running, Sequence file will not be generated.

Attachments:
  D:\git\lib.template\test\Library.Tests\TestResults\5d81aeb2-f632-461f-b151-7f96f9b8005f\testhost_22312_20201117T083038_crashdump.dmp
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 6 ms - Library.Tests.dll (netcoreapp2.1)
Test run for D:\git\lib.template\bin\Library.Tests\Debug\net472\Library.Tests.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collector 'Blame' message: All tests finished running, Sequence file will not be generated.

Attachments:
  D:\git\lib.template\test\Library.Tests\TestResults\22ff3440-7aec-4a15-b81e-a90ef4f607ca\testhost.net472.x86_12208_20201117T083041_crashdump.dmp
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 8 ms - Library.Tests.dll (net472)

Build succeeded.

D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1512: Single-line comments should not be followed by blank line [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
D:\git\lib.template\test\Library.Tests\CalculatorTests.cs(14,5): warning SA1005: Single line comment should begin with a space. [D:\git\lib.template\test\Library.Tests\Library.Tests.csproj]
    6 Warning(s)
    0 Error(s)

Time Elapsed 00:00:09.54

[AArnott]

@nohwnd noticed that this only happens when the tests run on runtimes before .NET 5.0. But since nearly everyone targets less than .NET 5.0 at this point, that seems like a crucial scenario to test and support.

[AArnott]

Just to be clear about the impact of this bug, when a pipeline is carefully written to collect crash dumps so that any test failures can be investigated, having a dump unnecessarily created ends up getting collected as an artifact, which slows down the pipeline and greatly swells the storage requirements for that cloud build account. The below screenshot shows how Windows agents (the only OS that supports dumps) collect nearly 1GB of dmp's vs. a few KB of files from the non-Windows agents where no dmp file creation bug exists:

[nohwnd]

This should be fixed on the latest release/5.1.2xx and master build. Please give it a try. I did locally and it no longer collects dump when there is no crash. I used version 5.0.200-preview.20601.7

inserted in #14764
inserted in #14782

[AArnott]

Thanks, @nohwnd. But it's still broken. Now instead of collecting dumps from every run (successful or not), it never collects dumps. .NET 5.0 works fine as was the case before. But <.NET 5.0 test runs now never collect crash dumps.
Check out this log from AzP.