roslyn.visualstudio.newintegrationtests.csharp.csharperrorlistnetcore.errorsafterclosingfile is flaky

[Cosifne]

See https://runfo.azurewebsites.net/search/tests/?q=started%3A~7%20definition%3Aroslyn-integration-ci%20name%3A%22roslyn.visualstudio.newintegrationtests.csharp.csharperrorlistnetcore.errorsafterclosingfile%20%22

Specific issues impacting this test include:

• Cascading test failures due to a harness failure are not clearly indicated microsoft/vs-extension-testing#124
• Integration tests should clearly indicate "failure to save" #62109

Runfo Tracking Issue: roslyn.visualstudio.newintegrationtests.csharp.csharperrorlistnetcore.errorsafterclosingfile

Build	Definition	Kind	Run Name
1839692	roslyn-integration-CI	PR 61646	1-Integration Debug OOP64_true OOPCoreClr_false
1839692	roslyn-integration-CI	PR 61646	1-Integration Debug OOP64_true OOPCoreClr_false
1837363	roslyn-integration-CI	PR 60219	1-Integration Release OOP64_false OOPCoreClr_false
1837363	roslyn-integration-CI	PR 60219	1-Integration Release OOP64_false OOPCoreClr_false
1837363	roslyn-integration-CI	PR 60219	1-Integration Debug OOP64_true OOPCoreClr_false
1837363	roslyn-integration-CI	PR 60219	1-Integration Debug OOP64_true OOPCoreClr_false
1835963	roslyn-integration-CI	PR 62047	1-Integration Release OOP64_false OOPCoreClr_false
1835963	roslyn-integration-CI	PR 62047	1-Integration Release OOP64_false OOPCoreClr_false
1835546	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1835546	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1835542	roslyn-integration-CI	PR 62035	1-Integration Release OOP64_false OOPCoreClr_false
1835542	roslyn-integration-CI	PR 62035	1-Integration Release OOP64_false OOPCoreClr_false
1833101	roslyn-integration-CI	PR 59369	1-Integration Release OOP64_false OOPCoreClr_false
1833101	roslyn-integration-CI	PR 59369	1-Integration Release OOP64_false OOPCoreClr_false
1831761	roslyn-integration-CI	PR 60219	1-Integration Debug OOP64_true OOPCoreClr_false
1831761	roslyn-integration-CI	PR 60219	1-Integration Debug OOP64_true OOPCoreClr_false
1831761	roslyn-integration-CI	PR 60219	1-Integration Release OOP64_false OOPCoreClr_false
1831761	roslyn-integration-CI	PR 60219	1-Integration Release OOP64_false OOPCoreClr_false
1831281	roslyn-integration-CI	PR 61997	1-Integration Debug OOP64_true OOPCoreClr_false
1831281	roslyn-integration-CI	PR 61997	1-Integration Debug OOP64_true OOPCoreClr_false
1831098	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1831098	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1831004	roslyn-integration-CI	PR 61989	1-Integration Release OOP64_false OOPCoreClr_false
1831004	roslyn-integration-CI	PR 61989	1-Integration Release OOP64_false OOPCoreClr_false
1830412	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1830412	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1830412	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1830412	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1830280	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1830280	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1830190	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1830190	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1830190	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1830190	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1829981	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1829981	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1829851	roslyn-integration-CI	PR 61975	1-Integration Debug OOP64_true OOPCoreClr_false
1829851	roslyn-integration-CI	PR 61975	1-Integration Debug OOP64_true OOPCoreClr_false
1827403	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1827403	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1827356	roslyn-integration-CI	PR 61937	1-Integration Release OOP64_false OOPCoreClr_false
1827356	roslyn-integration-CI	PR 61937	1-Integration Release OOP64_false OOPCoreClr_false
1827332	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1827332	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1827298	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1827298	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1827298	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1827298	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1827243	roslyn-integration-CI	PR 61934	1-Integration Release OOP64_false OOPCoreClr_false
1827243	roslyn-integration-CI	PR 61934	1-Integration Release OOP64_false OOPCoreClr_false
1826904	roslyn-integration-CI	PR 61926	1-Integration Debug OOP64_true OOPCoreClr_false
1826904	roslyn-integration-CI	PR 61926	1-Integration Debug OOP64_true OOPCoreClr_false
1826893	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1826893	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1826634	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1826634	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1826563	roslyn-integration-CI	PR 61899	1-Integration Debug OOP64_true OOPCoreClr_false
1826563	roslyn-integration-CI	PR 61899	1-Integration Debug OOP64_true OOPCoreClr_false
1825560	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1825560	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1825560	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1825560	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1825307	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1825307	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1825236	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1825236	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1825067	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1825067	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1825054	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1825054	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1824984	roslyn-integration-CI	PR 61903	1-Integration Debug OOP64_true OOPCoreClr_false
1824984	roslyn-integration-CI	PR 61903	1-Integration Debug OOP64_true OOPCoreClr_false
1824892	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1824892	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1824682	roslyn-integration-CI	PR 61895	1-Integration Release OOP64_false OOPCoreClr_false
1824682	roslyn-integration-CI	PR 61895	1-Integration Release OOP64_false OOPCoreClr_false
1824452	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1824452	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1824214	roslyn-integration-CI	PR 61884	1-Integration Debug OOP64_true OOPCoreClr_false
1824214	roslyn-integration-CI	PR 61884	1-Integration Debug OOP64_true OOPCoreClr_false
1824210	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1824210	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1824054	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1824054	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1823934	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1823934	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_false OOPCoreClr_false
1823934	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1823934	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1823359	roslyn-integration-CI	PR 61877	1-Integration Release OOP64_false OOPCoreClr_false
1823359	roslyn-integration-CI	PR 61877	1-Integration Release OOP64_false OOPCoreClr_false
1822879	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1822879	roslyn-integration-CI	Rolling	1-Integration Release OOP64_false OOPCoreClr_false
1822818	roslyn-integration-CI	PR 61870	1-Integration Debug OOP64_true OOPCoreClr_false
1822818	roslyn-integration-CI	PR 61870	1-Integration Debug OOP64_true OOPCoreClr_false
1822769	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1822769	roslyn-integration-CI	Rolling	1-Integration Debug OOP64_true OOPCoreClr_false
1822751	roslyn-integration-CI	PR 61871	1-Integration Release OOP64_false OOPCoreClr_false
1822751	roslyn-integration-CI	PR 61871	1-Integration Release OOP64_false OOPCoreClr_false
1822671	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false
1822671	roslyn-integration-CI	Rolling	1-Integration Release OOP64_true OOPCoreClr_false

Displaying 100 of 238 results

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	15	98

[sharwell]

@Cosifne this issue is bucketing errors incorrectly. The first three items currently in the list failed for:

1. A crash during a completely unrelated test

   Event Time (UTC): 6/22/2022 8:39:56 PM
   Event ID: 1026
   Data: Application: devenv.exe
   Framework Version: v4.0.30319
   Description: The process was terminated due to an unhandled exception.
   Exception Info: System.InvalidOperationException
      at System.Collections.ObjectModel.ObservableCollection`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].CheckReentrancy()
      at System.Collections.ObjectModel.ObservableCollection`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].RemoveItem(Int32)
      at Microsoft.VisualStudio.PlatformUI.NavigateToSession+NavigateToSearchOperation.RemoveOldResults(System.Collections.Generic.IList`1<Microsoft.VisualStudio.PlatformUI.NavigateToItemResult>, System.String ByRef)
      at Microsoft.VisualStudio.PlatformUI.NavigateToSession+NavigateToSearchOperation.<WaitForCompletion>b__30_0()
      at Microsoft.VisualStudio.PlatformUI.NavigateToWindow+SearchTask.SynchronizeResultList(System.Action)
      at Microsoft.VisualStudio.PlatformUI.NavigateToSession+NavigateToSearchOperation.WaitForCompletion()
      at Microsoft.VisualStudio.PlatformUI.NavigateToWindow+SearchTask.WaitForSearchCompletion()
      at Microsoft.VisualStudio.PlatformUI.NavigateToWindow+SearchTask.OnStartSearch()
      at Microsoft.VisualStudio.Platform.WindowManagement.WindowSearchHost+<>c__DisplayClass70_0.<StartSearch>b__0(System.Object)
      at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
      at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
      at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
      at System.Threading.ThreadPoolWorkQueue.Dispatch()
   

2. An exception in ENC

   Xunit.Sdk.TrueException: Actual and expected values differ. Expected shown in baseline of diff:
   +(Compiler) <unknown>(1, 1): warning AD0001: Analyzer 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueDiagnosticAnalyzer' threw an exception of type 'System.TypeLoadException' with message 'Method 'GetHotReloadUpdatesAsync' in type 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueLanguageService' from assembly 'Microsoft.CodeAnalysis.EditorFeatures, Version=42.42.42.42, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.'.
   +(Compiler) <unknown>(1, 1): warning AD0001: Analyzer 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueDiagnosticAnalyzer' threw an exception of type 'System.TypeLoadException' with message 'Method 'GetHotReloadUpdatesAsync' in type 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueLanguageService' from assembly 'Microsoft.CodeAnalysis.EditorFeatures, Version=42.42.42.42, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.'.
   +(Compiler) <unknown>(1, 1): warning AD0001: Analyzer 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueDiagnosticAnalyzer' threw an exception of type 'System.TypeLoadException' with message 'Method 'GetHotReloadUpdatesAsync' in type 'Microsoft.CodeAnalysis.EditAndContinue.EditAndContinueLanguageService' from assembly 'Microsoft.CodeAnalysis.EditorFeatures, Version=42.42.42.42, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.'.
   
   Expected: True
   Actual:   False
      at Xunit.Assert.True(Nullable`1 condition, String userMessage) in C:\Dev\xunit\xunit\src\xunit.assert\Asserts\BooleanAsserts.cs:line 95
   

3. A failure to save the source file

[sharwell]

Reopening to treat this as a story