CI hitting vm assert waitResult == WAIT_TIMEOUT in the tiered compilation manager

[AndyAyersMS]

https://dev.azure.com/dnceng/public/_build/results?buildId=1633314&view=ms.vss-test-web.build-test-results-tab (arm)

(ignore the main log and look at the console logs)

Assert failure(PID 547 [0x00000223], Thread: 580 [0x0244]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet

[hoyosjs]

cc: @kouvel

[AndyAyersMS]

Another example here: https://dev.azure.com/dnceng/public/_build/results?buildId=1670670&view=ms.vss-test-web.build-test-results-tab (Alpine arm)

[kouvel]

I suspect that this is happening during shutdown. The code is calling WaitForSingleObjectEx on an AutoResetEvent in a non-alertable fasion with a timeout. On Windows, I don't see any way for it to return anything other than WAIT_OBJECT_0 or WAIT_TIMEOUT. The failures appear to be happening only on Unixes.

@janvorli, is there any way that the PAL could return something else? I see some special-casing for TWS_EARLYDEATH in synchmanager.hpp/cpp in shutdown scenarios and perhaps there is a way it could return WAIT_FAILED in some corner cases. Or is there some other possible return value that I may have missed?

[kouvel]

Looks like a core dump is not available

[jkotas]

Failed again in #67814:

Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-67814-merge-13cfab2643634d36a7/System.Diagnostics.Process.Tests/1/console.28eadef9.log?sv=2019-07-07&se=2022-04-30T01%3A08%3A28Z&sr=c&sp=rl&sig=PEnfFrDD5jQ%2FBfOij6LHnhovcKo4X4JLOXM2QBwscqc%3D

...
None of the following programs were installed on this machine: xdg-open,gnome-open,kfmclient.

Assert failure(PID 724 [0x000002d4], Thread: 736 [0x02e0]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet


Assert failure(PID 1106 [0x00000452], Thread: 1118 [0x045e]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet

    System.Diagnostics.Tests.ProcessTests.TestProcessRecycledPid [SKIP]
      Condition(s) not met: "IsStressModeEnabledAndRemoteExecutorSupported"

Assert failure(PID 1226 [0x000004ca], Thread: 1253 [0x04e5]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet


Assert failure(PID 1212 [0x000004bc], Thread: 1254 [0x04e6]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet


Assert failure(PID 1893 [0x00000765], Thread: 1950 [0x079e]): waitResult == WAIT_TIMEOUT
    File: /__w/1/s/src/coreclr/vm/tieredcompilation.cpp Line: 507
    Image: /root/helix/work/correlation/dotnet

  Finished:    System.Diagnostics.Process.Tests