[NativeAOT] possible deadlock due to InitializeRuntime reentrancy if ModuleInitializer launches threads.

[VSadov]

While playing with alternatives for stress testing, I tried spawning a thread from a module initializer, that would do GC.Collect in a loop and other stress(y) things.

Some tests seem to hang with that:

• InitializeRuntime calls module initializers, which is arbitrary app code and may launch a thread.
• launching a thread waits for the thread to start running.
• to start running, a thread needs to do a reverse PInvoke, which spin-waits for InitializeRuntime to complete.

Perhaps we should finish runtime initialization before calling into module initializers?

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[VSadov]

Typical stacks:

Main thread:

 	ntdll.dll!NtWaitForSingleObject()	Unknown
 	KernelBase.dll!WaitForSingleObjectEx()	Unknown
 	CustomMainWithStubExe.dll!S_P_CoreLib_System_Threading_Thread__JoinInternal() Line 166	Unknown
 	CustomMainWithStubExe.dll!S_P_CoreLib_System_Threading_Thread__StartCore() Line 383	Unknown
 	CustomMainWithStubExe.dll!S_P_CoreLib__Module____cctor()	Unknown
 	CustomMainWithStubExe.dll!S_P_CoreLib_Internal_Runtime_CompilerHelpers_StartupCodeHelpers__RunInitializers() Line 177	Unknown
>	CustomMainWithStubExe.dll!S_P_CoreLib_Internal_Runtime_CompilerHelpers_StartupCodeHelpers__RunModuleInitializers() Line 165	Unknown
 	CustomMainWithStubExe.dll!Internal_CompilerGenerated__Module___NativeLibraryStartup()	Unknown
 	CustomMainWithStubExe.dll!InitializeRuntime() Line 210	C++
 	[Inline Frame] CustomMainWithStubExe.dll!Thread::EnsureRuntimeInitialized() Line 1225	C++
 	[Inline Frame] CustomMainWithStubExe.dll!Thread::ReversePInvokeAttachOrTrapThread(ReversePInvokeFrame *) Line 1187	C++
 	CustomMainWithStubExe.dll!RhpReversePInvokeAttachOrTrapThread2(ReversePInvokeFrame * pFrame) Line 1306	C++
 	CustomMainWithStubExe.dll!RhpReversePInvoke(ReversePInvokeFrame * pFrame) Line 1322	C++
 	CustomMainWithStubExe.dll!CustomMainWithStubExe_Program__IncrementExitCode() Line 30	Unknown
 	CustomMainWithStubExe.exe!wmain(int argc, wchar_t * * argv) Line 52	C++
 	CustomMainWithStubExe.exe!invoke_main() Line 91	C++
 	CustomMainWithStubExe.exe!__scrt_common_main_seh() Line 288	C++
 	CustomMainWithStubExe.exe!__scrt_common_main() Line 331	C++
 	CustomMainWithStubExe.exe!wmainCRTStartup(void * __formal) Line 17	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Extra thread:

 	ntdll.dll!NtDelayExecution()	Unknown
 	KernelBase.dll!SleepEx()	Unknown                       <=== Sleep(1) in a loop
>	[Inline Frame] CustomMainWithStubExe.dll!Thread::EnsureRuntimeInitialized() Line 1220	C++
 	[Inline Frame] CustomMainWithStubExe.dll!Thread::ReversePInvokeAttachOrTrapThread(ReversePInvokeFrame *) Line 1187	C++
 	CustomMainWithStubExe.dll!RhpReversePInvokeAttachOrTrapThread2(ReversePInvokeFrame * pFrame) Line 1306	C++
 	CustomMainWithStubExe.dll!RhpReversePInvoke(ReversePInvokeFrame * pFrame) Line 1322	C++
 	CustomMainWithStubExe.dll!S_P_CoreLib_System_Threading_Thread__ThreadEntryPoint() Line 225	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

[VSadov]

A simplest repro is to drop this inside System.GC and run tests.
(as in " build clr.alljits+clr.tools+clr.nativeaotlibs+clr.nativeaotruntime+libs -rc Checked -lc Release & src\tests\build.cmd nativeaot Checked tree nativeaot & src\tests\run.cmd runnativeaottests Checked ")

        internal sealed class Stress
        {
#pragma warning disable CA2255
            [ModuleInitializer]
#pragma warning restore CA2255
            internal static void Start()
            {
                Threading.Thread thread = null!;
                thread = new Threading.Thread(
                    () =>
                    {
                        System.Diagnostics.Stopwatch stopWatch = new System.Diagnostics.Stopwatch();
                        for (; ; )
                        {
                            stopWatch.Restart();
                            System.GC.Collect(0);
                            stopWatch.Stop();

                            Internal.Console.Write(".");

                            // proportional to GC pause
                            long pauseUsec = stopWatch.ElapsedTicks / 16;

                            stopWatch.Restart();
                            int i = 1;
                            do
                            {
                                i *= 2;
                                System.Threading.Thread.SpinWait(i);
                            }
                            while (stopWatch.ElapsedTicks < pauseUsec);

                            object[] o = new object[Random.Shared.Next(128) + 1];
                            o[0] = thread;
                            o = new object[Random.Shared.Next(128) + 1];
                            o[0] = o;
                            o = new object[Random.Shared.Next(128) + 1];
                            o[0] = o;
                        }
                    });

                thread.IsBackground = true;
                thread.Start();
            }
        }

[VSadov]

Tests that hang are:
CustomMainWithStubExe.exe
nativeaot.exe

Libraries tests seem to be running fine. Not sure what is special about the affected tests or whether the problem is user-reachable vs. some test-specific peculiarity.

Actually, perhaps once the tests above hang, no other tests runs at all, thus I see only these two hanged (i.e. all tests may be affected).

[MichalStrehovsky]

Not sure what is special about the affected tests

I think they all involve shared libraries. The stack above is specific to shared libraries due to:

runtime/src/coreclr/nativeaot/Bootstrap/main.cpp

Lines 205 to 208 in 1231a9a

	#ifdef NATIVEAOT_DLL
	// Run startup method immediately for a native library
	__managed__Startup();
	#endif // NATIVEAOT_DLL