[Crossgen2] Stack overflow in Crossgen2 compilation of the test src/tests/Loader/classloader/generics/regressions/DD117522/Test.csproj

[trylek]

OS: all
Architecture: all
Example run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=352091&view=results
Diagnostic info:

ILCompiler.TypeSystem.dll!Internal.TypeSystem.TypeSystemContext.InstantiatedTypeKey.InstantiatedTypeKeyHashtable.CreateValueFromKey(Internal.TypeSystem.TypeSystemContext.InstantiatedTypeKey key) Line 336	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.LockFreeReaderHashtable.CreateValueAndEnsureValueIsInTable(Internal.TypeSystem.TypeSystemContext.InstantiatedTypeKey key) Line 562	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.LockFreeReaderHashtable.GetOrCreateValue(Internal.TypeSystem.TypeSystemContext.InstantiatedTypeKey key) Line 581	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.TypeSystemContext.GetInstantiatedType(Internal.TypeSystem.MetadataType typeDef, Internal.TypeSystem.Instantiation instantiation) Line 345	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.InstantiatedType.InstantiateSignature(Internal.TypeSystem.Instantiation typeInstantiation, Internal.TypeSystem.Instantiation methodInstantiation) Line 254	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.InstantiatedType.InitializeBaseType() Line 60	C#
 	ILCompiler.TypeSystem.dll!Internal.TypeSystem.InstantiatedType.BaseType.get() Line 68	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 309	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 311	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 236	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 337	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 311	C#
 	[The 8 frame(s) above this were repeated 486 times]	
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 311	C#
>	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 236	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 337	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(Internal.TypeSystem.TypeDesc type) Line 311	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(Internal.TypeSystem.TypeDesc type) Line 74	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.CompilerTypeSystemContext.EnsureLoadableMethod(Internal.TypeSystem.MethodDesc method) Line 194	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.ReadyToRunLibraryRootProvider.CheckCanGenerateMethod(Internal.TypeSystem.MethodDesc method) Line 88	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.ReadyToRunLibraryRootProvider.RootMethods(Internal.TypeSystem.MetadataType type, string reason, ILCompiler.IRootingServiceProvider rootProvider) Line 67	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.ReadyToRunLibraryRootProvider.AddCompilationRoots(ILCompiler.IRootingServiceProvider rootProvider) Line 39	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.Compilation.Compilation(ILCompiler.DependencyAnalysisFramework.DependencyAnalyzerBase dependencyGraph, ILCompiler.DependencyAnalysis.NodeFactory nodeFactory, System.Collections.Generic.IEnumerable compilationRoots, Internal.IL.ILProvider ilProvider, ILCompiler.DevirtualizationManager devirtualizationManager, System.Collections.Generic.IEnumerable modulesBeingInstrumented, ILCompiler.Logger logger, ILCompiler.InstructionSetSupport instructionSetSupport) Line 66	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.ReadyToRunCodegenCompilation.ReadyToRunCodegenCompilation(ILCompiler.DependencyAnalysisFramework.DependencyAnalyzerBase dependencyGraph, ILCompiler.DependencyAnalysis.NodeFactory nodeFactory, System.Collections.Generic.IEnumerable roots, Internal.IL.ILProvider ilProvider, ILCompiler.Logger logger, ILCompiler.DevirtualizationManager devirtualizationManager, System.Collections.Generic.IEnumerable inputFiles, string compositeRootPath, ILCompiler.InstructionSetSupport instructionSetSupport, bool resilient, bool generateMapFile, bool generateMapCsvFile, bool generatePdbFile, System.Func printReproInstructions, string pdbPath, bool generatePerfMapFile, string perfMapPath, int perfMapFormatVersion, bool generateProfileFile, int parallelism, ILCompiler.ProfileDataManager profileData, ILCompiler.ReadyToRunMethodLayoutAlgorithm methodLayoutAlgorithm, ILCompiler.ReadyToRunFileLayoutAlgorithm fileLayoutAlgorithm, int customPESectionAlignment, bool verifyTypeAndFieldLayout) Line 339	C#
 	ILCompiler.ReadyToRun.dll!ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() Line 321	C#

The failure appeared this Wednesday and I suspect it's related to the change

#89415

(Fix failure to generate ngen pdb in crossgen) that made changes to EnsureLoadableType around the places I'm seeing in the debugger. I'm looking into fixing this but at this point I'm not yet completely clear on how is the managed type system supposed to deal with infinite recursion in type definition (the offending test exercises the runtime exception when trying to load a type with recursive definition).

/cc @dotnet/crossgen-contrib, @davidwrighton, @MichalStrehovsky, @tommcdon

[trylek]

I have taken a first look at this issue. I can confirm that reverting the abovementioned PR fixes the issue. With David's change I see that I can fix the problem locally when I comment out the line

runtime/src/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.Validation.cs

Line 337 in 6b6bdf0

((CompilerTypeSystemContext)type.Context).EnsureLoadableType(typeArg);

I'm however not sure whether it's the right thing to do as David highlights addition of this check in the PR description as one of its fundamental components. It seems to me that David went to great lengths to implement a complex recursion guard that should protect against various legal "semi-circular" type definitions but it's not really hardened against the illegal case where someone intentionally stresses the system by constructing a type that is really circular and non-loadable.