Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R2R jobs are failing with Fatal error. Internal CLR error. (0x80131506) #43198

Closed
sandreenko opened this issue Oct 9, 2020 · 3 comments · Fixed by #43261
Closed

R2R jobs are failing with Fatal error. Internal CLR error. (0x80131506) #43198

sandreenko opened this issue Oct 9, 2020 · 3 comments · Fixed by #43261
Labels
area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs test-run-core Test failures in .NET Core test runs
Milestone

Comments

@sandreenko
Copy link
Contributor

All recent outer loop runs are red because of all R2R jobs are failing.
The tests are failing with:

C:\h\w\BEC20A64\w\AE7909A8\e>dotnet C:\h\w\BEC20A64\p\xunit.console.dll baseservices\callconvs\baseservices.callconvs.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=baseservices.callconvs 
Fatal error. Internal CLR error. (0x80131506)
   at Xunit.XunitProject..ctor()
   at Xunit.ConsoleClient.CommandLine.GetProjectFile(System.Collections.Generic.List`1<System.Tuple`2<System.String,System.String>>)
   at Xunit.ConsoleClient.CommandLine.Parse(System.Predicate`1<System.String>)
   at Xunit.ConsoleClient.CommandLine..ctor(System.String[], System.Predicate`1<System.String>)
   at Xunit.ConsoleClient.CommandLine.Parse(System.String[])
   at Xunit.ConsoleClient.ConsoleRunner.EntryPoint(System.String[])
   at Xunit.ConsoleClient.Program.Main(System.String[])

#13590 could be related

I think it started with https://dev.azure.com/dnceng/public/_build/results?buildId=838111&view=ms.vss-test-web.build-test-results-tab but there are 250 changes in this chunk so can't find suspicion changes.

PTAL @hoyosjs @dotnet/runtime-infrastructure , cc @dotnet/jit-contrib

@sandreenko sandreenko added test-run-core Test failures in .NET Core test runs area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Oct 9, 2020
@sandreenko sandreenko added this to the 6.0.0 milestone Oct 9, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Oct 9, 2020
@hoyosjs
Copy link
Member

hoyosjs commented Oct 9, 2020

#13590 seems unrelated; the user claims it only reproduces with a debugger attached. Taking a look to see if anything stands out.

@hoyosjs
Copy link
Member

hoyosjs commented Oct 9, 2020

The diff for good -> bad is:

git log f98038a...746ca52 --pretty=oneline --ancestry-path
746ca52 [mono] Fix HelloWorld sample for OSX (#42916)
334bd84 Jit preparation for arm64 apple: Use bytes in fgArgTabEntry. (#42503)
eb91a96 [metadata] Use MONO_PROFILER_API on MonoClass getters in checked builds (#42819)
5e5e923 Fix error in code comment (#42931)
ee993b4 Update DataContractSerialization ILLink.Suppressions.xml (#42930)
1e15914 Fix nullable annotation on ISite.Container (#42918)
0be66cb Extract bundled files when IncludeAllContentForSelfExtract is set (#42435)
1c4910f JIT: show profile data for dot flowgraph dumps (#42657)
e82488d do not call RemoteCertificateValidationCallback on the same certificate again (#42836)
118eee9 DataContractSerialization doesn't work with TrimMode - link (#42824)
957fdb9 Test fixes for casing (#42879)
05c1c58 Suppress linker analysis warnings on GetImmutableDictionaryCreateRangeMethod (#42905)
29489d1 Fix xml doc comment for SimpleConsoleFormatterOptions.SingleLine (#42896)
8ffc0b3 Optimize Type.IsPrimitive (#42891)
13b457a Fix test framework crossgening (#42874)
994dca6 Add granular suppressions for linker warnings (#40691)

From the one you sent, looking at one of the dumps:

Stack of crash from baseservices.callconvs (AV)
 # ChildEBP RetAddr      
00 (Inline) --------     coreclr!MethodTable::GetInstantiation+0x13 [F:\workspace\_work\1\s\src\coreclr\src\vm\methodtable.cpp @ 5907] 
01 0057b530 72e4316d     coreclr!MethodTableBuilder::SetupMethodTable2+0x4cd [F:\workspace\_work\1\s\src\coreclr\src\vm\methodtablebuilder.cpp @ 10358] 
02 0057b800 72e406cd     coreclr!MethodTableBuilder::BuildMethodTableThrowing+0x226d [F:\workspace\_work\1\s\src\coreclr\src\vm\methodtablebuilder.cpp @ 1794] 
03 0057da50 72e3d8fe     coreclr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0xc7d [F:\workspace\_work\1\s\src\coreclr\src\vm\methodtablebuilder.cpp @ 12221] 
04 0057daa8 72e3bab8     coreclr!ClassLoader::CreateTypeHandleForTypeKey+0xee [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3206] 
05 (Inline) --------     coreclr!ClassLoader::DoIncrementalLoad+0x45 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3125] 
06 0057dfd8 72e63643     coreclr!ClassLoader::LoadTypeHandleForTypeKey_Body+0x6a8 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3890] 
07 0057e034 72e6275a     coreclr!ClassLoader::LoadTypeHandleForTypeKey+0xc3 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3609] 
08 0057e06c 72e94e86     coreclr!ClassLoader::LoadConstructedTypeThrowing+0x19d [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 1155] 
09 0057e0bc 72e3ddef     coreclr!ClassLoader::LoadCanonicalGenericInstantiation+0xa1 [F:\workspace\_work\1\s\src\coreclr\src\vm\generics.cpp @ 149] 
0a 0057e198 72e3d90e     coreclr!ClassLoader::CreateTypeHandleForNonCanonicalGenericInstantiation+0xcf [F:\workspace\_work\1\s\src\coreclr\src\vm\generics.cpp @ 199] 
0b 0057e1e8 72e3bab8     coreclr!ClassLoader::CreateTypeHandleForTypeKey+0xfe [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3219] 
0c (Inline) --------     coreclr!ClassLoader::DoIncrementalLoad+0x45 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3125] 
0d 0057e718 72e63643     coreclr!ClassLoader::LoadTypeHandleForTypeKey_Body+0x6a8 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3890] 
0e 0057e774 72e6275a     coreclr!ClassLoader::LoadTypeHandleForTypeKey+0xc3 [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 3609] 
0f 0057e7ac 72e0ae61     coreclr!ClassLoader::LoadConstructedTypeThrowing+0x19d [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 1155] 
10 0057e7ec 72e66492     coreclr!ClassLoader::LoadGenericInstantiationThrowing+0xab [F:\workspace\_work\1\s\src\coreclr\src\vm\clsload.cpp @ 1959] 
11 0057e894 72e1aa12     coreclr!SigPointer::GetTypeHandleThrowing+0xf32 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 1416] 
12 (Inline) --------     coreclr!ZapSig::DecodeType+0x77 [F:\workspace\_work\1\s\src\coreclr\src\vm\zapsig.cpp @ 757] 
13 0057ea60 72e7b515     coreclr!LoadDynamicInfoEntry+0x372 [F:\workspace\_work\1\s\src\coreclr\src\vm\jitinterface.cpp @ 13393] 
14 0057ea88 72ee88dc     coreclr!Module::FixupNativeEntry+0x5d [F:\workspace\_work\1\s\src\coreclr\src\vm\ceeload.cpp @ 10292] 
15 0057eab8 72e7aeb5     coreclr!Module::FixupDelayListAux<Module *,int (__thiscall Module::*)(CORCOMPILE_IMPORT_SECTION *,unsigned long,unsigned long *)>+0x51 [F:\workspace\_work\1\s\src\coreclr\src\vm\ceeload.inl @ 584] 
16 (Inline) --------     coreclr!Module::FixupDelayList+0x24 [F:\workspace\_work\1\s\src\coreclr\src\vm\ceeload.inl @ 487] 
17 0057eb94 72e7c728     coreclr!ReadyToRunInfo::GetEntryPoint+0x2b5 [F:\workspace\_work\1\s\src\coreclr\src\vm\readytoruninfo.cpp @ 896] 
18 (Inline) --------     coreclr!MethodDesc::GetPrecompiledR2RCode+0x3c [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 581] 
19 (Inline) --------     coreclr!MethodDesc::GetPrecompiledCode+0x46 [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 435] 
1a (Inline) --------     coreclr!MethodDesc::PrepareILBasedCode+0x67 [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 392] 
1b (Inline) --------     coreclr!MethodDesc::PrepareCode+0x67 [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 332] 
1c 0057ed98 72e7c29f     coreclr!CodeVersionManager::PublishVersionableCodeIfNecessary+0x188 [F:\workspace\_work\1\s\src\coreclr\src\vm\codeversion.cpp @ 1701] 
1d 0057ee2c 72e7c030     coreclr!MethodDesc::DoPrestub+0x13f [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 2127] 
1e 0057eeec 72edb0dd     coreclr!PreStubWorker+0x1b0 [F:\workspace\_work\1\s\src\coreclr\src\vm\prestub.cpp @ 1952] 
1f 0057ef10 07b196de     coreclr!ThePreStub+0x11 [F:\workspace\_work\1\s\src\coreclr\src\vm\i386\asmhelpers.asm @ 1544] 
20 0057ef50 07b18334     xunit_console!Xunit.ConsoleClient.CommandLine.GetProjectFile(System.Collections.Generic.List`1<System.Tuple`2<System.String,System.String>>)+0x2e
21 0057efc4 07b166b4     xunit_console!Xunit.ConsoleClient.CommandLine.Parse(System.Predicate`1<System.String>)+0x1b4
22 0057efe0 07b165f0     xunit_console!Xunit.ConsoleClient.CommandLine..ctor(System.String[], System.Predicate`1<System.String>)+0xac
23 0057eff4 07b15faa     xunit_console!Xunit.ConsoleClient.CommandLine.Parse(System.String[])+0x28
24 0057f0bc 07a65be8     xunit_console!Xunit.ConsoleClient.ConsoleRunner.EntryPoint(System.String[])+0x52
25 0057f100 72edadcf     xunit_console!Xunit.ConsoleClient.Program.Main(System.String[])+0xc0
26 0057f10c 72e61dd2     coreclr!CallDescrWorkerInternal+0x34 [F:\workspace\_work\1\s\src\coreclr\src\vm\i386\asmhelpers.asm @ 612] 
27 0057f138 72e6199e     coreclr!CallDescrWorkerWithHandler+0x65 [F:\workspace\_work\1\s\src\coreclr\src\vm\callhelpers.cpp @ 70] 
28 0057f1b0 72eaa486     coreclr!MethodDescCallSite::CallTargetWorker+0x1ac [F:\workspace\_work\1\s\src\coreclr\src\vm\callhelpers.cpp @ 552] 
29 (Inline) --------     coreclr!MethodDescCallSite::Call_RetArgSlot+0x15 [F:\workspace\_work\1\s\src\coreclr\src\vm\callhelpers.h @ 458] 
2a 0057f2c4 72eaa344     coreclr!RunMainInternal+0x88 [F:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1468] 
2b 0057f334 72ea90e2     coreclr!RunMain+0x9f [F:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1538] 
2c 0057f598 72ea8c30     coreclr!Assembly::ExecuteMainMethod+0x17e [F:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1648] 
2d 0057f664 72ea8aa6     coreclr!CorHost2::ExecuteAssembly+0x140 [F:\workspace\_work\1\s\src\coreclr\src\vm\corhost.cpp @ 384] 
2e 0057f6b8 7320d480     coreclr!coreclr_execute_assembly+0xc6 [F:\workspace\_work\1\s\src\coreclr\src\dlls\mscoree\unixinterface.cpp @ 431] 
2f (Inline) --------     hostpolicy!coreclr_t::execute_assembly+0x2b [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\coreclr.cpp @ 89] 
30 0057f75c 7320d613     hostpolicy!run_app_for_context+0x299 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 246] 
31 0057f790 7320da7d     hostpolicy!run_app+0x33 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 275] 
32 0057f8a0 7326e296     hostpolicy!corehost_main+0xed [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 398] 
33 0057f920 7326f883     hostfxr!execute_app+0x19a [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 146] 
34 0057f978 73270916     hostfxr!`anonymous namespace'::read_config_and_execute+0x71 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 521] 
35 0057f9d4 7326f9ab     hostfxr!fx_muxer_t::handle_exec_host_command+0xc7 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 1001] 
36 0057fa70 7326d1c7     hostfxr!fx_muxer_t::execute+0xb5 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 580] 
37 0057fb0c 01018a80     hostfxr!hostfxr_main_startupinfo+0x67 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\hostfxr.cpp @ 50] 
38 0057fc50 01018c90     dotnet!exe_start+0x3f0 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 236] 
39 0057fc60 0101a20c     dotnet!wmain+0x66 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 302] 
3a (Inline) --------     dotnet!invoke_main+0x1c [D:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
3b 0057fca8 778b62c4     dotnet!__scrt_common_main_seh+0xfa [D:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
3c 0057fcbc 77a01e39     kernel32!BaseThreadInitThunk+0x24 [d:\rs1\base\win32\client\thread.c @ 64] 
3d 0057fd04 77a01e04     ntdll!__RtlUserThreadStart+0x2f [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997] 
3e 0057fd14 00000000     ntdll!_RtlUserThreadStart+0x1b [d:\rs1\minkernel\ntdll\rtlstrt.c @ 914] 

Instantiation MethodTable::GetInstantiation()
{
    LIMITED_METHOD_CONTRACT;
    SUPPORTS_DAC;
    if (HasInstantiation())
    {
        PTR_GenericsDictInfo  pDictInfo = GetGenericsDictInfo();
->        TADDR base = dac_cast<TADDR>(&(GetPerInstInfo()[pDictInfo->m_wNumDicts-1]));
        return Instantiation(PerInstInfoElem_t::GetValueMaybeNullAtPtr(base)->GetInstantiation(), pDictInfo->m_wNumTyPars);
    }
    else
    {
        return Instantiation();
    }
}

Method Table & generic:

0:000> dx -r1 pMT
pMT                 : 0x7beb258 [Type: MethodTable *]
    [=0x731a3c20] s_pMethodDataCache : 0x73b540 [Type: MethodDataCache *]
    [=0x731a3c1c] s_fUseParentMethodData : 1 [Type: int]
    [=0x731a3c18] s_fUseMethodDataCache : 1 [Type: int]
    [+0x000] m_dwFlags        : 0x10 [Type: unsigned long]
    [+0x004] m_BaseSize       : 0x20 [Type: unsigned long]
    [+0x008] m_wFlags2        : 0xe [Type: unsigned short]
    [+0x00a] m_wToken         : 0x0 [Type: unsigned short]
    [+0x00c] m_wNumVirtuals   : 0x9 [Type: unsigned short]
    [+0x00e] m_wNumInterfaces : 0x2 [Type: unsigned short]
    [+0x010] m_pParentMethodTable [Type: IndirectPointer<MethodTable *>]
    [+0x014] m_pLoaderModule  [Type: PlainPointer<Module *>]
    [+0x018] m_pWriteableData [Type: PlainPointer<MethodTableWriteableData *>]
    UNION_MASK       : 0x3 [Type: unsigned long]
    [+0x01c] m_pEEClass       [Type: PlainPointer<EEClass *>]
    [+0x01c] m_pCanonMT       [Type: PlainPointer<unsigned long>]
    [+0x020] m_pPerInstInfo   [Type: PlainPointer<PlainPointer<Dictionary *> *>]
    [+0x020] m_ElementTypeHnd : 0x0 [Type: unsigned long]
    [+0x020] m_pMultipurposeSlot1 : 0x0 [Type: unsigned long]
    [+0x024] m_pInterfaceMap  [Type: PlainPointer<InterfaceInfo_t *>]
    [+0x024] m_pMultipurposeSlot2 : 0x7beb290 [Type: unsigned long]
    [=0x7311c478] c_DispatchMapSlotOffsets [Type: unsigned char [0]]
    [=0x7311c470] c_NonVirtualSlotsOffsets [Type: unsigned char [0]]
    [=0x7311c460] c_ModuleOverrideOffsets [Type: unsigned char [0]]
    [=0x73129648] c_OptionalMembersStartOffsets [Type: unsigned char [0]]
0:000> dx pMT->m_pPerInstInfo
pMT->m_pPerInstInfo                 [Type: PlainPointer<PlainPointer<Dictionary *> *>]
    [+0x000] m_ptr            : 0x0 [Type: unsigned long]

So GetPerInstInfo stated returning null. But in that set of changes I don't see anything that glaringly changes m_pPerInstInfo directly so it's not quite as obvious. The only other change I see that could've changed R2R behavior is 13b457a

@jeffschwMSFT jeffschwMSFT removed the untriaged New issue has not been triaged by the area owner label Oct 9, 2020
@trylek
Copy link
Member

trylek commented Oct 9, 2020

I think this was definitely in some way affected by JanV's change to switch over framework compilation to use the r2rtest tool, the only trouble is that so far I've been unable to nail down the exact problem. I haven't yet managed to repro this locally even though it's most likely that some subtle bug in framework compilation is to blame. I have a PR out where I fixed the remaining inconsistencies w.r.t. the r2rtest framework compilation change,

#43183

and a bit of a good news is that with the change all Crossgen2 legs seem to fail in a relatively consistent manner - crashing upon launch of the XUnit wrappers. I continue investigating this, an interesting tell-tale sign in some runs is a failure to "validate the layout of System.Object", I'm currently following up on this one in the hope that it will shed more light on the inconsistency.

@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs test-run-core Test failures in .NET Core test runs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants