Test failure: baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.cmd

[v-wenyuxu]

Failed in: runtime-coreclr outerloop 20240825.2

Failed tests:

coreclr linux arm Checked no_tiered_compilation @ (Alpine.316.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7
    - baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.cmd

Error message:

 
Assert failure(PID 212 [0x000000d4], Thread: 212 [0x00d4]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
    File: /__w/1/s/src/coreclr/vm/crst.cpp:765
    Image: /root/helix/work/correlation/crossgen2/crossgen2

[createdump] The app model does not support crash report generation
[createdump] Failure took 0ms
waitpid() returned successfully (wstatus 00000086) WEXITSTATUS 0 WTERMSIG 6
/root/helix/work/workitem/e/baseservices/baseservices/../TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh: line 211:   212 Aborted                 (core dumped) "$CORE_ROOT/crossgen2/crossgen2" --out r2r/BasicTestWithMcj.dll BasicTestWithMcj.dll -r $CORE_ROOT/*.dll
BEGIN: coreclr_initialize failed - Error: 0x80070057
END: coreclr_initialize failed - Error: 0x80070057

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/TieredCompilation/BasicTestWithMcj/output.txt
Raw output:
BEGIN EXECUTION
[createdump] Target process is alive
./RunBasicTestWithMcj.sh --runCustomTest /root/helix/work/workitem/e/baseservices/TieredCompilation/BasicTestWithMcj/ BasicTestWithMcj.dll ''
Collect profile without R2R, use profile without R2R
Collect profile with R2R, use profile with R2R
Exe path: /root/helix/work/correlation/corerun
Properties:
    TRUSTED_PLATFORM_ASSEMBLIES = /root/helix/work/correlation/System.ServiceModel.Web.dll:/root/helix/work/correlation/System.Composition.Hosting.dll:/root/helix/work/correlation/System.Net.dll:/root/helix/work/correlation/System.ServiceProcess.ServiceController.dll:/root/helix/work/correlation/Microsoft.CodeAnalysis.dll:/root/helix/work/correlation/System.Net.WebSockets.dll:/root/helix/work/correlation/System.Threading.Tasks.dll:/root/helix/work/correlation/System.Net.Primitives.dll:/root/helix/work/correlation/System.DirectoryServices.dll:/root/helix/work/correlation/Microsoft.Diagnostics.Tracing.TraceEvent.dll:/root/helix/work/correlation/Microsoft.Extensions.Options.ConfigurationExtensions.dll:/root/helix/work/correlation/xunit.execution.dotnet.dll:/root/helix/work/correlation/System.Net.Sockets.dll:/root/helix/work/correlation/Microsoft.Extensions.Hosting.Systemd.dll:/root/helix/work/correlation/System.Runtime.Serialization.Json.dll:/root/helix/work/correlation/System.Threading.Channels.dll:/root/helix/work/correlation/System.Diagnostics.DiagnosticSource.dll:/root/helix/work/correlation/Microsoft.Extensions.Configuration.Xml.dll:/root/helix/work/correlation/xunit.runner.utility.netcoreapp10.dll:/root/helix/work/correlation/System.ComponentModel.DataAnnotations.dll:/root/helix/work/correlation/System.Security.Cryptography.Cng.dll:/root/helix/work/correlation/System.Security.AccessControl.dll:/root/helix/work/correlation/System.Formats.Cbor.dll:/root/helix/work/correlation/System.Buffers.dll:/root/helix/work/correlation/System.Runtime.CompilerServices.Unsafe.dll:/root/helix/work/correlation/System.IO.Pipes.dll:/root/helix/work/correlation/System.ServiceProcess.dll:/root/helix/work/correlation/System.Runtime.Extensions.dll:/root/helix/work/correlation/System.Threading.Thread.dll:/root/helix/work/correlation/System.Dynamic.Runtime.dll:/root/helix/work/correlation/WindowsBase.dll:/root/helix/work/correlation/Microsoft.Extensions.Logging.EventSource.dll:/root/helix/work/correlation/Microsoft.Extensions.FileProviders.Physical.dll:/root/helix/work/correlation/System.ComponentModel.Composition.Registration.dll:/root/helix/work/correlation/System.IO.FileSystem.Primitives.dll:/root/helix/work/correlation/System.Collections.dll:/root/helix/work/correlation/System.Security.Principal.Windows.dll:/root/helix/work/correlation/xunit.console.dll:/root/helix/work/correlation/Microsoft.Extensions.FileProviders.Composite.dll:/root/helix/work/correlation/System.Runtime.Serialization.Primitives.dll:/root/helix/work/correlation/System.Configuration.ConfigurationManager.dll:/root/helix/

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor23|0_24(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[jkotas]

Introduced by #80154?

cc @am11 @jakobbotsch

[am11]

@jkotas #106965 (comment), this is failing in main as well:

https://dev.azure.com/dnceng-public/public/_build?definitionId=134&_a=summary

so this was unrelated to #80154 change.

[am11]

Can't say for sure when it started to happen because old logs are deleted on helix side, but by the number of failures on those jobs, it seems like it's between 1^st and 2^nd of august: e3684bc...30b34e6, i.e. e3684bc had fewer failures and next run on the next day against 30b34e6 had many.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[jkotas]

This was introduced by #105671

We may need to add a new lock dependency to https://github.com/dotnet/runtime/blob/main/src/coreclr/inc/CrstTypes.def

cc @AaronRobinsonMSFT