mono minijit runtime tests failing with coreclr_initialize failed - Error: 0x80004005
#72114
Labels
area-Infrastructure
blocking-clean-ci
Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
test-failure
Milestone
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
|
marking future until we see it again |
|
@lewing it happens more often - see updated top post with Kusto query. Overall about 2x per week - marking it as 'blocking-clean-ci'. Also, it is causing massive failures on the runs, which makes it hard to analyze results -- see for example Rolling build from 7/29 - 1912276 has 200 failures. |
karelz
added
the
blocking-clean-ci
karelz
changed the title
minijit runtime tests failing with coreclr_initialize failed - Error: 0x80004005 on OSX/x64minijit runtime tests failing with coreclr_initialize failed - Error: 0x80004005
|
I tried to run |
|
According to https://support.microsoft.com/en-us/topic/you-may-receive-error-code-0x80004005-or-other-error-codes-when-you-try-to-start-a-windows-xp-based-computer-a15f5b2f-642d-24ac-4912-1570a6bcedec. @lambdageek Do you have any thoughts on this? |
|
runtime/src/mono/mono/mini/monovm.c Lines 207 to 208 in 5eb776d |
|
cc @agocke |
I don't have more info than above + details from Kusto. You might be able to get to the run from JobID and other records in Kusto. Unfortunately, I don't know how to do it. |
|
@karelz thanks for checking! We have enough information to move forward now. |
|
Tagging subscribers to this area: @directhex Issue DetailsFailures 4/5-8/15 (incl. PRs):
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains testNameSubstring
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where (methodName == '') or (Method == methodName)
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
//| extend DefinitionName = PropertiesJson.DefinitionName
| project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);Hit on #72094 . cc @lewing
|
|
According to @lambdageek
@LakshanF Are you the best person to take a look at this issue? |
So... looking at runtime/src/coreclr/hosts/corerun/corerun.cpp Line 358 in a389c12 So... it loaded something. and that something had Maybe this load is hitting a problem: runtime/src/coreclr/hosts/corerun/corerun.cpp Line 277 in a389c12 I wonder if it's possible that the mono |
|
Seems outside Mono scope at this time. Not certain what the best area label is though. Also moving to 8.0.0 given the failure is not deterministic and seems like a test/infrastructure issue, not a real customer issue. cc @LakshanF - is this something you can help investigate? Please update area label appropriately. |
|
Another massive hit on: @SamMonoRT who is taking action on this? Can you please find someone on your team or another team? This is blocking CI ... |
|
Hit in #75858 |
|
I believe it is as @lambdageek said:
When generating CORE_ROOT, both the coreclr and mono versions of Relevant logs from the runtime pipeline build: |
|
The coreclr version is being included runtime/src/tests/Common/Directory.Build.targets Lines 101 to 111 in 3435b39 Perhaps we just need to condition those under |
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue DetailsFailures 4/5-9/8 (incl. PRs and 7.0 branches):
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains testNameSubstring
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where (methodName == '') or (Method == methodName)
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
//| extend DefinitionName = PropertiesJson.DefinitionName
| project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);Hit on #72094 . cc @lewing
|
ghost
added
the
in-pr
ghost
removed
the
in-pr
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Failures 4/5-9/8 (incl. PRs and 7.0 branches):
Hit on #72094 .
Build. From one of the logs:
cc @lewing
The text was updated successfully, but these errors were encountered: