-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mono minijit
runtime tests failing with coreclr_initialize failed - Error: 0x80004005
#72114
Comments
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
marking future until we see it again |
@lewing it happens more often - see updated top post with Kusto query. Overall about 2x per week - marking it as 'blocking-clean-ci'. Also, it is causing massive failures on the runs, which makes it hard to analyze results -- see for example Rolling build from 7/29 - 1912276 has 200 failures. |
minijit
runtime tests failing with coreclr_initialize failed - Error: 0x80004005
on OSX/x64minijit
runtime tests failing with coreclr_initialize failed - Error: 0x80004005
I tried to run |
According to https://support.microsoft.com/en-us/topic/you-may-receive-error-code-0x80004005-or-other-error-codes-when-you-try-to-start-a-windows-xp-based-computer-a15f5b2f-642d-24ac-4912-1570a6bcedec. @lambdageek Do you have any thoughts on this? |
runtime/src/mono/mono/mini/monovm.c Lines 207 to 208 in 5eb776d
|
cc @agocke |
I don't have more info than above + details from Kusto. You might be able to get to the run from JobID and other records in Kusto. Unfortunately, I don't know how to do it. |
@karelz thanks for checking! We have enough information to move forward now. |
Tagging subscribers to this area: @directhex Issue DetailsFailures 4/5-8/15 (incl. PRs):
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains testNameSubstring
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where (methodName == '') or (Method == methodName)
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
//| extend DefinitionName = PropertiesJson.DefinitionName
| project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true); Hit on #72094 .
cc @lewing
|
According to @lambdageek
@LakshanF Are you the best person to take a look at this issue? |
So... looking at runtime/src/coreclr/hosts/corerun/corerun.cpp Line 358 in a389c12
So... it loaded something. and that something had Maybe this load is hitting a problem: runtime/src/coreclr/hosts/corerun/corerun.cpp Line 277 in a389c12
I wonder if it's possible that the mono |
Seems outside Mono scope at this time. Not certain what the best area label is though. Also moving to 8.0.0 given the failure is not deterministic and seems like a test/infrastructure issue, not a real customer issue. cc @LakshanF - is this something you can help investigate? Please update area label appropriately. |
Another massive hit on: @SamMonoRT who is taking action on this? Can you please find someone on your team or another team? This is blocking CI ... |
Hit in #75858
|
I believe it is as @lambdageek said:
When generating CORE_ROOT, both the coreclr and mono versions of
Relevant logs from the runtime pipeline build: |
The coreclr version is being included runtime/src/tests/Common/Directory.Build.targets Lines 101 to 111 in 3435b39
Perhaps we just need to condition those under |
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue DetailsFailures 4/5-9/8 (incl. PRs and 7.0 branches):
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains testNameSubstring
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where (methodName == '') or (Method == methodName)
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
//| extend DefinitionName = PropertiesJson.DefinitionName
| project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true); Hit on #72094 .
cc @lewing
|
Failures 4/5-9/8 (incl. PRs and 7.0 branches):
Hit on #72094 .
Build. From one of the logs:
cc @lewing
The text was updated successfully, but these errors were encountered: