profiler\\multiple\\multiple\\multiple.cmd failing on windows arm64

[AndyAyersMS]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=574998
Build error leg or test failing: profiler\multiple\multiple\multiple.cmd
Pull request: #98777

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Unhandled exception. System.Exception: Profiler tests are expected to contain the text 'PROFILER TEST PASSES'",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=574998
Error message validated: [Unhandled exception. System.Exception: Profiler tests are expected to contain the text 'PROFILER TEST PASSES']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/28/2024 1:41:53 AM UTC

Report

Build	Definition	Test	Pull Request
655183	dotnet/runtime	profiler.multiple.WorkItemExecution	#101291

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	1	1

[AlitzelMendez]

Known issues update: This issue has been affected by the issue described in: dotnet/dnceng#2065. Since all the issues opened after 02/20 can cause the build analysis to fail, I will exclude this issue from the build analysis and add a tracking label to restore it as a known issue once this problem is resolved.

[ghost]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=574998
Build error leg or test failing: profiler\multiple\multiple\multiple.cmd
Pull request: #98777

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Unhandled exception. System.Exception: Profiler tests are expected to contain the text 'PROFILER TEST PASSES'",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	AndyAyersMS
Assignees:	-
Labels:	area-Diagnostics-coreclr, blocking-clean-ci, untriaged
Milestone:	-

[janvorli]

I will give it a quick look.

[tommcdon]

@davmason is this fixed by #98685?

[jkotas]

It does not appear to be fixed. It was just hit by #98741 on windows arm64 checked:

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-98741-merge-ff52306935e5461bb1/profiler.multiple/1/console.3ed85ab5.log?helixlogtype=result

        Requesting detach!!
        RequestProfilerDetach hr=0x0
        Requesting detach!!
        Requesting detach!!
        ProfilerDetachSucceeded _detachCount=1
        RequestProfilerDetach hr=0x0
        RequestProfilerDetach hr=0x0
        ProfilerDetachSucceeded _detachCount=2
        
        Test failed: Profiler tests are expected to contain the text 'PROFILER TEST PASSES' in the console output of the profilee app to indicate a passing test. Usually it is printed from the Shutdown() method of the profiler implementation. This text was not found in the output above.
        Expected: 100
        Actual: -532462766
        END EXECUTION - FAILED

[AndyAyersMS]

Also failing in jitstress: #98787

[davmason]

This is a new failure mode, the test waits on an event for 5 minutes and is timing out waiting for all profilers to detach. I made a PR to increase the timeout here #99014

For the jitstress failures, it is marked as JitOptimizationSensitive, shouldn't it be blocked from running under jitstress?

[AndyAyersMS]

I don't know if zapdisable gets auto-detected as jit stress. Technically it's not true jit stress, it just happens to be running as part of the jit stress job.

set DOTNET_TieredCompilation=0
set DOTNET_ReadyToRun=0
set DOTNET_ZapDisable=1

[davmason]

Hopefully fixed with the increased timeout in #99014. If this continues to fail I can disable on arm64

[jkotas]

5 minutes is a lot of time. I doubt that increasing it to 10 minutes is going to help.

The most likely explanation of the symptoms is that the test is hitting intermittent hang (deadlock, etc.). Is it possible to modify the test to produce dumps when it is timing out so that we can see what's going on?

[davmason]

Opened #99075 to throw an exception when we time out rather than just log

[jkotas]

Reactivating. This is still failing on number of PRs.

[kunalspathak]

Failed again in https://dev.azure.com/dnceng-public/public/_build/results?buildId=655183&view=ms.vss-test-web.build-test-results-tab&runId=16217960&resultId=100465&paneView=debug

[jkotas]

The error message tracked by this issue can have number of different causes. The root cause of the failure tracked by this issue was fixed. This issue should not be reactivated. A new issue should be opened instead.

[jkotas]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=655183&view=ms.vss-test-web.build-test-results-tab&runId=16217960&resultId=100465&paneView=debug

This is #100114