slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1158893	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50998
1158875	dotnet/runtime	osx-arm64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1158764	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120129
1158761	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120128
1158760	dotnet/performance	Performance sdk_scenarios windows 24H2 x64 Open 8.0	Log	dotnet/performance#4982
1158755	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51016
1158739	dotnet/sdk	Ubuntu_22_04 Build_Release	Log	dotnet/sdk#50916
1158664	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119432
1158535	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1158529	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1158499	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2801785	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT osx x64 perfiphone12mini net10.0	Log
1158435	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1158475	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#120029
1158093	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#119859
2798593	dotnet-runtime	SuperPMI collect aspnet2 run osx arm64 checked	Log
1158405	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120111
1158387	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120110
1158371	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120107
1158346	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1150572	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119803
1158209	dotnet/performance	Performance mlnet windows 24H2 x64 Open 8.0	Log	dotnet/performance#4952
1158201	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#120083
1158143	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1158142	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1158096	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1158133	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#106880
1158097	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1158102	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#119575
1157284	dotnet/runtime	osx-arm64 Release NativeAOT	Log
1157919	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1157908	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1157534	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1157488	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
2801380	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2801229	dotnet-sdk	TestBuild: macOS (x64)	Log	#53607
2801064	dotnet-sdk	AoT: macOS (x64)	Log	#53613
1157371	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118941
1157003	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1157010	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120057
1156685	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120032
1156731	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#119894
2801063	dotnet-sdk	AoT: macOS (x64)	Log	#53608
1156725	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1157142	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1157016	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1156691	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120033
2800741	dotnet-performance	Performance roslyn ubuntu 2204 x64 Tiger 9.0	Log
1156921	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1156915	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1156886	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50975
2800634	dotnet-sdk	Darwin	Log	#53551
2800632	dotnet-sdk	Darwin	Log	#53552
2800633	dotnet-sdk	Darwin	Log	#53553
1156543	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
2800436	dotnet-sdk	Darwin	Log	#53551
1156724	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2800504	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1156529	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120026
1156697	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120034
2800438	dotnet-sdk	Darwin	Log	#53553
2800437	dotnet-sdk	Darwin	Log	#53552
2800413	dotnet-install-scripts	MacOSTests	Log
1156526	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120025
2800307	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2800310	dotnet-sdk	TestBuild: macOS (x64)	Log	#53607
1156466	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120024
1156372	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120021
1156363	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2800239	dotnet-sdk	TestBuild: macOS (x64)	Log	#53651
2800233	dotnet-sdk	Darwin	Log	#53553
2800234	dotnet-sdk	Darwin	Log	#53552
2800232	dotnet-sdk	Darwin	Log	#53551
1156258	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1156208	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1156156	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#120011
1156217	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119894
2800142	dotnet-runtime	osx-arm64 release CrossAOT_Mono crossaot	Log
1155184	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#119967
1156169	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1156097	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1155811	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50967
1155594	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1155673	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119978
1155704	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#119980
1155707	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119981
1152061	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#113385
1155935	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
2799801	dotnet-sdk	Darwin	Log	#53553
2799797	dotnet-sdk	Darwin	Log	#53551
2799799	dotnet-sdk	Darwin	Log	#53552
2799786	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2799706	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1155667	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
2799458	dotnet-runtime	osx-arm64 Release NativeAOT_Libraries	Log	#53614
1155593	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1155581	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#119974
1155641	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#119977
1155615	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119894
1155611	dotnet/performance	Performance powershell ubuntu 2204 x64 Open 9.0	Log
Displaying 100 of 519 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
32	177	519

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.