slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1550332	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132216
1550330	dotnet/aspnetcore	Tests: macOS	Log
1550329	dotnet/aspnetcore	Tests: macOS	Log
1550309	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131538
1550248	dotnet/maui	Controls Shell	Log
1550200	dotnet/maui	Controls Shell	Log	dotnet/maui#37034
1550100	dotnet/maui	Controls CollectionView	Log
1550160	dotnet/aspnetcore	Tests: macOS	Log
1550121	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129734
1550029	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log	dotnet/maui#37310
1550103	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1550066	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR		dotnet/runtime#130910
1549985	dotnet/aspnetcore	Tests: macOS	Log
1549857	dotnet/aspnetcore	Tests: macOS	Log
1549839	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129506
1549835	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132163
1549828	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132204
1549827	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1549825	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1549822	dotnet/runtime	osx-x64 Release Libraries_CheckedCoreCLR	Log
1549754	dotnet/maui	Controls Shell	Log	dotnet/maui#37302
1549765	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132201
1549701	dotnet/aspnetcore	Tests: macOS	Log
1549674	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log	dotnet/maui#37143
1549654	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR	Log
1549541	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132101
1549280	dotnet/maui	Controls Shell	Log	dotnet/maui#37257
1549325	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128675
1549257	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132176
1549175	dotnet/runtime	osx-arm64 Release NativeAOT_Libs	Log
1549173	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1549123	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131654
1549045	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1549043	dotnet/aspnetcore	Tests: macOS	Log
1548649	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132139
1548518	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132115
1548422	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37278
1548416	dotnet/maui	Controls Layout	Log	dotnet/maui#37307
1548419	dotnet/maui	Controls Shell	Log	dotnet/maui#37034
1548236	dotnet/maui	Controls CollectionView	Log
1548380	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131771
1548280	dotnet/maui	Controls Shell	Log	dotnet/maui#37147
1548352	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130023
1548348	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132124
1548310	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130688
1548289	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131746
1548256	dotnet/maui	Controls Shell	Log	dotnet/maui#36212
1548230	dotnet/maui	Controls Shell	Log
1548233	dotnet/maui	Controls Layout	Log
1548269	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1548179	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131952
1548154	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132079
1548144	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132126
1548049	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132080
1547969	dotnet/runtime	osx-arm64 Debug PALTests	Log
1547916	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132117
1547893	dotnet/aspnetcore	Tests: macOS	Log
1547811	dotnet/maui	Controls Layout	Log	dotnet/maui#37302
1547822	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131976
1547794	dotnet/maui	Controls Layout	Log	dotnet/maui#36212
1547726	dotnet/runtime	iossimulator-arm64 Release AllSubsets_NativeAOT	Log
1547654	dotnet/maui	Controls CollectionView	Log
1547353	dotnet/maui	Controls Shell	Log
1547522	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132101
1547486	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36212
1547469	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131821
1547464	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130368
1547431	dotnet/maui	Controls Shell	Log	dotnet/maui#37143
1547365	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37232
1547345	dotnet/maui	Controls Layout	Log
1547312	dotnet/aspnetcore	Tests: macOS	Log
1547208	dotnet/runtime	osx-x64 Debug PALTests	Log
1547122	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36212
1547084	dotnet/maui	Controls Shell	Log
1547068	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132023
1547055	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132082
1546722	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
1546713	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
1547040	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132081
1546886	dotnet/maui	Controls Shell	Log	dotnet/maui#37278
1546926	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132079
1546477	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1546872	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1546808	dotnet/runtime	osx-x64 Debug PALTests	Log
1546783	dotnet/aspnetcore	Tests: macOS	Log
1546719	dotnet/maui	Controls CollectionView	Log
1546606	dotnet/maui	Controls Layout	Log	dotnet/maui#36541
1546612	dotnet/maui	Controls Layout	Log	dotnet/maui#37252
1546544	dotnet/aspnetcore	Tests: macOS	Log
1546479	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1546474	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1546385	dotnet/maui	Controls Layout	Log
1546341	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37111
1546315	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132013
1546261	dotnet/aspnetcore	Tests: macOS	Log
1546101	dotnet/maui	Controls Layout	Log
1545956	dotnet/maui	Controls Shell	Log	dotnet/maui#36212
1545868	dotnet/aspnetcore	Tests: macOS	Log
1545728	dotnet/aspnetcore	Tests: macOS	Log
1545673	dotnet/aspnetcore	Tests: macOS	Log
Displaying 100 of 1179 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
34	159	1179

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.