restarted. Azure DevOps can't recover from restarts.

[lewing]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
834756	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#105427
834730	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108700
834641	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108697
834592	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108696
834511	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#104562
834506	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107282
833102	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108687
834191	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108693
834140	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108684
833560	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108684
833550	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108655
833473	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108681
833474	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108681
833101	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#108687
2555730	dotnet-dotnet	AzureLinux_x64_Cross_ShortStack_Mono_LLVMAOT_arm64
833033	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108086
833028	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108086
832843	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108684
832526	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108655
2555528	dotnet-dotnet	AzureLinux_x64_Cross_Alpine_arm
832493	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108673
832436	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108670
832377	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108478
832361	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108665
832299	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108489
832245	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108634
832222	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108663
832215	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108535
832189	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108414
832138	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108560
832019	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#102249
831994	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107282
831963	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108572
831959	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108661
831915	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108578
831832	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#105267
831810	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108561
831809	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108561
831752	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108636
831704	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108188
831703	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#108188
831507	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#102249
831508	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#102249
831473	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107179
831455	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107282
2555267	dotnet-dotnet	AzureLinux_x64_Cross_Pgo_x64
831378	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108655
831372	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#108582
831374	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108582
831309	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107179
831255	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108537
831254	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108537
831226	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108582
831214	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#105427
831078	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108647
830894	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108492
830817	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108645
830779	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108635
830456	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108639
830428	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108636
830406	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108634
830394	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108633
830278	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108514
830252	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108620
830248	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108620
830183	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108403
830163	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108324
830153	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108535
830146	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108629
830134	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108089
830096	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108414
830090	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108032
830045	dotnet/roslyn	Test_Linux_Debug
830059	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108602
829985	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108247
829908	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108620
829888	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108614
822762	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	Log	dotnet/runtime#108411
829829	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108618
829824	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108576
829820	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108602
829782	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108403
829750	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108535
829719	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#104488
829705	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108616
829702	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108537
829698	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108032
829527	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108043
828704	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107231
828665	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#107230
828661	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108568
828658	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108568
828644	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#102144
828578	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108576
828566	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108542
828563	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108568
827220	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108370
828552	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108542
828547	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108576
828544	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108568
Displaying 100 of 730 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
41	246	730

[am11]

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

[am11]

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

[steveisok]

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?