Tests crashing in CI with no dump: exit code 137 means SIGKILL Killed

[jkotas]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=527171
Build error leg or test failing: System.Text.Json.Tests
Pull request: #96894

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "exit code 137 means SIGKILL Killed",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=527171
Error message validated: exit code 137 means SIGKILL Killed
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/25/2024 7:09:47 PM UTC

Report

Build	Definition	Test	Pull Request
567012	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#98463
566833	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution
566781	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#98450
566756	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#98451
566582	dotnet/runtime	System.IO.Tests.WorkItemExecution
565830	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565824	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565832	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565816	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565833	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565817	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565820	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565836	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565839	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565826	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565841	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565825	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565821	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565822	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
565516	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#97135
565131	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#97878
565109	dotnet/runtime	System.IO.Tests.WorkItemExecution	#98385
564673	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#97881
564516	dotnet/runtime	System.IO.Tests.WorkItemExecution	#98337
564229	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564232	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564237	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564223	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564230	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564225	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564214	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564216	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564210	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564234	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564235	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564236	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564217	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
564215	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562164	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98261
557869	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98134
562928	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98299
562011	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#96370
562893	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98303
562843	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98117
562846	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98281
562835	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98302
562830	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98301
562799	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98298
562788	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98199
562784	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98299
562774	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97877
562752	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97537
562684	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97644
562536	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98277
562528	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98296
562540	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97508
562456	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98226
562378	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562382	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562384	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562379	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562375	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562424	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98294
562428	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#95001
562381	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562376	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562372	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562412	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98293
562367	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562374	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562368	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562380	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562371	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562395	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98087
562365	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
562392	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98291
562353	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98289
562319	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#96650
562305	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97079
562250	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98152
562222	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97976
562110	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98284
562076	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98283
562028	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98281
562017	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98263
562020	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98273
562005	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98278
561996	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97537
561892	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97508
561889	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#98277
561839	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561838	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561834	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561851	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561855	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561836	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561837	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561854	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561848	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
561847	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
Displaying 100 of 1066 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
19	249	1066

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=527171
Build error leg or test failing: System.IO.Tests.File_GetSetTimes_SafeFileHandle.WritingShouldUpdateWriteTime_After_SetLastAccessTime
Pull request: #96894

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "exit code 137 means SIGKILL Killed eg by kill",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	jkotas
Assignees:	-
Labels:	area-Infrastructure-libraries, blocking-clean-ci, Known Build Error
Milestone:	-

[jkotas]

  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 7275 of 7334 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 6)
./RunTests.sh: line 180:  5589 Killed: 9               "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B0B70943/w/A9E50974/e
----- end Mon Jan 15 01:12:38 PST 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill

We need dumps to make this diagnosable.

[agocke]

SIGKILL is a pretty unusual way to take down the process... do we know if there's anything in the infra which can produce a SIGKILL?

[jkotas]

Exit code 137 can be caused by OOM.

[agocke]

It doesn't look like we have a mechanism to grab dumps if it is OOM, though: #52521