System.Threading.Tasks.Dataflow.Tests failing with NRE in S.T.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks

[radical]

Failures per Runfo - last 30 days - status as of 8/24 (unrelated failures omitted):

Day	Run	Details
8/13	PR 1942101	Alpine.314.Arm32.Open
8/13	PR 1941527	Alpine.314.Arm32.Open
8/11	PR 1936915	Alpine.314.Arm32.Open
8/10	PR 1935321	Alpine.314.Arm32.Open
8/6	PR 1927814	Alpine.314.Arm32.Open
8/4	Rolling run AM 1922656 - extra-platforms	Console log - Debian.10.Arm32.Open
8/4	PR 1924659	Console log - Alpine.314.Arm32.Open
8/2	PR 1919458	Alpine.314.Arm32.Open
8/1	PR 1916028	Alpine.314.Arm32.Open
7/29	Rolling run 1912276	Alpine.314.Arm32.Open
7/28	PR 1909906	Alpine.314.Arm32.Open

Potentially related: #73247 also begins failing on arm32 on 7/28. This issue first failed at 12:18pm, and #73247 starts ~5 hours later at 5:25pm. Both tests continue to fail ~once a day from this point onwards.

Hit on an unrelated PR - #73423 .
Build(Libraries Test Run checked coreclr Linux_musl arm Release), and log:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Threading.Tasks.Dataflow.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Threading.Tasks.Dataflow.Tests (found 328 of 336 test cases)
  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ConcurrentExclusiveSchedulerPair.cs:line 461
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 938
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Threading.Tasks.Dataflow.Tests.runtimeconfig.json --depsfile System.Threading.Tasks.Dataflow.Tests.deps.json xunit.console.dll System.Threading.Tasks.Dataflow.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Aug 5 03:50:48 UTC 2022 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

I could not find an existing issue for this.

Build	Kind	Start Time
1912276	Rolling	2022-29-07

[ghost]

Tagging subscribers to this area: @dotnet/area-system-threading-tasks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hit on an unrelated PR - #73423 .
Build(Libraries Test Run checked coreclr Linux_musl arm Release), and log:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Threading.Tasks.Dataflow.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Threading.Tasks.Dataflow.Tests (found 328 of 336 test cases)
  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ConcurrentExclusiveSchedulerPair.cs:line 461
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 938
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Threading.Tasks.Dataflow.Tests.runtimeconfig.json --depsfile System.Threading.Tasks.Dataflow.Tests.deps.json xunit.console.dll System.Threading.Tasks.Dataflow.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Aug 5 03:50:48 UTC 2022 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

I could not find an existing issue for this.

Author:	radical
Assignees:	-
Labels:	area-System.Threading.Tasks, blocking-clean-ci
Milestone:	-

[joperezr]

I'm taking a look at this by trying to analyze the dump so I can get which test is causing this.

[karelz]

@joperezr FYI: I updated top post -- it didn't seem to happen 8/13-8/24 at all.

[jkotas]

Likely fixed as part of #73405 (comment)