HttpClient getting deadlocked on InvalidateHttp2Connection

[Randomly939384]

Description

Occasionally in production environment I'm facing an issue when HttpClient gets deadlocked on InvalidateHttp2Connection.

More about environment
Machine Ubuntu 22.04.2 LTS.
Dotnet runtime version 8.0.0.
HttpClient using http 2 version.
The application is doing hundreds of rest requests per minute to the same domain using dotnet HttpClient. Additionally in the same process it uses C++ DLL which has background threads also performing rest requests to the same endpoint using http 1.1. C++ part may do thousands requests per minute.
The app is running 24/7.
Occasionally once a week it will stop doing HttpClient requests. After examining the dump, i can see it gets deadlocked.

In the dump can see that Thread1 entered Http2Connection.SyncObject in ProcessGoAwayFrame and waiting on HttpConnectionPool.SyncObject in InvalidateHttp2Connection. Thread2 entered HttpConnectionPool.SyncObject in CleanCacheAndDisposeIfUnused and waiting on Http2Connection.SyncObject in GetIdleTicks

Both threads using the same Http2Connection and HttpConnectionPool objects:
0x00007f73e958c7c0 System.Net.Http.Http2Connection
0x00007f740049bb28 System.Net.Http.HttpConnectionPool

Raw output of dotnet dump analysis:

Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
13305 00007F6D6C5F4FB8            7         1 00007F6B2898D390 2b673a 224   00007f740049bc38 System.Collections.Generic.List`1[[System.Net.Http.HttpConnection, System.Net.Http]]
13306 00007F6D6C5F4F60            3         1 00007F6D685C18F0 2c8a6f 230   00007f73e9590630 System.Collections.Generic.Dictionary`2[[System.Int32, System.Private.CoreLib],[System.Net.Http.Http2Connection+Http2Stream, System.Net.Http]]
-----------------------------
Total           20914
Free            10844

setthread 224                                                                                                                                                  
clrstack -a                                                                                                                                                    
OS Thread Id: 0x2b673a (224)
        Child SP               IP Call Site
00007F6D7E7FB570 00007fef82c91117 [HelperMethodFrame_1OBJ: 00007f6d7e7fb570] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F6D7E7FB6C0 00007FEF0CCCCBD3 System.Net.Http.Http2Connection.GetIdleTicks(Int64) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 1799]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f73e958c7c0
        nowTicks (<CLR reg>) = 0x000000057e929216
    LOCALS:
        0x00007F6D7E7FB6D0 = 0x00007f73e9590630
        <no data>
        <no data>

00007F6D7E7FB700 00007FEF0CCCC741 System.Net.Http.HttpConnectionPool.<CleanCacheAndDisposeIfUnused>g__IsUsableConnection|123_2(System.Net.Http.HttpConnectionBase, Int64, System.TimeSpan, System.TimeSpan) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2356]
    PARAMETERS:
        connection (<CLR reg>) = 0x00007f73e958c7c0
        nowTicks (<CLR reg>) = 0x000000057e929216
        pooledConnectionLifetime = <no data>
        pooledConnectionIdleTimeout = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>

00007F6D7E7FB770 00007FEF0CCCBF78 System.Net.Http.HttpConnectionPool.<CleanCacheAndDisposeIfUnused>g__ScavengeConnectionList|123_1[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.List`1<System.__Canon>, System.Collections.Generic.List`1<System.Net.Http.HttpConnectionBase> ByRef, Int64, System.TimeSpan, System.TimeSpan) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2306]
    PARAMETERS:
        list (<CLR reg>) = 0x00007f73e9590e00
        toDispose (<CLR reg>) = 0x00007f6d7e7fb7e8
        nowTicks (<CLR reg>) = 0x000000057e929216
        pooledConnectionLifetime = <no data>
        pooledConnectionIdleTimeout = <no data>
    LOCALS:
        0x00007F6D7E7FB784 = 0x0000000000000000
        <no data>
        <no data>

00007F6D7E7FB7C0 00007FEF0CCCC57D System.Net.Http.HttpConnectionPool.CleanCacheAndDisposeIfUnused() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2283]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f740049bb28
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        0x00007F6D7E7FB7D0 = 0x00007f740049bc38
        <no data>
        0x00007F6D7E7FB7DC = 0x0000000000000074
        <no data>

00007F6D7E7FB820 00007FEF0CCCC327 System.Net.Http.HttpConnectionPoolManager.RemoveStalePools() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs @ 464]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f734939f148
    LOCALS:
        <CLR reg> = 0x00007f756a1358b8
        <no data>
        <no data>
        <no data>
        <no data>

00007F6D7E7FB900 00007FEF0CCCC1AB System.Net.Http.HttpConnectionPoolManager+<>c.<.ctor>b__11_0(System.Object) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs @ 106]
    PARAMETERS:
        this = <no data>
        s = <no data>
    LOCALS:
        <no data>
        <no data>

00007F6D7E7FB920 00007FEF09B807E5 System.Threading.TimerQueueTimer.Fire(Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs @ 683]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f73493a0870
        isThreadPool = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6D7E7FB980 00007FEF09B7E7C0 System.Threading.TimerQueue.FireNextTimers() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs @ 331]
    PARAMETERS:
        this = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6D7E7FB9F0 00007FEF09B761C0 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 913]
    LOCALS:
        <CLR reg> = 0x00007f71590ba688
        <CLR reg> = 0x00007f74626c3ad8
        <no data>
        <CLR reg> = 0x00007f74626c3c90
        <CLR reg> = 0x00007f74b53cb0e8
        <CLR reg> = 0x000000007e929216
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6D7E7FBA60 00007FEF0D2D6029 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 102]
    LOCALS:
        <CLR reg> = 0x00007f71590bc8f0
        <CLR reg> = 0x00007f71590bcb60
        <CLR reg> = 0x00007f71590bcc08
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6D7E7FBC50 00007fef8289fa07 [DebuggerU2MCatchHandlerFrame: 00007f6d7e7fbc50]

setthread 230                                                                                                                                                  
clrstack -a                                                                                                                                                    
OS Thread Id: 0x2c8a6f (230)
        Child SP               IP Call Site
00007F6B83FFDFF0 00007fef82c91117 [HelperMethodFrame_1OBJ: 00007f6b83ffdff0] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F6B83FFE140 00007FEF0CD9C284 System.Net.Http.HttpConnectionPool.InvalidateHttp2Connection(System.Net.Http.Http2Connection) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 1866]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f740049bb28
        connection (<CLR reg>) = 0x00007f73e958c7c0
    LOCALS:
        <CLR reg> = 0x0000000000000000
        0x00007F6B83FFE150 = 0x00007f740049bc38
        <no data>
        <no data>

00007F6B83FFE190 00007FEF0CD9C070 System.Net.Http.Http2Connection.Shutdown() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 263]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f73e958c7c0
    LOCALS:
        <no data>

00007F6B83FFE1E0 00007FEF0CDA1FE5 System.Net.Http.Http2Connection.ProcessGoAwayFrame(FrameHeader) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 1072]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f73e958c7c0
        frameHeader = <no data>
    LOCALS:
        <CLR reg> = 0x0000000000000001
        <no data>
        <CLR reg> = 0x00007f74d1ba1150
        <CLR reg> = 0x00007f74d1ba11e0
        0x00007F6B83FFE1F0 = 0x00007f73e9590630
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFE2C0 00007FEF0C5352EE System.Net.Http.Http2Connection+<ProcessIncomingFramesAsync>d__62.MoveNext() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 575]
    PARAMETERS:
        this (0x00007F6B83FFE2D8) = 0x00007f73e9590d18
    LOCALS:
        0x00007F6B83FFE634 = 0x00000000ffffffff
        0x00007F6B83FFE2D0 = 0x00007f73e958c7c0
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFE670 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>
    LOCALS:
        0x00007F6B83FFE690 = 0x00007f7569d5d728
        <no data>
        0x00007F6B83FFE688 = 0x0000000000000000
        0x00007F6B83FFE680 = 0x0000000000000000
        0x00007F6B83FFE678 = 0x0000000000000000
        <no data>
        <no data>

00007F6B83FFE6C0 00007FEF0C53B75D System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[System.Net.Http.Http2Connection+<ProcessIncomingFramesAsync>d__62, System.Net.Http]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 368]
    PARAMETERS:
        this (<CLR reg>) = 0x00007f73e9590cd8
        threadPoolThread = <no data>
    LOCALS:
        <CLR reg> = 0x0000000000000000
        <no data>

00007F6B83FFE700 00007FEF0C51F675 System.Net.Security.SslStream+<ReadAsyncInternal>d__163`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext() [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs @ 925]
    PARAMETERS:
        this = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFE7E0 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>
    LOCALS:
        0x00007F6B83FFE800 = 0x00007f7569d5d728
        <no data>
        0x00007F6B83FFE7F8 = 0x0000000000000000
        0x00007F6B83FFE7F0 = 0x0000000000000000
        0x00007F6B83FFE7E8 = 0x0000000000000000
        <no data>
        <no data>

00007F6B83FFE830 00007FEF0C51DBD0 System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext() [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs @ 745]
    PARAMETERS:
        this = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFE8D0 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>
    LOCALS:
        0x00007F6B83FFE8F0 = 0x00007f7569d5d728
        <no data>
        0x00007F6B83FFE8E8 = 0x0000000000000000
        0x00007F6B83FFE8E0 = 0x0000000000000000
        0x00007F6B83FFE8D8 = 0x0000000000000000
        <no data>
        <no data>

00007F6B83FFE920 00007FEF0C529E84 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, System.Memory`1<Byte>, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 102]
    PARAMETERS:
        this = <no data>
        bytesTransferred = <no data>
        socketAddress = <no data>
        receivedFlags = <no data>
        socketError = <no data>

00007F6B83FFE940 00007FEF0BEC349E System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
    PARAMETERS:
        this (<CLR reg>) = 0x00007f715602bbd8
    LOCALS:
        <CLR reg> = 0x00007f715602d2c8
        <no data>
        <CLR reg> = 0x000000007e929216

00007F6B83FFE9F0 00007FEF09B761C0 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 913]
    LOCALS:
        <CLR reg> = 0x00007f71590ba688
        <CLR reg> = 0x00007f74a96b9e98
        <no data>
        <CLR reg> = 0x00007f74a96ba358
        <CLR reg> = 0x00007f7569d5d728
        <CLR reg> = 0x000000007e929216
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFEA60 00007FEF0D2D6029 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 102]
    LOCALS:
        <CLR reg> = 0x00007f71590bc8f0
        <CLR reg> = 0x00007f71590bcb60
        <CLR reg> = 0x00007f71590bcc08
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F6B83FFEC50 00007fef8289fa07 [DebuggerU2MCatchHandlerFrame: 00007f6b83ffec50 

Reproduction Steps

Unable to reproduce locally, happens in prod environment once a week

Expected behavior

Expect it will not deadlock

Actual behavior

It deadlocks

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Occasionally in production environment I'm facing an issue when HttpClient gets deadlocked on InvalidateHttp2Connection.

More about environment
Machine Ubuntu 22.04.2 LTS.
Dotnet runtime version 8.0.0.
HttpClient using http 2 version.
The application is doing hundreds of rest requests per minute to the same domain using dotnet HttpClient. Additionally in the same process it uses C++ DLL which has background threads also performing rest requests to the same endpoint using http 1.1. C++ part may do thousands requests per minute.
The app is running 24/7.
Occasionally once a week it will stop doing HttpClient requests. After examinng the dump, i can see it gets deadlocked.

Raw output of dotnet dump analysis:

> syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 13305 00007F6D6C5F4FB8 7 1 00007F6B2898D390 2b673a 224 00007f740049bc38 System.Collections.Generic.List1[[System.Net.Http.HttpConnection, System.Net.Http]]
13306 00007F6D6C5F4F60 3 1 00007F6D685C18F0 2c8a6f 230 00007f73e9590630 System.Collections.Generic.Dictionary`2[[System.Int32, System.Private.CoreLib],[System.Net.Http.Http2Connection+Http2Stream, System.Net.Http]]

Total 20914
Free 10844

setthread 224
clrstack -a
OS Thread Id: 0x2b673a (224)
Child SP IP Call Site
00007F6D7E7FB570 00007fef82c91117 [HelperMethodFrame_1OBJ: 00007f6d7e7fb570] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F6D7E7FB6C0 00007FEF0CCCCBD3 System.Net.Http.Http2Connection.GetIdleTicks(Int64) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 1799]
PARAMETERS:
this () = 0x00007f73e958c7c0
nowTicks () = 0x000000057e929216
LOCALS:
0x00007F6D7E7FB6D0 = 0x00007f73e9590630

00007F6D7E7FB700 00007FEF0CCCC741 System.Net.Http.HttpConnectionPool.g__IsUsableConnection|123_2(System.Net.Http.HttpConnectionBase, Int64, System.TimeSpan, System.TimeSpan) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2356]
PARAMETERS:
connection () = 0x00007f73e958c7c0
nowTicks () = 0x000000057e929216
pooledConnectionLifetime =
pooledConnectionIdleTimeout =
LOCALS:

00007F6D7E7FB770 00007FEF0CCCBF78 System.Net.Http.HttpConnectionPool.g__ScavengeConnectionList|123_1[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.List1<System.__Canon>, System.Collections.Generic.List1<System.Net.Http.HttpConnectionBase> ByRef, Int64, System.TimeSpan, System.TimeSpan) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2306]
PARAMETERS:
list () = 0x00007f73e9590e00
toDispose () = 0x00007f6d7e7fb7e8
nowTicks () = 0x000000057e929216
pooledConnectionLifetime =
pooledConnectionIdleTimeout =
LOCALS:
0x00007F6D7E7FB784 = 0x0000000000000000

00007F6D7E7FB7C0 00007FEF0CCCC57D System.Net.Http.HttpConnectionPool.CleanCacheAndDisposeIfUnused() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 2283]
PARAMETERS:
this () = 0x00007f740049bb28
LOCALS:




0x00007F6D7E7FB7D0 = 0x00007f740049bc38

0x00007F6D7E7FB7DC = 0x0000000000000074

00007F6D7E7FB820 00007FEF0CCCC327 System.Net.Http.HttpConnectionPoolManager.RemoveStalePools() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs @ 464]
PARAMETERS:
this () = 0x00007f734939f148
LOCALS:
= 0x00007f756a1358b8

00007F6D7E7FB900 00007FEF0CCCC1AB System.Net.Http.HttpConnectionPoolManager+<>c.<.ctor>b__11_0(System.Object) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs @ 106]
PARAMETERS:
this =
s =
LOCALS:

00007F6D7E7FB920 00007FEF09B807E5 System.Threading.TimerQueueTimer.Fire(Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs @ 683]
PARAMETERS:
this () = 0x00007f73493a0870
isThreadPool =
LOCALS:

00007F6D7E7FB980 00007FEF09B7E7C0 System.Threading.TimerQueue.FireNextTimers() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs @ 331]
PARAMETERS:
this =
LOCALS:

00007F6D7E7FB9F0 00007FEF09B761C0 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 913]
LOCALS:
= 0x00007f71590ba688
= 0x00007f74626c3ad8

= 0x00007f74626c3c90
= 0x00007f74b53cb0e8
= 0x000000007e929216

00007F6D7E7FBA60 00007FEF0D2D6029 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 102]
LOCALS:
= 0x00007f71590bc8f0
= 0x00007f71590bcb60
= 0x00007f71590bcc08

00007F6D7E7FBC50 00007fef8289fa07 [DebuggerU2MCatchHandlerFrame: 00007f6d7e7fbc50]

setthread 230
clrstack -a
OS Thread Id: 0x2c8a6f (230)
Child SP IP Call Site
00007F6B83FFDFF0 00007fef82c91117 [HelperMethodFrame_1OBJ: 00007f6b83ffdff0] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F6B83FFE140 00007FEF0CD9C284 System.Net.Http.HttpConnectionPool.InvalidateHttp2Connection(System.Net.Http.Http2Connection) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs @ 1866]
PARAMETERS:
this () = 0x00007f740049bb28
connection () = 0x00007f73e958c7c0
LOCALS:
= 0x0000000000000000
0x00007F6B83FFE150 = 0x00007f740049bc38

00007F6B83FFE190 00007FEF0CD9C070 System.Net.Http.Http2Connection.Shutdown() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 263]
PARAMETERS:
this () = 0x00007f73e958c7c0
LOCALS:

00007F6B83FFE1E0 00007FEF0CDA1FE5 System.Net.Http.Http2Connection.ProcessGoAwayFrame(FrameHeader) [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 1072]
PARAMETERS:
this () = 0x00007f73e958c7c0
frameHeader =
LOCALS:
= 0x0000000000000001

= 0x00007f74d1ba1150
= 0x00007f74d1ba11e0
0x00007F6B83FFE1F0 = 0x00007f73e9590630

00007F6B83FFE2C0 00007FEF0C5352EE System.Net.Http.Http2Connection+d__62.MoveNext() [/_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs @ 575]
PARAMETERS:
this (0x00007F6B83FFE2D8) = 0x00007f73e9590d18
LOCALS:
0x00007F6B83FFE634 = 0x00000000ffffffff
0x00007F6B83FFE2D0 = 0x00007f73e958c7c0

00007F6B83FFE670 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
PARAMETERS:
executionContext =
callback =
state =
LOCALS:
0x00007F6B83FFE690 = 0x00007f7569d5d728

0x00007F6B83FFE688 = 0x0000000000000000
0x00007F6B83FFE680 = 0x0000000000000000
0x00007F6B83FFE678 = 0x0000000000000000

00007F6B83FFE6C0 00007FEF0C53B75D System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[System.Net.Http.Http2Connection+d__62, System.Net.Http]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 368]
PARAMETERS:
this () = 0x00007f73e9590cd8
threadPoolThread =
LOCALS:
= 0x0000000000000000

00007F6B83FFE700 00007FEF0C51F675 System.Net.Security.SslStream+d__163`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext() [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs @ 925]
PARAMETERS:
this =
LOCALS:

00007F6B83FFE7E0 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
PARAMETERS:
executionContext =
callback =
state =
LOCALS:
0x00007F6B83FFE800 = 0x00007f7569d5d728

0x00007F6B83FFE7F8 = 0x0000000000000000
0x00007F6B83FFE7F0 = 0x0000000000000000
0x00007F6B83FFE7E8 = 0x0000000000000000

00007F6B83FFE830 00007FEF0C51DBD0 System.Net.Security.SslStream+d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext() [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs @ 745]
PARAMETERS:
this =
LOCALS:

00007F6B83FFE8D0 00007FEF09B80B0C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
PARAMETERS:
executionContext =
callback =
state =
LOCALS:
0x00007F6B83FFE8F0 = 0x00007f7569d5d728

0x00007F6B83FFE8E8 = 0x0000000000000000
0x00007F6B83FFE8E0 = 0x0000000000000000
0x00007F6B83FFE8D8 = 0x0000000000000000

00007F6B83FFE920 00007FEF0C529E84 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, System.Memory`1, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 102]
PARAMETERS:
this =
bytesTransferred =
socketAddress =
receivedFlags =
socketError =

00007F6B83FFE940 00007FEF0BEC349E System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
PARAMETERS:
this () = 0x00007f715602bbd8
LOCALS:
= 0x00007f715602d2c8

= 0x000000007e929216

00007F6B83FFE9F0 00007FEF09B761C0 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 913]
LOCALS:
= 0x00007f71590ba688
= 0x00007f74a96b9e98

= 0x00007f74a96ba358
= 0x00007f7569d5d728
= 0x000000007e929216

00007F6B83FFEA60 00007FEF0D2D6029 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 102]
LOCALS:
= 0x00007f71590bc8f0
= 0x00007f71590bcb60
= 0x00007f71590bcc08

00007F6B83FFEC50 00007fef8289fa07 [DebuggerU2MCatchHandlerFrame: 00007f6b83ffec50`

so Thread1 entered Http2Connection.SyncObject in ProcessGoAwayFrame and waiting on HttpConnectionPool.SyncObject in InvalidateHttp2Connection. Thread2 entered HttpConnectionPool.SyncObject in CleanCacheAndDisposeIfUnused and waiting on Http2Connection.SyncObject in GetIdleTicks

both threads using the same Http2Connection and HttpConnectionPool objects:
0x00007f73e958c7c0 System.Net.Http.Http2Connection
0x00007f740049bb28 System.Net.Http.HttpConnectionPool

Reproduction Steps

Unable to reproduce locally, happens in prod environment once a week

Expected behavior

Expect it will not deadlock

Actual behavior

It deadlocks

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	Randomly939384
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[MihaZupan]

Thank you for the report and for including the stack traces.

We have a lock ordering problem here where:

• the cleanup timer will take the pool lock and then try to obtain the lock on the connection
• the connection shutdown will take the connection lock and then attempt to obtain the lock on the pool

Likely introduced by #90094 (related: #95621 that we recently fixed and backported).

Triage: Moving to 9.0

[MihaZupan]

The fix for this one has been merged in #99889 and should be released in .NET 9.0 preview 4.
Backported to .NET 8.0.5 in #100086

@Randomly939384 you mentioned this happens about once a week, how many times has it occurred?
Would you be willing to validate that the issue is resolved using recent builds of 9.0, or on a custom patched build of 8.0?