.NET SignalR client getting stuck sending messages

[navidmatin]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Since updating to v5.0.10 or newer of the .NET SignalR client (from v3.1.5) we have noticed that some of the clients get into a state that causes connections to lock up. This causes the client to go offline without closing the connection or triggering our client reconnect logic.
The initial observation was that the SendAsync (which is an extension method that calls SendCoreAsync) freezes and causes the connection to timeout and the client to get disconnected. After adding a cancellation token with a timeout of 10 seconds, we managed to prevent the client from getting locked up but we are still seeing an issue that the connection state is Connected but it cannot send messages to our SignalRHub.
Additionally, as part of the efforts to fix this, I added a call to StopAsync to stop the connection and restart it. However, StopAsync also freezes when the client is in this state. After further research, I realized that StopAsync does not respect the cancellation token so had to remove the call to this method.

Expected Behavior

• SignalR client should be able to send messages to the server if it is connected to the SignalR Hub.
• (in my opinion) the SignalR client should have a default timeout value (similar to HttpClient) to prevent lock-up in case of failure to send messages to the server.
• Cancellation token that gets passed into StopAsync should not get ignored.

Steps To Reproduce

Unfortunately, I have not been able to reproduce this locally, but I have observed this reliably in the production environment.

Exceptions (if any)

No response

.NET Version

Client .Net 4.6.1 | Server .NET 6.0

Anything else?

We are using 10 US-West and 10 US-East instances of Azure SignalR Service and we have observed this behavior on ServerSentEvent and Websocket transports.

[navidmatin]

For the StopAsync method not respecting the CancellationToken I was thinking to put up a PR and update StopAsyncCore(bool disposing) to accept a cancellation token and pass this cancellation token to https://github.com/dotnet/aspnetcore/blob/main/src/SignalR/clients/csharp/Client.Core/src/HubConnection.cs#L497 and https://github.com/dotnet/aspnetcore/blob/main/src/SignalR/clients/csharp/Client.Core/src/HubConnection.cs#L484
But I wasn't sure if ignoring the cancellation token was intentional or a bug.

[BrennanConroy]

But I wasn't sure if ignoring the cancellation token was intentional or a bug.

It is intentional, canceling Stop doesn't really make sense in the traditional sense of cancellation. It might be ok to flow it through to do an ungraceful disconnect, but that's not something we've really looked into. #5294 (comment)

Are you able to get logs and/or a dump from the hanging clients? It would be nice to root cause the issue instead of adding some timeout that doesn't even seem to work for your scenario:

we managed to prevent the client from getting locked up but we are still seeing an issue that the connection state is Connected but it cannot send messages to our SignalRHub.

Hi @navidmatin. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[navidmatin]

Yes we managed to get some logs today:

2022-04-14 08:55:46,761 [INFO ] Transport is stopping.
2022-04-14 08:55:51,847 [ERROR] Error while sending to '<ourHubEndpoint>&negotiateVersion=1&asrs_request_id=xrBzlWoGAAA%3D&id=ua-qEjQ0yLUY2ZC1ZefgRAb80931431'.
 Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
   at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Http.Connections.Client.Internal.AccessTokenHttpMessageHandler.<SendAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Http.Connections.Client.Internal.LoggingHttpMessageHandler.<SendAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Http.Connections.Client.Internal.SendUtils.<SendMessages>d__0.MoveNext() 

Note that this did not trigger the ConnectionClosed event and 10 minutes later when we tried to send a message to the server the HubConnection state was Connected
Also, reiterate that when we are in this state we cannot stop the connection from the client side either.

[navidmatin]

Short term fix we have downgraded the SignalR client to v3.1.24 and that has fixed this issue.

[BrennanConroy]

Are you able to provide a minimal repro app as a repo on GitHub?

I'm wondering if this is the same problem as #32900. The issue was caused by System.Text.Json asking for a different version of System.Buffers than it claimed to be using and the fix was to adding a binding redirect for 4.0.3.0 of System.Buffers.