SocketsHttpHandler HTTP/2 pings are broken in multiple scenarios

[MihaZupan]

HTTP/2 pings are used to keep connections from being completely idle, and to detect dead ones.
With long lived requests that have no implied timeout (e.g. gRPC streaming), they are the way to eventually detect and abort unresponsive servers.

The way our logic works today, we'll only run heartbeat logic on HTTP/2 connections that are in the _availableHttp2Connections list.

This means that pings are broken in the following scenarios:

1. The connection reached its stream limit and was temporarily removed from the available list.
2. We received a GOAWAY frame indicating that existing requests will be processed. The first thing we do when receiving the frame is to mark the connection as shut down, and remove it from the available pool.
3. If the SocketsHttpHandler instance is disposed. Today any existing requests on the instance will continue as normal, but we're also stopping the heartbeat timer and therefore breaking pings. This case could be looked at as user error to a degree though.

The 2nd issue was hit by a user who reported it on Discord (after lots of back and forth to figure out why things aren't working as expected). In their case they are using long-lived gRPC streams, and configured HTTP/2 ping keep alives and timeouts. When the backend service was restarting/scaling down/failing etc., they observed seeing a GOAWAY frame, sometimes eventually followed by the request failing due to the connection observing an EOF.
But sometimes, they would receive a GOAWAY frame and then nothing happened, all requests appearing stuck forever. This is the case where we might not notice that a TCP connection is dead until we try writing to it / enforce a reading timeout - precisely what HTTP/2 pings are supposed to do.

Example traces

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[fabianoliver]

Thanks a ton for your insight over the past days @MihaZupan !

It would be absolutely fantastic if an upcoming fix could be included in a future service patch for NET10 (and possibly prior supported NET versions). Or of course, if there's even a preview build of some NuGet package with a fix (not sure that's currently possible/available outside of the runtime itself?), we'd be more than happy to use that too for the time being!

Since I'm sure the bar for those is quite high though, I'll add a bit more colour from our end:

• Our architecture is heavily streaming based. We run around 100 microservices, totalling hundreds of unary server-streaming grpc endoints, and running with many hundreds of pods
• The issue in effect means that anything from an ungraceful pod shutdown to a number of other events can lead to client streams silently breaking, often just in one or few pods
• This issue has been haunting us for over half a year now - and that is since we first spotted it behaviourally, I'm rather sure it must have happened before already
• We have had about half a dozen serious prod incidents based on this issue since then, and ...
• Countless (ca weekly) near-misses, i.e. us spotting it before users did in a significant way

This issue is quite tricky as:

It can be extremely hard to spot that and when this issue happens

• Streams just die silently. There are no exceptions, no stream terminations, there are no logs or metrics indicating any issues
• The only way for us to monitor this so far is with business logic specific metrics, which needs to be hand-wired and tuned for every single endpoint
• Imagine eg a change stream over some database - you can't really reason about any expected message frequency. Even for one and the same table, maybe its many per seconds, maybe its none for a few days
• The best alerts we have had so far only cover a low single digit percentage of all of our gRPC endoints, misfire regularly, and have still missed some incidents

Difficult to figure out what is going wrong

• No indications of any errors in logs or metrics
• Since case 2 of issue only happens when the TCP shutdown isn't graceful, it's very hard to reproduce - until the very end, I didn't manage to reproduce it with any minimal example, we could only ever reproduce it with an actual prod service (likely load on that service affecting its shutdown and ultimately triggering this), which makes debugging a lot less fun
• Lacking specific pointers of what's going wrong, we had to start investigating in order of probable causes, starting with the most likely on: Our own code. Our streaming operators, our grpc interceptors, our server setup, our infra, and just about everything else
• After that didn't yield much, we started trace logging gRPC on a dedicated repro client (extensiveness of trace logs would not have been feasible on real prod services)
• gRPC trace logs suggested gRPC stream wrappers were not disposed in faulty cases, which pointed towards the HTTP stack
• We then started logging Http events and after days of sifting through logs and pestering Mihu, confirmed that something didn't seem right about Heartbeats

So in short, in the absence of any error indications, it was quite a long way to figure out what was going on - could imagine there might be other devs out there battling through a similar investigation process.

Wide impact on the ecosystem

• This likely affects pretty much all stacks using anything leveraging long running HTTP2 streams
• Including of course everyone using gRPC server streaming methods (and probably non-streaming unary calls without a deadline etc.)

There is currently no way (that I'm aware of) to circumvent or mitigate the issue

• On our end, incidents seem likely to continue until a fixed runtime is available
• At the moment, nothing came to mind in terms of how we could circumvent this on the client side of things (timeouts and similar aren't really applicable as our streams are generally infinite)
• In terms of alerting, even with the knowledge we have now, so far no immediate way to even generically alert on this issue comes to mind

Thanks a lot for looking into this!