Client unhandled exception at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)

[osexpert]

What version of gRPC and what language are you using?

Grpc.Net.Client 2.76.0

What operating system (Linux, Windows,...) and version?

Windows 10

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.net 10

What did you do?

If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.

Read things from server:

try
{
    while (await call.ResponseStream.MoveNext().ConfigureAwait(false))
    {
        // stuff
    }
}
catch (Exception e)
{
    // log
}

Some time later, I came back and I was met by unhandled excpetion.

What did you expect to see?

No unhandled exception:-)

What did you see instead?

It is strange, as I am catching all. Also the unhandled exception stack trace is cut short and start out of nowhere.

.NETCoreApp,Version=v10.0
System.Net.Http.HttpProtocolException: The HTTP/2 server closed the connection. HTTP/2 error code 'STREAM_CLOSED' (0x5). (HttpProtocolError)
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)

Looking at the code, I find it a bit supicious:

E:\git\grpc-dotnet\src\Grpc.Net.Client\Internal\HttpContentClientStreamReader.cs

Line 112:

                // Save move next task to track whether it is complete
                _moveNextTask = MoveNextCore(cancellationToken);

Anything else we should know about your project / environment?

I asked Claude:

Yes, there's something interesting here. Look at this pattern carefully:

lock (_moveNextLock)
{
    _moveNextTask = MoveNextCore(cancellationToken);  // assigned INSIDE lock
}

return _moveNextTask;  // read OUTSIDE lock

_moveNextTask is assigned inside the lock, but the field is read again at return _moveNextTask outside the lock. These are two separate field accesses. If anything modifies _moveNextTask between the lock being released and the return, the caller gets back a different task than the one that was just created — and the original faulted task has no reference, becomes unobserved, and its exception escapes via the unobserved task machinery.

Now look at the catch block inside MoveNextCore:

catch (Exception ex)
{
    var newException = _call.ResolveException("Error reading next message.", ex, out var status, out var resolvedException);
    if (!_call.ResponseFinished)
    {
        _call.ResponseStreamEnded(status.Value, finishedGracefully: false);  // ? this
    }
    ...
    throw;
}

_call.ResponseStreamEnded completes CallTask. That completion could trigger continuations. And look at the very beginning of MoveNext():

if (_call.CallTask.IsCompletedSuccessfully())
{
    ...
    return Task.FromException<bool>(_call.CreateRpcException(status));
}

If a continuation triggered by ResponseStreamEnded causes MoveNext() to be called again — even indirectly — it would acquire the lock, set _moveNextTask to a NEW task, and release the lock. Then the original caller's return _moveNextTask reads the NEW task, returns it, and the ORIGINAL faulted task with the raw HttpProtocolException has no observer.

This would perfectly explain:

• Raw HttpProtocolException escaping unwrapped
• Zero user frames — the task was never awaited by user code
• It being a one-off race condition

[osexpert]

I think the code could probably be changed to

        Task<bool> moveNextTask;
        lock (_moveNextLock)
        {
            using (_call.StartScope())
            {
                // Pending move next need to be awaited first
                if (IsMoveNextInProgressUnsynchronized)
                {
                    var ex = new InvalidOperationException("Can't read the next message because the previous read is still in progress.");
                    Log.ReadMessageError(_logger, ex);
                    return Task.FromException<bool>(ex);
                }

                moveNextTask = MoveNextCore(cancellationToken);
                // Save move next task to track whether it is complete
                _moveNextTask = moveNextTask;
            }
        }
        return moveNextTask;

to avoid returning _moveNextTask being possibly overwritten by someone else

[osexpert]

But also this code is breaking some common guidelines about async. The using is not valid here when MoveNextCore(cancellationToken) is not awaited. StartScope will be disposed before MoveNextCore(cancellationToken) is completed, and even if it may work, this is "wrong".
The wrongness probably stems from the lock (_moveNextLock), since lock can not cover await. So it is wrongness in many different layers here.

[osexpert]

I made a PR for the moveNextTask: #2738

The using (_call.StartScope()) is worse, but this is a different story, so ignoring it for now. Would probably need to await MoveNextCore and lock with SemaphoreSlim, to make the using cover the execution of MoveNextCore. It may be that the using is not ment to cover the execution, but hard to say. Without explicitly stated, I would categorize this as a bug.

[osexpert]

I hope that after the PR, it will no longer show non rooted stack traces:-)