Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HTTP/3] map QuicException to OCE in case of a cancellation #98220

Closed
alexandrehtrb opened this issue Feb 9, 2024 · 6 comments · Fixed by #103081
Closed

[HTTP/3] map QuicException to OCE in case of a cancellation #98220

alexandrehtrb opened this issue Feb 9, 2024 · 6 comments · Fixed by #103081
Assignees
Labels
area-System.Net.Http bug in-pr There is an active PR which will close this issue when it is merged
Milestone

Comments

@alexandrehtrb
Copy link

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Feb 9, 2024
@ghost
Copy link

ghost commented Feb 9, 2024

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author: alexandrehtrb
Assignees: -
Labels:

area-System.Net.Http, untriaged

Milestone: -

@ghost
Copy link

ghost commented Feb 20, 2024

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author: alexandrehtrb
Assignees: -
Labels:

untriaged, area-System.Net.Quic

Milestone: -

@MihaZupan
Copy link
Member

Moving to Quic area given the QUIC_STATUS_INVALID_PARAMETER failure

@ManickaP
Copy link
Member

The server exception is dotnet/aspnetcore#45105, please upvote it there so that we know this troubles our users.

And I assume that the client exception happens when you cancel the token. So the only thing here is the way we bubble up the exception when CT gets cancelled. We should wrap it in OCE instead of HRE.

Moving back to HTTP as this is in S.N.Http.

@ghost
Copy link

ghost commented Feb 20, 2024

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author: alexandrehtrb
Assignees: -
Labels:

area-System.Net.Http, untriaged, area-System.Net.Quic

Milestone: -

@ManickaP ManickaP removed area-System.Net.Quic untriaged New issue has not been triaged by the area owner labels Feb 20, 2024
@ManickaP ManickaP added this to the 9.0.0 milestone Feb 20, 2024
@ManickaP ManickaP added the bug label Feb 20, 2024
@ManickaP ManickaP changed the title HTTP/3 stress test has occasional QuicException [HTTP/3] map QuicException to OCE in case of a cancellation May 23, 2024
@liveans liveans self-assigned this Jun 5, 2024
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Jun 5, 2024
@ManickaP
Copy link
Member

Based on the recent HTTP stress report: #42211 (comment) we should address the same problem on the response stream level. I.e.: on this stack:

client-1  | System.Net.Http.HttpRequestException: Error while copying content to a stream.
client-1  |  ---> System.Net.Http.HttpIOException: An error occurred while sending the request. (Unknown)
client-1  |  ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
client-1  |  ---> System.Net.Quic.QuicException: Operation aborted.
client-1  |    at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
client-1  |    at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 627
client-1  |    at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 650
client-1  |    at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 679
client-1  | --- End of stack trace from previous location ---
client-1  |    at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
client-1  |    at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 803
client-1  |    at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 431
client-1  |    at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1175
client-1  |    at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 482
client-1  |    at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
client-1  |    at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
client-1  |    --- End of inner exception stack trace ---
client-1  |    --- End of inner exception stack trace ---
client-1  |    at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1268
client-1  |    at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
client-1  |    at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 106
client-1  |    at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
client-1  |    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
client-1  |    --- End of inner exception stack trace ---
client-1  |    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
client-1  |    at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
client-1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372
client-1  | --- End of stack trace from previous location ---
client-1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204
client-1  | 
client-1  | 	 8: POST Duplex Slow          Fail: 1	Timestamps: 13:41:02.1372148, Duration: 00:00:00.0249275, Cancelled: True
client-1  | 	    TOTAL                     Fail: 1

@github-actions github-actions bot locked and limited conversation to collaborators Jul 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Net.Http bug in-pr There is an active PR which will close this issue when it is merged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants