Http Stress Status Report #42211

ManickaP · 2020-09-14T18:17:06Z

Http Stress Status Report

What we've run so far:

OS	HTTP 1.1	HTTP 2.0	Notes
Windows	4+ h (Mana) -> 2 errors	12 h (JJ) -> 11 errors
	6 h (Miha) -> 8 errors	6 h (Miha) -> 7 errors
	10 h (Miha) -> 53 errors	7 h (Miha) -> 139 errors	7 h = 1 + 6 h
Linux	17 h (JJ) -> 0 errors	12 h (JJ) -> 608 errors	we should rerun as this may be environmental problem
	12 h (Furtik) -> 0 errors	12 h (Furtik) -> 104 errors	ran on released runtime (not on master)

HTTP 2.0 Error Statistics

Error Type	Linux	Windows
Success	135,727,755	105,273,552
Errors	712	157
System.Threading.Tasks.TaskCanceledException: The operation was canceled.	37	8
System.Threading.Tasks.TaskCanceledException: A task was canceled.	13	4
System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.	464	18
System.Net.Sockets.SocketException (32): Broken pipe	198	0
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.	0	118
System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.	0	9

HTTP 1.1 Error Statistics

Error Type	Linux	Windows
Success	171,673,491	128,658,822
Errors	0	61
System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.	0	61

What we need to run:

More HTTP 1.1 Linux runs to confirm that we're clear. (easy, hi pri)
More HTTP 2.0 Linux runs to confirm that we have all error types captured. (easy, hi pri)
HTTP 2.0 tests against HTTPSys to eliminate/confirm server as the culprit. (mid, mid pri)
Run the matrix against 3.1 and compare. (hard, mid pri)

Existing issues, root caused:

HTTP 1.1 connect cancellation deadlock: Cancellation of Socket.ConnectAsync intermitently hangs #42198
HTTP 2.0 sending RESET before END_STREAM: HTTP/2 stress server intermitently fails in duplex scenarios #42200
HTTP 2.0 throwing TaskCancelledException as a reaction on GO_AWAY: HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
NEW HTTP 2.0 infinite loop when connecting H2C to HTTP 1.0 localhost server: H2C connection to HTTP.sys on localhost causes infinite loop #42259

Discovered exceptions, not-investigated:

HTTP 2.0 System.Net.Sockets.SocketException (32): Broken pipe. (Linux only)
HTTP 2.0 System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
~~HTTP 2.0 System.Threading.Tasks.TaskCanceledException: The operation was canceled.~~ HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Threading.Tasks.TaskCanceledException: A task was canceled. --> might be covered by HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host. (Windows only)
HTTP 2.0 System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. (Windows only)
NEW HTTP 1.1 System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Windows only)

The discovered exceptions confirm what we've collected so far from the pipelines: #40388.

Distributable tasks by priority:

More HTTP 1.1 Linux runs: http11run
More HTTP 2.0 Linux runs: http20run
Investigate Windows HTTP 1.1 errors: winErr3
Investigate Windows HTTP 2.0 errors: winErr1, winErr2
Provide fix for #42200
Provide fix for #42198
Help with HTTPSys client connection errors: httpSys: put on back-burner
Run the tests against .NET Core 3.1: net31: put on back-burner

Tips and Tricks for investigations:

Clear up docker containers and images after a product code change (docker container prune && docker image prune -a)
Once container is built, switch -b might be omitted for subsequent re-runs (skips the runtime build)
Don't use containers for investigations, they're slow and rebuild takes too long
- To run the app against locally built runtime, swap artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/ with the globally installed runtime (/usr/share/dotnet/shared/Microsoft.NETCore.App/your-latest-5.0-version)
  - Make a backup of the global runtime!
  - Using testhost's corerun didn't work for me since the app depends on ASP .NET Core SDK
  - If you change the product code, rebuild just System.Net.Http and copy System.Net.Http.dll to the global runtime again
- Build System.Net.Http/tests/StressTests/HttpStress
- Open 2 terminals and run:
  - server: dotnet run -runMode server -aspnetlog
    - -aspnetlog: console logging of server errors
    - -serverUri https://localhost:5002: bind to a different port (when running multiple tests in parallel)
  - client: dotnet run -runMode client
    - -serverUri https://localhost:5002: connect to a different port (when running multiple tests in parallel)
    - -ops 1 2 3: run only operation 1, 2 and 3 (GET, PUT Slow, etc...)
    - -trace: saves internal client/server traces in a log file, very verbose, useable only for very short runs
  - more options in source code: https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L37-L62

If you have any improvements to the stress app or the containers, please create a PR and don't keep it just for yourself.
If you have more tips and tricks for running the tests, please share them.

The text was updated successfully, but these errors were encountered:

ghost · 2020-09-14T18:17:08Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Fixes: stress client double read of content fixed fixed stress client hangs at start and stop leveraged HttpVersionPolicy increased pipeline timeout since we doubled the runs fixed base docker images to avoid missing IO.Pipelines Kestrel exception. Re-hauled tracing: added server file logging added log file rotation Minor renames. Contributes to: #42211 and #42198

antonfirsov · 2021-03-22T16:59:28Z

The description of #49999 has some useful links to stack traces according how failures occur today. The "new" HTTP 1.1 connection timeout seems to be a dominant issue on CI, failing on almost every Windows run.

antonfirsov · 2021-05-10T11:18:56Z

In the last 20 CI runs, there were 3 occurrences of the "System.Net.Sockets.SocketException (10054)" Windows + HTTP 2.0 case. I won't be surprised, if it's also a firewall issue being eventually solved by #52381. If not we should look into it.

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

antonfirsov · 2021-06-28T10:36:02Z

I browsed through the failures of the last 25 days in the CI, and it seems that we are facing the same errors as las September. The only thing worth mentioning is that I've noticed a TaskCanceledException failure with a stack trace that comes from PUT Slow, instead of POST ExpectContinue I can't decide if it's covered by #42472 or not, we may want to check it:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
-> System.TimeoutException: A task was canceled.
-> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, Boolean finalFlush, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1482
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1275
at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 623
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 291
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1941
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 833
at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 897
at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 910
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 548

Full CI run

karelz · 2021-07-29T08:03:13Z

@antonfirsov can you please add note about your analysis from last week?

antonfirsov · 2021-07-29T14:40:36Z

Report 7/29

with manual observations after a brief look through the CI output of between 7/13-7/29, excluding the HTTP3 failures:

There is a new failure type now tracked under Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159
Other failures I've seen are known and described in the opening post
There are 2 cases where the Job run is unfinished. Likely an AZP anomaly, but we should keep monitoring if it happens again

Going back a bit more I see signs of #55261 (see here), but it doesn't seem to happen since 7/8 (EDIT: Because the test was disabled on 7/8).

antonfirsov · 2021-08-05T15:14:19Z

Report 8/5

brief look through the CI output between 7/29-8/5:

No new failure types in the test runs
Add SslApplicationProtocol.Http3 #56775 broke the last build, which is being fixed in Fix HttpStress build #56912

antonfirsov · 2021-08-13T09:51:06Z

Report 8/13

No new failure types in HTTP 1.1 & 2.0 in the last 1 week.

antonfirsov · 2021-08-24T15:19:42Z

Report 8/24

No new failure types in HTTP 1.1 & 2.0 since last report
There is one ~~occurrence~~ occurence of an unexpected ObjectDisposedException that happened after merging Do not drain HttpContentReadStream if the connection is disposed #57287. I'm reluctant to reopen Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159 right away, will do local runs and keep watching the CI, and open an issue if it happens again.

ManickaP · 2021-08-24T15:35:44Z

The occurrence is on 13.8., isn't that before the fix got merged?

antonfirsov · 2021-08-24T16:06:07Z

Sorry, linked wrong build, this one happened on the 18th:
https://dev.azure.com/dnceng/public/_build/results?buildId=1301334&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=45b31e99-b18c-506f-46cf-7d3a4d5140a9

karelz · 2021-08-25T12:15:23Z

@Vijay-Nirmal please file a separate issue with additional information. This issue is tracking general stress test runs, which may or may not be related to what you see. It is fine to link this issue. Thanks!

antonfirsov · 2024-01-18T16:30:57Z

Report 2024-01-18

Looking Linux runs since 2024-01-05. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x crash, malloc(): unaligned tcache chunk detected

client_1  | [createdump] Gathering state for process 7 dotnet
client_1  | [createdump] Crashing thread 0014 signal 6 (0006)
client_1  | [createdump] Writing minidump with heap to file /dumps-share/coredump.7
client_1  | [createdump] Written 556867584 bytes (135954 pages) to core file
client_1  | [createdump] Target process is alive
client_1  | [createdump] Dump successfully written in 737ms

https://dev.azure.com/dnceng-public/public/_build/results?buildId=525074&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

Crash dump attached. We should look at it if it happens again.
DumpsLinux.zip

Time out at shutdown (##[error]The task has timed out):
https://dev.azure.com/dnceng-public/public/_build/results?buildId=527377&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0
Last run: 20240118.3

SslStress on `main`

11x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240118.3

`release/8.0` branch

HttpStress on `release/8.0`

No failures on Linux
Last run: #20240118.2

SslStress on `release/8.0`

No failures on Linux
Last run: 20240118.2

antonfirsov · 2024-02-15T16:25:17Z

Report 2024-02-15

Looking Linux runs since 2024-01-18. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x crash, malloc(): mismatching next->prev_size (unsorted) in the HTTP/3.0 run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=567734&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

client_1  | malloc(): mismatching next->prev_size (unsorted)
client_1  | Aborted (core dumped)
httpstress_client_1 exited with code 134

This looks very similar to the malloc crash detected in the previous run. Waiting for one more occurence to open an issue.

Crash dump:
DumpsLinux (1).zip

Last run: 20240215.2

SslStress on `main`

8 x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240215.2

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (comment)
Last run: 20240215.1

SslStress on `release/8.0`

No failures on Linux
Last run: 20240215.1

antonfirsov · 2024-02-29T15:56:47Z

Report 2024-02-29

Looking Linux runs since 2024-01-15. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x H3_REQUEST_INCOMPLETE with QuicException: Stream aborted by peer (269) on Linux:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=580954&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_INCOMPLETE' (0x10d). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (269).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 578
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 633
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 665
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 266
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 997

⬆️ this is a new error, we should open an issue if it happens again.

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240229.3

SslStress on `main`

No failures on Linux
Last run: 20240229.3

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240229.2

SslStress on `release/8.0`

No failures on Linux
Last run: 20240229.2

antonfirsov · 2024-03-19T16:01:57Z

Report 2024-03-19

Looking Linux runs since 2024-02-29. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (comment)
1 x [HttpStress] malloc crashes in HTTP/3 runs #99960
Last run: 20240319.2

SslStress on `main`

4 x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240319.3

`release/8.0` branch

HttpStress on `release/8.0`

1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240319.3

SslStress on `release/8.0`

No failures on Linux
Last run: 20240319.4

antonfirsov · 2024-03-26T16:29:34Z

Report 2024-03-26

Looking Linux runs since 2024-03-19. Windows runs are up on main since Sunday.

`main` (9.0) branch

HttpStress on `main`

1 x [HttpStress] malloc crashes in HTTP/3 runs #99960 (Linux)
1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 (Linux)
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (Windows)

SslStress on `main`

1 x [SslStress] SocketException (111): Connection refused #94222

`release/8.0` branch

Windows builds are still broken, #100145 needs a backport.

HttpStress on `release/8.0`

1 x [SslStress] SocketException (111): Connection refused #94222

SslStress on `release/8.0`

No failures on Linux

antonfirsov · 2024-04-02T17:06:59Z

Report 2024-04-02

Looking at all runs since 2024-03-26.

`main` (9.0) branch

HttpStress on `main`

200-300 hits of HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 per run on Linux
30-50 hits of [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534 per run on Linux, 1 hit on Windows

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 on Linux

SslStress on `release/8.0`

1x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044. Waiting for more stats before reopening.

antonfirsov · 2024-04-09T14:46:38Z

Report 2024-04-09

Looking at runs since 2024-04-02.

`main` (9.0) branch

HttpStress on `main`

200-300 hits of HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 per run on Linux
30-50 hits of [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534 per run on Linux

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

2 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (1 on Windows 1 on Linux)

SslStress on `release/8.0`

No failures

antonfirsov · 2024-04-26T18:28:46Z

Report 2024-04-26

Looking at runs since 2024-04-09.

`main` (9.0) branch

HttpStress on `main`

The extreme amount of hits for #76183 became lower, and hits for #100534 ceased, however we are seeing a couple of failures similar to #76183 accumulating. Let's see if the frequency changes before opening separate issues.

13 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183, happens on both Linux and Windows
2 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 H3_REQUEST_INCOMPLETE on Windows
2 x H3_FRAME_ERROR in POST Duplex Slow in a single run on Windows

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_FRAME_ERROR' (0x106). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.SkipUnknownPayloadAsync(Int64 payloadLength, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1080
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 848
   at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1284
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1175
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1258
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372

1 x H3_REQUEST_REJECTED in GET Aborted on Windows, similarly to Http Stress Status Report #42211 (comment)

https://dev.azure.com/dnceng-public/public/_build/results?buildId=651163&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_REJECTED' (0x10b). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 580
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 635
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 308
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1208
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1247
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 106
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 542
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 288
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 311

2 x HttpIOException with Unknown error code in POST Duplex Slow on Windows

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpIOException: An error occurred while sending the request. (Unknown)
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 615
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 638
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 419
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 481
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1268
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-09T16:20:21Z

Report 2024-05-09

Looking at runs since 2024-04-26.

`main` (9.0) branch

HttpStress on `main`

There was a temporary issue with dotnet-install that reduced the number of Windows runs, but it is gone now.
Linux builds are failing because of the broken msquic/main build. @CarnaViire will file a fix for that.
Known failures in the remaining runs:

SslStress on `main`

There was a temporary issue with dotnet-install that reduced the number of Windows runs, but it is gone now.
No failures otherwise

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-14T16:08:48Z

Report 2024-05-14

Looking at runs since 2024-05-09.

`main` (9.0) branch

HttpStress on `main`

#102085 switched the Linux runs to use msquic v2.3.5 instead of their main branch which fixed #101625.

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HttpStress] HttpIOException with HttpRequestError=Unknown and QuicError=OperationAborted on Windows #102202
1 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue on Linux, which is a failure we didn't see before. Might be the reuslt of the v2.3.5 switch.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=674960&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Failure Type 1/2:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_ID_ERROR' (0x108). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 845
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 301
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 87
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 430
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_10>d.MoveNext() in C:\app\ClientOperations.cs:line 408
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-21T16:52:49Z

Report 2024-05-21

Looking at runs since 2024-05-14.

`main` (9.0) branch

HttpStress on `main`

5 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
7 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
1 x [HttpStress] HttpIOException with HttpRequestError=Unknown and QuicError=OperationAborted on Windows #102202
1 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue on Windows, similar to the failure in the previous report.

SslStress on `main`

1 x transient build failure
No failures otherwise

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-30T14:33:46Z

Report 2024-05-30

Looking at runs since 2024-05-21.

`main` (9.0) branch

HttpStress on `main`

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-06T14:42:42Z

Report 2024-06-06

Looking at runs since 2024-05-30.

`main` (9.0) branch

HttpStress on `main`

[HttpStress] All runs of POST Multipart Data failing #103128 spoiled all runs across all OS-es and versions

SslStress on `main`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-11T15:40:17Z

Report 2024-06-11

Looking at runs since 2024-06-06.

`main` (9.0) branch

HttpStress on `main`

4 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
2 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue in the same run on Windows, similar to the failure in the previous report, but this time in POST Duplex Slow and GET Parameters

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

ManickaP · 2024-06-11T17:41:51Z

2 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue in the same run on Windows, similar to the failure in the #42211 (comment), but this time in POST Duplex Slow and GET Parameters

Looks like: #98220, Ahmet has a PR up for this in: #103081, but I see it happens in reading the content after SendAsync, so we should address that as well.

antonfirsov · 2024-06-18T15:50:58Z

Report 2024-06-18

Looking at runs since 2024-06-11.

`main` (9.0) branch

HttpStress on `main`

3 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-25T18:05:13Z

Report 2024-06-25

Looking at runs since 2024-06-18.

`main` (9.0) branch

HttpStress on `main`

2 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
5 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559, Windows / Linux
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534, Windows

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-07-10T12:28:20Z

Report 2024-07-10

Looking at runs since 2024-06-25.

`main` (9.0) branch

HttpStress on `main`

We assume these issues got fixed with an upstream update starting with 2024-07-09.

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-08-20T14:09:59Z

Report 2024-08-20

Looking at runs since 2024-07-21.

Windows builds are broken since August 11: #106694

`main` (9.0) branch

HttpStress on `main`

17 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534, Windows
2 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559, Windows
1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959, Windows

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-08-27T15:55:10Z

Report 2024-08-27

Looking at runs since 2024-08-20.

`main` (9.0) branch

HttpStress on `main`

Looking at Linux runs only becaue of #106694.

SslStress on `main`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-09-03T15:35:19Z

Report 2024-09-03

Looking at runs since 2024-08-27.

`main` (9.0) branch

HttpStress on `main`

Looking at Linux runs only becaue of #106694.

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183

SslStress on `main`

No failure.

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-09-10T15:50:39Z

Report 2024-09-10

Looking at runs since 2024-09-03.

`main` branch

HttpStress on `main`

4 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x H3_INTERNAL_ERROR in PUT Slow on Linux. Previously not seen in this form, but caused by QuicException: Stream aborted by peer (258) which is not new.
1 x H3_REQUEST_REJECTED as in Http Stress Status Report #42211 (comment)

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_INTERNAL_ERROR' (0x102). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (258).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 607
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 662
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 694
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 323
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 850
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 387
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 214
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 291
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 370
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 323
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 335
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 111
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 434
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 453
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 206

SslStress on `main`

No failure.

`release/9.0` branch

HttpStress on `release/9.0`

4 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559 on Linux
2 x [HttpStress] H3_FRAME_ERROR in Post Duplex Slow #102880 on Windows

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-10-08T15:48:02Z

Report 2024-10-08

Looking at runs since 2024-09-10.

`main` branch

HttpStress on `main`

10 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
3 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559 (FRAME_UNEXPECTED)
7 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 (H3_REQUEST_INCOMPLETE)
4 x [HttpStress] H3_FRAME_ERROR in Post Duplex Slow #102880 (H3_FRAME_ERROR)

SslStress on `main`

No failure.

`release/9.0` branch

HttpStress on `release/9.0`

24 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
6 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 (H3_REQUEST_INCOMPLETE)
5 x [HttpStress] H3_FRAME_ERROR in Post Duplex Slow #102880 (H3_FRAME_ERROR)
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534 (QuicException)
1 x H3_CLOSED_CRITICAL_STREAM

client_1  |  ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_CLOSED_CRITICAL_STREAM' (0x104). (HttpProtocolError)
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 301
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 370
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 323
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 335
client_1  |    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 111
client_1  |    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 434
client_1  |    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
client_1  |    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 453
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 206

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-10-22T15:43:16Z

Report 2024-10-22

Looking at runs since 2024-09-08.

`main` branch

HttpStress on `main`

Some issues are still there, regardless of the aspnet fix

2 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-10-29T16:48:04Z

Report 2024-10-29

Looking at runs since 2024-10-22.

`main` branch

HttpStress on `main`

Some issues are still there, regardless of the aspnet fix

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

ManickaP added the area-System.Net.Http label Sep 14, 2020

ManickaP added this to the 6.0.0 milestone Sep 14, 2020

ManickaP self-assigned this Sep 14, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP mentioned this issue Sep 16, 2020

HTTP stress test improvements #42313

Merged

karelz mentioned this issue Sep 21, 2020

HttpStress tests failing in difference scenarios on sending data to server #40388

Closed

karelz unassigned ManickaP Sep 21, 2020

karelz added the bug label Sep 21, 2020

ManickaP mentioned this issue Mar 22, 2021

Various HTTP Stress test failures on the CI #49999

Closed

antonfirsov mentioned this issue Apr 7, 2021

HttpStress: connection failures with HTTP 1.1 caused by Windows Firewall #50854

Closed

antonfirsov mentioned this issue May 6, 2021

HttpStress: Disable firewall in Windows runs #52381

Merged

antonfirsov mentioned this issue May 12, 2021

HttpStress: also disable firewall for HTTP2 runs #52643

Merged

antonfirsov added a commit that referenced this issue May 18, 2021

HttpStress: also disable firewall for HTTP2 (#52643)

8fb2eb5

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

karelz modified the milestones: 6.0.0, 7.0.0 Jul 29, 2021

This comment has been minimized.

Sign in to view

antonfirsov mentioned this issue May 30, 2024

[HttpStress] H3_FRAME_ERROR in Post Duplex Slow #102880

Closed

antonfirsov mentioned this issue Jun 11, 2024

SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

Closed

ManickaP mentioned this issue Jun 11, 2024

[HTTP/3] map QuicException to OCE in case of a cancellation #98220

Closed

antonfirsov modified the milestones: 9.0.0, 10.0.0 Jun 25, 2024

Http Stress Status Report #42211

Http Stress Status Report #42211

Comments

ManickaP commented Sep 14, 2020 • edited Loading

Http Stress Status Report

What we've run so far:

HTTP 2.0 Error Statistics

HTTP 1.1 Error Statistics

What we need to run:

Existing issues, root caused:

Discovered exceptions, not-investigated:

Distributable tasks by priority:

Tips and Tricks for investigations:

ghost commented Sep 14, 2020

antonfirsov commented Mar 22, 2021

antonfirsov commented May 10, 2021 • edited Loading

antonfirsov commented Jun 28, 2021

karelz commented Jul 29, 2021

antonfirsov commented Jul 29, 2021 • edited by karelz Loading

Report 7/29

antonfirsov commented Aug 5, 2021

Report 8/5

antonfirsov commented Aug 13, 2021

Report 8/13

antonfirsov commented Aug 24, 2021 • edited Loading

Report 8/24

ManickaP commented Aug 24, 2021

antonfirsov commented Aug 24, 2021 • edited Loading

This comment has been minimized.

karelz commented Aug 25, 2021

antonfirsov commented Jan 18, 2024

Report 2024-01-18

main (9.0) branch

release/8.0 branch

antonfirsov commented Feb 15, 2024

Report 2024-02-15

main (9.0) branch

release/8.0 branch

antonfirsov commented Feb 29, 2024 • edited Loading

Report 2024-02-29

main (9.0) branch

release/8.0 branch

antonfirsov commented Mar 19, 2024

Report 2024-03-19

main (9.0) branch

release/8.0 branch

antonfirsov commented Mar 26, 2024

Report 2024-03-26

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 2, 2024

Report 2024-04-02

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 9, 2024

Report 2024-04-09

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 26, 2024 • edited Loading

Report 2024-04-26

main (9.0) branch

release/8.0 branch

antonfirsov commented May 9, 2024 • edited Loading

Report 2024-05-09

main (9.0) branch

release/8.0 branch

antonfirsov commented May 14, 2024

Report 2024-05-14

main (9.0) branch

release/8.0 branch

antonfirsov commented May 21, 2024

Report 2024-05-21

main (9.0) branch

release/8.0 branch

antonfirsov commented May 30, 2024

Report 2024-05-30

main (9.0) branch

release/8.0 branch

antonfirsov commented Jun 6, 2024

Report 2024-06-06

ManickaP commented Sep 14, 2020 •

edited

Loading

antonfirsov commented May 10, 2021 •

edited

Loading

antonfirsov commented Jul 29, 2021 •

edited by karelz

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented Feb 29, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented Apr 26, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented May 9, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch