Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Http Stress Status Report #42211

Open
ManickaP opened this issue Sep 14, 2020 · 72 comments
Open

Http Stress Status Report #42211

ManickaP opened this issue Sep 14, 2020 · 72 comments
Assignees
Milestone

Comments

@ManickaP
Copy link
Member

ManickaP commented Sep 14, 2020

Http Stress Status Report

What we've run so far:

OS HTTP 1.1 HTTP 2.0 Notes
Windows 4+ h (Mana) -> 2 errors 12 h (JJ) -> 11 errors
6 h (Miha) -> 8 errors 6 h (Miha) -> 7 errors
10 h (Miha) -> 53 errors 7 h (Miha) -> 139 errors 7 h = 1 + 6 h
Linux 17 h (JJ) -> 0 errors 12 h (JJ) -> 608 errors we should rerun as this may be environmental problem
12 h (Furtik) -> 0 errors 12 h (Furtik) -> 104 errors ran on released runtime (not on master)

HTTP 2.0 Error Statistics

Error Type Linux Windows
Success 135,727,755 105,273,552
Errors 712 157
System.Threading.Tasks.TaskCanceledException:
The operation was canceled.
37 8
System.Threading.Tasks.TaskCanceledException:
A task was canceled.
13 4
System.IO.IOException:
The response ended prematurely while waiting for the next frame from the server.
464 18
System.Net.Sockets.SocketException (32):
Broken pipe
198 0
System.Net.Sockets.SocketException (10054):
An existing connection was forcibly closed by the remote host.
0 118
System.Net.Sockets.SocketException (10053):
An established connection was aborted by the software in your host machine.
0 9

HTTP 1.1 Error Statistics

Error Type Linux Windows
Success 171,673,491 128,658,822
Errors 0 61
System.Net.Sockets.SocketException (10061):
No connection could be made because the target machine actively refused it.
0 61

What we need to run:

  • More HTTP 1.1 Linux runs to confirm that we're clear. (easy, hi pri)
  • More HTTP 2.0 Linux runs to confirm that we have all error types captured. (easy, hi pri)
  • HTTP 2.0 tests against HTTPSys to eliminate/confirm server as the culprit. (mid, mid pri)
  • Run the matrix against 3.1 and compare. (hard, mid pri)

Existing issues, root caused:

Discovered exceptions, not-investigated:

The discovered exceptions confirm what we've collected so far from the pipelines: #40388.

Distributable tasks by priority:

  • More HTTP 1.1 Linux runs: http11run
  • More HTTP 2.0 Linux runs: http20run
  • Investigate Windows HTTP 1.1 errors: winErr3
  • Investigate Windows HTTP 2.0 errors: winErr1, winErr2
  • Provide fix for #42200
  • Provide fix for #42198
  • Help with HTTPSys client connection errors: httpSys: put on back-burner
  • Run the tests against .NET Core 3.1: net31: put on back-burner

Tips and Tricks for investigations:

  • Clear up docker containers and images after a product code change (docker container prune && docker image prune -a)
  • Once container is built, switch -b might be omitted for subsequent re-runs (skips the runtime build)
  • Don't use containers for investigations, they're slow and rebuild takes too long
    • To run the app against locally built runtime, swap artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/ with the globally installed runtime (/usr/share/dotnet/shared/Microsoft.NETCore.App/your-latest-5.0-version)
      • Make a backup of the global runtime!
      • Using testhost's corerun didn't work for me since the app depends on ASP .NET Core SDK
      • If you change the product code, rebuild just System.Net.Http and copy System.Net.Http.dll to the global runtime again
    • Build System.Net.Http/tests/StressTests/HttpStress
    • Open 2 terminals and run:

If you have any improvements to the stress app or the containers, please create a PR and don't keep it just for yourself.
If you have more tips and tricks for running the tests, please share them.

@ManickaP ManickaP added this to the 6.0.0 milestone Sep 14, 2020
@ManickaP ManickaP self-assigned this Sep 14, 2020
@ghost
Copy link

ghost commented Sep 14, 2020

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 14, 2020
@ManickaP ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 14, 2020
@karelz karelz added the bug label Sep 21, 2020
ManickaP added a commit that referenced this issue Sep 23, 2020
Fixes:
    stress client double read of content fixed
    fixed stress client hangs at start and stop
    leveraged HttpVersionPolicy
    increased pipeline timeout since we doubled the runs
    fixed base docker images to avoid missing IO.Pipelines Kestrel exception.

Re-hauled tracing:
    added server file logging
    added log file rotation

Minor renames.

Contributes to: #42211 and #42198
@antonfirsov
Copy link
Member

The description of #49999 has some useful links to stack traces according how failures occur today. The "new" HTTP 1.1 connection timeout seems to be a dominant issue on CI, failing on almost every Windows run.

@antonfirsov
Copy link
Member

antonfirsov commented May 10, 2021

In the last 20 CI runs, there were 3 occurrences of the "System.Net.Sockets.SocketException (10054)" Windows + HTTP 2.0 case. I won't be surprised, if it's also a firewall issue being eventually solved by #52381. If not we should look into it.

antonfirsov added a commit that referenced this issue May 18, 2021
We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.
@antonfirsov
Copy link
Member

I browsed through the failures of the last 25 days in the CI, and it seems that we are facing the same errors as las September. The only thing worth mentioning is that I've noticed a TaskCanceledException failure with a stack trace that comes from PUT Slow, instead of POST ExpectContinue I can't decide if it's covered by #42472 or not, we may want to check it:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
-> System.TimeoutException: A task was canceled.
-> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, Boolean finalFlush, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1482
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1275
at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 623
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 291
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1941
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 833
at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 897
at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 910
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 548

Full CI run

@karelz karelz modified the milestones: 6.0.0, 7.0.0 Jul 29, 2021
@karelz
Copy link
Member

karelz commented Jul 29, 2021

@antonfirsov can you please add note about your analysis from last week?

@antonfirsov
Copy link
Member

antonfirsov commented Jul 29, 2021

Report 7/29

with manual observations after a brief look through the CI output of between 7/13-7/29, excluding the HTTP3 failures:

Going back a bit more I see signs of #55261 (see here), but it doesn't seem to happen since 7/8 (EDIT: Because the test was disabled on 7/8).

@antonfirsov
Copy link
Member

Report 8/5

brief look through the CI output between 7/29-8/5:

@antonfirsov
Copy link
Member

Report 8/13

No new failure types in HTTP 1.1 & 2.0 in the last 1 week.

@antonfirsov
Copy link
Member

antonfirsov commented Aug 24, 2021

Report 8/24

@ManickaP
Copy link
Member Author

The occurrence is on 13.8., isn't that before the fix got merged?

@antonfirsov
Copy link
Member

antonfirsov commented Aug 24, 2021

@Vijay-Nirmal

This comment has been minimized.

@karelz
Copy link
Member

karelz commented Aug 25, 2021

@Vijay-Nirmal please file a separate issue with additional information. This issue is tracking general stress test runs, which may or may not be related to what you see. It is fine to link this issue. Thanks!

@antonfirsov
Copy link
Member

Report 2024-01-18

Looking Linux runs since 2024-01-05. Windows builds are broken across all branches, see #95750.

main (9.0) branch

HttpStress on main

  • 1 x crash, malloc(): unaligned tcache chunk detected
client_1  | [createdump] Gathering state for process 7 dotnet
client_1  | [createdump] Crashing thread 0014 signal 6 (0006)
client_1  | [createdump] Writing minidump with heap to file /dumps-share/coredump.7
client_1  | [createdump] Written 556867584 bytes (135954 pages) to core file
client_1  | [createdump] Target process is alive
client_1  | [createdump] Dump successfully written in 737ms

https://dev.azure.com/dnceng-public/public/_build/results?buildId=525074&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

Crash dump attached. We should look at it if it happens again.
DumpsLinux.zip

SslStress on main

release/8.0 branch

HttpStress on release/8.0

  • No failures on Linux
  • Last run: #20240118.2

SslStress on release/8.0

  • No failures on Linux
  • Last run: 20240118.2

@antonfirsov
Copy link
Member

Report 2024-02-15

Looking Linux runs since 2024-01-18. Windows builds are broken across all branches, see #95750.

main (9.0) branch

HttpStress on main

client_1  | malloc(): mismatching next->prev_size (unsorted)
client_1  | Aborted (core dumped)
httpstress_client_1 exited with code 134

This looks very similar to the malloc crash detected in the previous run. Waiting for one more occurence to open an issue.

Crash dump:
DumpsLinux (1).zip

  • Last run: 20240215.2

SslStress on main

release/8.0 branch

HttpStress on release/8.0

SslStress on release/8.0

  • No failures on Linux
  • Last run: 20240215.1

@antonfirsov
Copy link
Member

antonfirsov commented Feb 29, 2024

Report 2024-02-29

Looking Linux runs since 2024-01-15. Windows builds are broken across all branches, see #95750.

main (9.0) branch

HttpStress on main

  • 1 x H3_REQUEST_INCOMPLETE with QuicException: Stream aborted by peer (269) on Linux:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=580954&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_INCOMPLETE' (0x10d). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (269).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 578
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 633
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 665
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 266
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 997

⬆️ this is a new error, we should open an issue if it happens again.

SslStress on main

  • No failures on Linux
  • Last run: 20240229.3

release/8.0 branch

HttpStress on release/8.0

SslStress on release/8.0

  • No failures on Linux
  • Last run: 20240229.2

@antonfirsov
Copy link
Member

@antonfirsov
Copy link
Member

Report 2024-03-26

Looking Linux runs since 2024-03-19. Windows runs are up on main since Sunday.

main (9.0) branch

HttpStress on main

SslStress on main

release/8.0 branch

Windows builds are still broken, #100145 needs a backport.

HttpStress on release/8.0

SslStress on release/8.0

  • No failures on Linux

@antonfirsov
Copy link
Member

@antonfirsov
Copy link
Member

Report 2024-04-09

Looking at runs since 2024-04-02.

main (9.0) branch

HttpStress on main

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

antonfirsov commented Apr 26, 2024

Report 2024-04-26

Looking at runs since 2024-04-09.

main (9.0) branch

HttpStress on main

The extreme amount of hits for #76183 became lower, and hits for #100534 ceased, however we are seeing a couple of failures similar to #76183 accumulating. Let's see if the frequency changes before opening separate issues.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_FRAME_ERROR' (0x106). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.SkipUnknownPayloadAsync(Int64 payloadLength, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1080
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 848
   at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1284
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1175
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1258
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372

https://dev.azure.com/dnceng-public/public/_build/results?buildId=651163&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_REJECTED' (0x10b). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 580
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 635
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 308
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1208
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1247
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 106
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 542
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 288
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 311
  • 2 x HttpIOException with Unknown error code in POST Duplex Slow on Windows

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpIOException: An error occurred while sending the request. (Unknown)
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 615
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 638
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 419
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 481
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1268
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

antonfirsov commented May 9, 2024

Report 2024-05-09

Looking at runs since 2024-04-26.

main (9.0) branch

HttpStress on main

SslStress on main

  • There was a temporary issue with dotnet-install that reduced the number of Windows runs, but it is gone now.
  • No failures otherwise

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-05-14

Looking at runs since 2024-05-09.

main (9.0) branch

HttpStress on main

#102085 switched the Linux runs to use msquic v2.3.5 instead of their main branch which fixed #101625.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=674960&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Failure Type 1/2:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_ID_ERROR' (0x108). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 845
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 301
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 87
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 430
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_10>d.MoveNext() in C:\app\ClientOperations.cs:line 408
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-05-21

Looking at runs since 2024-05-14.

main (9.0) branch

HttpStress on main

SslStress on main

  • 1 x transient build failure
  • No failures otherwise

release/8.0 branch

HttpStress on release/8.0

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-05-30

Looking at runs since 2024-05-21.

main (9.0) branch

HttpStress on main

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-06-06

Looking at runs since 2024-05-30.

main (9.0) branch

HttpStress on main

SslStress on main

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-06-11

Looking at runs since 2024-06-06.

main (9.0) branch

HttpStress on main

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@ManickaP
Copy link
Member Author

2 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue in the same run on Windows, similar to the failure in the #42211 (comment), but this time in POST Duplex Slow and GET Parameters

Looks like: #98220, Ahmet has a PR up for this in: #103081, but I see it happens in reading the content after SendAsync, so we should address that as well.

@antonfirsov
Copy link
Member

@antonfirsov antonfirsov modified the milestones: 9.0.0, 10.0.0 Jun 25, 2024
@antonfirsov
Copy link
Member

@antonfirsov
Copy link
Member

antonfirsov commented Jul 10, 2024

Report 2024-07-10

Looking at runs since 2024-06-25.

main (9.0) branch

HttpStress on main

We assume these issues got fixed with an upstream update starting with 2024-07-09.

SslStress on main

  • No failures

release/8.0 branch

HttpStress on release/8.0

  • No failures

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-09-03

Looking at runs since 2024-08-27.

main (9.0) branch

HttpStress on main

Looking at Linux runs only becaue of #106694.

SslStress on main

  • No failure.

release/8.0 branch

HttpStress on release/8.0

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-09-10

Looking at runs since 2024-09-03.

main branch

HttpStress on main

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_INTERNAL_ERROR' (0x102). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (258).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 607
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 662
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 694
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 323
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 850
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 387
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 214
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 291
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 370
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 323
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 335
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 111
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 434
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 453
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 206

SslStress on main

  • No failure.

release/9.0 branch

HttpStress on release/9.0

SslStress on release/9.0

  • No failures

release/8.0 branch

HttpStress on release/8.0

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

Report 2024-10-08

Looking at runs since 2024-09-10.

main branch

HttpStress on main

SslStress on main

  • No failure.

release/9.0 branch

HttpStress on release/9.0

client_1  |  ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_CLOSED_CRITICAL_STREAM' (0x104). (HttpProtocolError)
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 301
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 370
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 323
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, Activity waitForConnectionActivity, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 335
client_1  |    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 111
client_1  |    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 434
client_1  |    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
client_1  |    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 453
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 206

SslStress on release/9.0

  • No failures

release/8.0 branch

HttpStress on release/8.0

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on release/8.0

  • No failures

@antonfirsov
Copy link
Member

antonfirsov commented Oct 22, 2024

@antonfirsov
Copy link
Member

Report 2024-10-29

Looking at runs since 2024-10-22.

main branch

HttpStress on main

Some issues are still there, regardless of the aspnet fix

SslStress on main

  • No failures

release/9.0 branch

HttpStress on release/9.0

SslStress on release/9.0

  • No failures

release/8.0 branch

HttpStress on release/8.0

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on release/8.0

  • No failures

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants