Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various HTTP Stress test failures on the CI #49999

Closed
antonfirsov opened this issue Mar 22, 2021 · 3 comments
Closed

Various HTTP Stress test failures on the CI #49999

antonfirsov opened this issue Mar 22, 2021 · 3 comments
Labels
area-System.Net.Http
Milestone
6.0.0

Comments

@antonfirsov
Copy link
Member

antonfirsov commented Mar 22, 2021 •
edited
Loading

HTTP stress tests are failing on the CI with various Failure Types on both Windows and Linux, although the latter seems to happen less frequently.

I keep trying, but so far I was unable to reproduce the failures locally. Based on CI logs, there are several different Failure Types:

1. Windows + HTTP 1.1 connection timeout in HttpConnectionPool.ConnectToTcpHostAsync:

SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type1-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1045441&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9&l=1244

This is the most common failure, happens on almost every build!

2. Windows / Linux + HTTP 2.0 IOException in Http2Connection.ReadFrameAsync:

IOException, The response ended prematurely while waiting for the next frame from the server
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type2-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=702

Note: looks similar to #1586.

3. Windows + HTTP 2.0 SocketException (10054) in Http2Connection.ProcessIncomingFramesAsync:

SocketException (10054): An existing connection was forcibly closed by the remote host. thrown from SslStream.ReadAsync
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type3-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1390

4. Windows/Linux + HTTP 2.0 timeout in Http2Connection.SendStreamDataAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type4-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1540

5. Windows/Linux + HTTP 2.0 timeout in Http2Stream.WaitFor100ContinueAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type5-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=757
@antonfirsov antonfirsov added this to the 6.0.0 milestone Mar 22, 2021
@ghost
Copy link

ghost commented Mar 22, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

HTTP stress tests are failing on the CI with various Failure Types on both Windows and Linux, although the latter seems to happen less frequently.

I keep trying, but so far I was unable to reproduce the failures locally. Based on CI logs, there are several different Failure Types:

1. Windows + HTTP 1.1 SocketException (10060) in HttpConnectionPool.ConnectToTcpHostAsync:

SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type1-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1045441&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9&l=1244

This is the most common failure, happens on almost every build!

2. Windows / Linux + HTTP 2.0 IOException in Http2Connection.ReadFrameAsync:

IOException, The response ended prematurely while waiting for the next frame from the server
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type2-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=702

Note: looks similar to #1586.

3. Windows + HTTP 2.0 SocketException (10054) in Http2Connection.ProcessIncomingFramesAsync:

SocketException (10054): An existing connection was forcibly closed by the remote host. thrown from SslStream.ReadAsync
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type3-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1390

4. Windows/Linux + HTTP 2.0 timeout in Http2Connection.SendStreamDataAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type4-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1540

5. Windows/Linux + HTTP 2.0 timeout in Http2Stream.WaitFor100ContinueAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type5-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=757

Author: antonfirsov
Assignees: -
Labels:

area-System.Net.Http
Milestone: 6.0.0

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 22, 2021
@antonfirsov antonfirsov mentioned this issue Mar 22, 2021
Merged
@ManickaP
Copy link
Member

Please be aware of this issue: #42211
Some of the problems you're hitting are known and filed.

@antonfirsov
Copy link
Member Author

@ManickaP thanks, totally missed that. I think #42211 mostly covers everything, so I'm closing this as a duplicate.

What (probably) changed since September is that System.Net.Sockets.SocketException (10060) became a dominant failure on Windows, at least on CI.

@antonfirsov antonfirsov mentioned this issue Mar 22, 2021
Open
@ghost ghost locked as resolved and limited conversation to collaborators Apr 21, 2021
@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Oct 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net.Http
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
3 participants
@karelz @antonfirsov @ManickaP