Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Net.Security.Tests failed with System.IO.IOException #52362

Closed
VincentBu opened this issue May 6, 2021 · 15 comments
Closed

System.Net.Security.Tests failed with System.IO.IOException #52362

VincentBu opened this issue May 6, 2021 · 15 comments
Labels
arch-arm64 area-System.Net.Security JitStress CLR JIT issues involving JIT internal stress modes os-windows test-run-core Test failures in .NET Core test runs
Milestone

Comments

@VincentBu
Copy link
Contributor

Run: runtime-coreclr libraries-jitstress 20210505.1

Failed tests:

net6.0-windows-Release-arm64-CoreCLR_checked-no_tiered_compilation-Windows.10.Arm64v8.Open
 -System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3)
 -System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3)

Error message:

System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.


Stack trace
   at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|181_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 953
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 456
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 358
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncSslHelper(EncryptionPolicy encryptionPolicy, SslProtocols clientSslProtocols, SslProtocols serverSslProtocols, RemoteCertificateValidationCallback certificateCallback) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 155
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(SslProtocols serverProtocol) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 102
--- End of stack trace from previous location ---
@VincentBu VincentBu added arch-arm64 area-System.Net.Security os-windows JitStress CLR JIT issues involving JIT internal stress modes labels May 6, 2021
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label May 6, 2021
@ghost
Copy link

ghost commented May 6, 2021

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-coreclr libraries-jitstress 20210505.1

Failed tests:

net6.0-windows-Release-arm64-CoreCLR_checked-no_tiered_compilation-Windows.10.Arm64v8.Open
 -System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3)
 -System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3)

Error message:

System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.


Stack trace
   at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|181_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 953
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 456
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 358
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncSslHelper(EncryptionPolicy encryptionPolicy, SslProtocols clientSslProtocols, SslProtocols serverSslProtocols, RemoteCertificateValidationCallback certificateCallback) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 155
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(SslProtocols serverProtocol) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 102
--- End of stack trace from previous location ---
Author: VincentBu
Assignees: -
Labels:

JitStress, arch-arm64, area-System.Net.Security, os-windows

Milestone: -

@karelz karelz added this to the Future milestone May 6, 2021
@karelz karelz removed the untriaged New issue has not been triaged by the area owner label May 6, 2021
@VincentBu

This comment has been minimized.

@karelz
Copy link
Member

karelz commented May 20, 2021

Kusto -- methods to query:

  or Method == 'ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success'
  or Method == 'ServerAsyncAuthenticate_EachSupportedProtocol_Success'

Failures 3/18-9/2 (incl. PRs):

Date Build OS Test
5/5 no_tiered_compilation Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
5/19 jitstress2_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
5/22 jitstress2_jitstressregs0x80 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/9 jitstress2_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/13 zapdisable Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/14 no_tiered_compilation Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/15 zapdisable Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/22 jitstress2_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/26 jitstress2_jitstressregs4 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/30 Official run Windows.Server.Core.1909.Amd64.Open ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/4 tailcallstress Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/5 jitstress1 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/8 PR #55298 Windows.10.Amd64.Client21H1.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/11 jitstressregs0x1000 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/21 no_tiered_compilation Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/23 PR #55298 Windows.10.Amd64.Client21H1.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/31 tailcallstress Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/1 jitstressregs0x80 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/4 tailcallstress Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/13 PR #57282 - jitstress1_tiered Windows.10.Arm64v8.Open ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15 jitstress2 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15 defaultpgo Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15 dynamicpgo Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15 nopgo Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/1 jitstress1_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/2 jitstress2 Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/4 jitstress1_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/6 jitstress2_tiered Windows.10.Arm64v8.Open ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success
ServerAsyncAuthenticate_EachSupportedProtocol_Success

It is suspicious that the tests fail only with JitStress and all tests at once, also always together with test failures in #52361 -- perhaps some kind of environmental problem?

@VincentBu

This comment has been minimized.

@VincentBu

This comment has been minimized.

@VincentBu

This comment has been minimized.

@VincentBu

This comment has been minimized.

@VincentBu

This comment has been minimized.

@VincentBu

This comment has been minimized.

@danmoseley
Copy link
Member

danmoseley commented Aug 16, 2021

@BruceForstall could you help hypothesize why this only fails for jitstress? Is this a particularly slow form of jitstress?

@karelz karelz added the test-run-core Test failures in .NET Core test runs label Aug 24, 2021
@BruceForstall
Copy link
Member

BruceForstall commented Aug 30, 2021

@BruceForstall could you help hypothesize why this only fails for jitstress? Is this a particularly slow form of jitstress?

Based on the table above, the test fails in many different types of jitstress. Some are slower (like zapdisable), but many should be much faster than normal, e.g., "no_tiered_compilation" which just says "turn on the JIT optimizer; don't allow tiering". Possibly some timing issue is exacerbated by the code running faster (due to the JIT optimizing the code)?

Note that one problem with most libraries tests is that the default run style is with tiered compilation enabled. That means that the test runs with no JIT optimization until it runs enough times to be "tiered up" / optimized. But most test code never reaches that level. However, the jitstress modes all disable tiering, forcing JIT optimization.

Has anyone tried reproducing these locally?

@VincentBu

This comment has been minimized.

@danmoseley
Copy link
Member

Again coincident with #52361. @karelz probably one could dupe against the other.

@karelz
Copy link
Member

karelz commented Sep 2, 2021

@danmoseley the exceptions are different, so keeping 2 different issues isn't bad. No need to dupe them.

@wfurt
Copy link
Member

wfurt commented Feb 8, 2022

I think this duplicate of #52361. Depending on timing we can surface the server exception but if the client wins, we would see connection close as the server aborted and the exception is still to be processed.

await TestConfiguration.WhenAllOrAnyFailedWithTimeout(t1, t2);

@wfurt wfurt closed this as completed Feb 8, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Mar 11, 2022
@karelz karelz modified the milestones: Future, 7.0.0 Apr 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm64 area-System.Net.Security JitStress CLR JIT issues involving JIT internal stress modes os-windows test-run-core Test failures in .NET Core test runs
Projects
None yet
Development

No branches or pull requests

5 participants