Unhandled exception from AcceptCompletionCallback

[pepone]

2024-09-18T17:42:06.9787720Z Unhandled exception. System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')
2024-09-18T17:42:06.9788630Z    at System.Net.IPEndPoint.Create(SocketAddress socketAddress)
2024-09-18T17:42:06.9789430Z    at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAccept(SocketAddress remoteSocketAddress)
2024-09-18T17:42:06.9790370Z    at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32 bytesTransferred, SocketFlags flags)
2024-09-18T17:42:06.9791310Z    at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
2024-09-18T17:42:06.9793120Z    at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError)
2024-09-18T17:42:06.9794100Z    at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
2024-09-18T17:42:06.9795710Z    at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-09-18T17:42:06.9796250Z    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-09-18T17:42:06.9798780Z    at System.Threading.Thread.StartCallback()

The exception is similar to the one from #81437, but this might indicate a different problem, exceptions from FinishOperationAccept are not handled.

[rzikm]

This report is not actionable for us without knowing more about the scenario when the exception occurred. Does this exception happen frequently? Are you able to reproduce it with some small code example you could share with us?

[pepone]

This is happening on macOS, when accepting IPv6 TCP connections. I thought there is a bug here because .NET runtime not handling the exception and as a result the callback passed to BeginAccept is not called.

The underlying cause is probably the same as https://github.com/dotnet/runtime/issues/81437 unreliable OS call, and it is not easy to reproduce.

[antonfirsov]

We haven't understood the root cause of #81437. In fact, it has likely stopped occurring in our CI or the frequency is so low, it's below of our radar.

You haven't answered the questions from #108026 (comment). Does the error happen sporadically or predictably? Any code you can possibly share? Any chance for creating a self-contained repro?

This is happening on macOS, when accepting IPv6 TCP connections.

Are you using DualMode listener sockets (created by new Socket(SocketType.Stream, ProtocolType.Tcp)? Does the issue disappear if you change it to be IPv6-only (created by new Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp)?

[wfurt]

I think the underlying issue is same as #102663. Would you be able to run custom code @pepone ? Perhaps we can come up with more instrumentation and/or improved error handling. If that is feasible you can grab binaries from PR or you could rebuild the code changes yourself.

[pepone]

Would you be able to run custom code @pepone ?

Sure, I can try with some custom code.

[wfurt]

can you rebuild #108334 and give it shot @pepone ? The PR should also create binaries if that is easier....

[pepone]

@wfurt I hit #104105 when trying to build your branch with Xcode 16, Where can I download the binaries from your PR for macOS ARM64?

[rzikm]

@pepone you should be able to download the OSX ARM64 build artifacts (Debug) from https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/820467/artifacts?artifactName=libraries_bin_osx_x64_Debug&api-version=7.1&%24format=zip. you need only the System.Net.Sockets.dll from the testhost/shared/... directory.

In case you have problems accessing the above url, I attached the relevant files here directly
macAccept.zip

Publish app as self-contained and overwrite the System.Net.Sockets.dll in the publish output to test the change.

[pepone]

With this PR I hit an assert instead of getting the unhandled exception:

testing connection closure... Process terminated. Assertion Failed
   at System.Net.Sockets.SocketAsyncEventArgs.CompleteAcceptOperation(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError)
   at System.Net.Sockets.SocketAsyncContext.AcceptOperation.InvokeCallback(Boolean allowPooling)
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

To make this easy to reproduce I run my test while I also stress the CPU with stress -c 10.