[Android] `System.Net.Http.Functional.Tests.csproj` fails with SIGABRT in libcoreclr.so

[kotlarmilos]

Description

The System.Net.Http.Functional.Tests.csproj tests fails with SIGABRT in libcoreclr.so.

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1072372&view=logs&j=c9e87f9f-b882-5f07-e8c2-f8cdb4f422a2&t=f555a9c0-ef97-5d93-0768-acf0622a0bcb

Stack trace:

--------- beginning of crash
06-19 17:31:27.433 32316 32346 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 32346 (Thread-27), pid 32316 (unctional.Tests)
06-19 17:31:27.603   317   317 I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstoneProto
06-19 17:31:27.620   824   824 I tombstoned: received crash request for pid 32346
06-19 17:31:27.622   317   317 I crash_dump64: performing dump of process 32316 (target tid = 32346)
06-19 17:31:27.647   317   317 E DEBUG   : failed to read /proc/uptime: Permission denied
06-19 17:31:28.350 32316 32347 W System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates
06-19 17:31:28.350 32316 32347 W System.err: 	at com.android.org.conscrypt.ActiveSession.checkPeerCertificatesPresent(ActiveSession.java:342)
06-19 17:31:28.350 32316 32347 W System.err: 	at com.android.org.conscrypt.ActiveSession.getPeerCertificates(ActiveSession.java:182)
06-19 17:31:28.350 32316 32347 W System.err: 	at com.android.org.conscrypt.ExternalSession.getPeerCertificates(ExternalSession.java:106)
06-19 17:31:28.350 32316 32347 W System.err: 	at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:129)
06-19 17:31:28.350 32316 32347 W System.err: 	at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:31)
06-19 17:31:28.359 32316   320 D DOTNET  : GetJNIEnv: Registering JNI thread detach. env ptr 0x79faad5690. Key: -2147483621
06-19 17:31:28.361 32316   320 D DOTNET  : DetachThreadFromJNI: Detaching thread from JNI
06-19 17:31:28.454 32316   325 I DOTNET  : 	[PASS] System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.ResetResponseStream_FrameReceived_Ignored
06-19 17:31:28.473   317   317 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
06-19 17:31:28.473   317   317 F DEBUG   : Build fingerprint: 'google/sargo/sargo:12/SP2A.220505.008/8782922:user/release-keys'
06-19 17:31:28.473   317   317 F DEBUG   : Revision: 'MP1.0'
06-19 17:31:28.473   317   317 F DEBUG   : ABI: 'arm64'
06-19 17:31:28.473   317   317 F DEBUG   : Timestamp: 2025-06-19 17:31:27.646457240+0700
06-19 17:31:28.473   317   317 F DEBUG   : Process uptime: 0s
06-19 17:31:28.474   317   317 F DEBUG   : Cmdline: net.dot.System.Net.Http.Functional.Tests
06-19 17:31:28.474   317   317 F DEBUG   : pid: 32316, tid: 32346, name: Thread-27  >>> net.dot.System.Net.Http.Functional.Tests <<<
06-19 17:31:28.474   317   317 F DEBUG   : uid: 18891
06-19 17:31:28.474   317   317 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
06-19 17:31:28.474   317   317 F DEBUG   : Abort message: 'Aborting process.
06-19 17:31:28.474   317   317 F DEBUG   : '
06-19 17:31:28.474   317   317 F DEBUG   :     x0  0000000000000000  x1  0000000000007e5a  x2  0000000000000006  x3  0000007921ace8f0
06-19 17:31:28.474   317   317 F DEBUG   :     x4  0000000000000010  x5  0000000000000010  x6  0000000000000010  x7  7f7f7f7f7f7f7fff
06-19 17:31:28.474   317   317 F DEBUG   :     x8  00000000000000f0  x9  38c58ef9a5274b11  x10 0000000000000000  x11 ffffff80fffffbdf
06-19 17:31:28.474   317   317 F DEBUG   :     x12 0000000000000001  x13 0000000000000013  x14 0000007921acd750  x15 0015efd51866d02f
06-19 17:31:28.474   317   317 F DEBUG   :     x16 0000007bcc31c050  x17 0000007bcc2f8db0  x18 0000007920ed6000  x19 0000000000007e3c
06-19 17:31:28.474   317   317 F DEBUG   :     x20 0000000000007e5a  x21 00000000ffffffff  x22 0000007921acf070  x23 0000007921acefb0
06-19 17:31:28.474   317   317 F DEBUG   :     x24 0000007921aceda0  x25 000000796ab62ad8  x26 0000007a2aaf1810  x27 0000000000000000
06-19 17:31:28.474   317   317 F DEBUG   :     x28 000000787c068f0c  x29 0000007921ace970
06-19 17:31:28.474   317   317 F DEBUG   :     lr  0000007bcc2abaa0  sp  0000007921ace8d0  pc  0000007bcc2abacc  pst 0000000000000000
06-19 17:31:28.474   317   317 F DEBUG   : backtrace:
06-19 17:31:28.474   317   317 F DEBUG   :       #00 pc 000000000004facc  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: cd7952cb40d1a2deca6420c2da7910be)
06-19 17:31:28.474   317   317 F DEBUG   :       #01 pc 000000000055bf48  /data/app/~~mrujYj1jzE7T6A8Gnh7KXw==/net.dot.System.Net.Http.Functional.Tests-BQexdHUGj2jsIPYUDPH9vw==/lib/arm64/libcoreclr.so!libcoreclr.so (BuildId: caf343564a88f0057576a28be1974d7963d7f22d)
06-19 17:31:28.474   317   317 F DEBUG   :       #02 pc 000000000055be14  /data/app/~~mrujYj1jzE7T6A8Gnh7KXw==/net.dot.System.Net.Http.Functional.Tests-BQexdHUGj2jsIPYUDPH9vw==/lib/arm64/libcoreclr.so!libcoreclr.so (BuildId: caf343564a88f0057576a28be1974d7963d7f22d)
06-19 17:31:28.474   317   317 F DEBUG   :       #03 pc 00000000004d7878  /data/app/~~mrujYj1jzE7T6A8Gnh7KXw==/net.dot.System.Net.Http.Functional.Tests-BQexdHUGj2jsIPYUDPH9vw==/lib/arm64/libcoreclr.so!libcoreclr.so (BuildId: caf343564a88f0057576a28be1974d7963d7f22d)
06-19 17:31:28.474   317   317 F DEBUG   :       #04 pc 0000000000065a80  /memfd:doublemapper (deleted)

Tasks

• Fix failing tests
• Add it to the set of smoke tests

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[kotlarmilos]

At some point before the failure, a managed exception occurs:

 Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
                                                                                                       at System.Net.SafeDeleteSslContext.WriteToConnection(IntPtr connection, Byte* data, Int32 dataLength)
2023-09-22 14:40:40.505 20540-22454 DOTNET                  net...tem.Net.Http.Functional.Tests  A  Aborting process.

Since the method is marked with [UnmanagedCallersOnly], it is invoked from native code, so the managed stack trace does not show the native caller.

The WriteToConnection method is passed as a function pointer here:

Interop.AndroidCrypto.SSLStreamInitialize(handle, isServer, managedContextHandle, &ReadFromConnection, &WriteToConnection, InitialBufferSize, peerHost);

Could it be that the function pointer WriteToConnection is null? If that were the case, I would expect InitializeSslContext to appear in the stack trace, but it doesn't. This suggests that the crash occurs inside WriteToConnection, possibly because the first argument is null.

Here's the implementation:

[UnmanagedCallersOnly]
private static unsafe void WriteToConnection(IntPtr connection, byte* data, int dataLength)
{
    SafeDeleteSslContext? context = (SafeDeleteSslContext?)GCHandle.FromIntPtr(connection).Target;
    Debug.Assert(context != null);

    var inputBuffer = new ReadOnlySpan<byte>(data, dataLength);

    context._outputBuffer.EnsureAvailableSpace(dataLength);
    inputBuffer.CopyTo(context._outputBuffer.AvailableSpan);
    context._outputBuffer.Commit(dataLength);
}

The connection parameter is cast as:

SafeDeleteSslContext? context = (SafeDeleteSslContext?)GCHandle.FromIntPtr(connection).Target;

This means the connection must be a valid GCHandle to a SafeDeleteSslContext object. If connection is not a valid GCHandle, or if the target has been released, the cast will return null and trigger a NullReferenceException.

@simonrozsival Could this be the root cause? Could the native layer invoke the delegate after the GCHandle has been released?

[simonrozsival]

Could the native layer invoke the delegate after the GCHandle has been released?

It appears to be the case. The GCHandle a weak handle, it's allocated here:

runtime/src/libraries/System.Net.Security/src/System/Net/Security/Pal.Android/SafeDeleteSslContext.cs

Line 230 in 8adea36

IntPtr managedContextHandle = GCHandle.ToIntPtr(GCHandle.Alloc(this, GCHandleType.Weak));

For some reason, the SafeDeleteSslContext is collected while we're still receiving data from the stream. I can help with more thorough investigation of this issue next week if needed.