AIO #1

adamsitnik · 2020-02-20T16:10:17Z

No description provided.

…andle 0 reads using it

…r they are layout it memory

This adds support in the JIT emitter for Vector Load/Store structure instructions (C3.2.10 - Arm Architecture Reference Manual): - LD1 (1-4 registers) - LD2 - LD3 - LD4 - LD1R - LD2R - LD3R - LD4R - ST1 (1-4 registers) - ST2 - ST3 - ST4 in the following addressing modes: - Base register only - Post-indexed by a 64-bit register - Post-indexed by an immediate, equal to the number of bytes transferred Also adds support in JitDump for printing of * A SIMD vector register list. For example, ld1 {v5.16b, v6.16b, v7.16b, v8.16b}, [x9] * A SIMD vector element list. For example, st1 {v0.b}[3], [x1],#1

adamsitnik · 2020-04-09T15:25:42Z

src/libraries/Common/src/Interop/Unix/System.Native/Interop.Aio.cs

+
+            // these fields swap for big endian
+            // https://github.com/torvalds/linux/blob/0a679e13ea30f85a1aef0669ee0c5a9fd7860b34/include/uapi/linux/aio_abi.h#L77-L83
+            private uint _swappedField_1;


this is a little bit tricky: Linux Kernel has different order of these two fields depending on the endianess...

I've encapsulated that my making them private and adding internal properties with some simple logic that handles the endianess

I wouldn't bother too much for fields that aren't used.

adamsitnik · 2020-04-09T15:26:48Z

src/libraries/Native/Unix/System.Native/pal_networking.c

@@ -56,6 +56,10 @@
 #if HAVE_LINUX_CAN_H
 #include <linux/can.h>
 #endif
+#if HAVE_LINUX_AIO
+#include <linux/aio_abi.h>


important: the AIO is available on Linux, but not on all Unixes (like MacOS), hence I needed this switch

AIO is Linux only. No other OS will have this. Using #if defined(__linux) is also an option.

I guess we do not support kernels where linux/aio_abi.h is not available, right?

adamsitnik · 2020-04-09T15:29:13Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

-                    return false;
-                }
-
-                Debug.Assert(oldState == (int)State.Waiting);


I wanted to make sure that we NEVER run into the situation where we set to running a non-waiting operation (Debug.Assert is not preventing this for Release builds)

adamsitnik · 2020-04-09T15:29:45Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

@@ -310,6 +310,110 @@ public void DoAbort()

            public abstract void InvokeCallback(bool allowPooling);

+            internal virtual bool TryBatch(SocketAsyncContext context, int id, ref Interop.Sys.IoControlBlock ioControlBlock) => false;


minor: Using ref to keep it both safe and fast

adamsitnik · 2020-04-09T15:30:01Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

@@ -310,6 +310,110 @@ public void DoAbort()

            public abstract void InvokeCallback(bool allowPooling);

+            internal virtual bool TryBatch(SocketAsyncContext context, int id, ref Interop.Sys.IoControlBlock ioControlBlock) => false;
+
+            internal virtual void HandleBatchEvent(in Interop.Sys.IoEvent ioControlBlock)


minor: using in to avoid expensive struct copies on Linux

It only works for readonly structs AFAIK.

adamsitnik · 2020-04-09T15:31:16Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

+                        return SocketError.AddressNotAvailable;
+                    case Interop.Error.EAFNOSUPPORT:
+                        return SocketError.AddressFamilyNotSupported;
+                    case (Interop.Error)11:


important: this method requires a rewrite - AIO is Linux system call and it returns different error codes that what we have in Interop.Error. A good example is Interop.Error.EAGAIN which is not 11 (while here it must be 11)

It would make sense for this to be in the pal IoGetEvents. The errnos returned can be mapped to SocketError in the IoEvent.

@tmds very good idea!

@adamsitnik @tmds my understanding is that none of these errors codes are mapped correctly.
io_event.Res will contain Linux error codes which have nothing to do with the pal_errno.h codes starting with 0x1. I'm planning to map them in IoGetEvents as suggested, reusing the common error mapping logic.
Please correct me if my understanding is wrong!

Note: currently I'm debugging a case of ESPIPE (29 - "Illegal seek").

Please correct me if my understanding is wrong!

@antonfirsov You are correct

adamsitnik · 2020-04-09T15:32:03Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

-            void IThreadPoolWorkItem.Execute() => AssociatedContext.ProcessAsyncReadOperation(this);
+            void IThreadPoolWorkItem.Execute()
+            {
+                if (IsCompleted)


for batched operations, we dispatch them to thread pool to more or less only run continuation on thread pool thread

adamsitnik · 2020-04-09T15:39:12Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

@@ -136,6 +140,8 @@ public bool TryRegister(SafeSocketHandle socket, out Interop.Error error)
        //
        private bool IsFull { get { return _nextHandle == MaxHandles; } }

+        private bool IsAio => _aioContext.Ring != null;


this might need a cleaner check or even moving entire AIO logic to a standalone type

adamsitnik · 2020-04-09T15:40:20Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

+                }
+                else
+                {
+                    Debug.Assert(Interop.Sys.IsAioSupported() == false, "When AIO is supported IoSetup should always succeed");


important: IoSetup might fail - on MacOS the API is simply missing, on Linux the user might not have enough of resources for the method to succeed

I added this check to make sure that our Debug tests on Linux test AIO for real

adamsitnik · 2020-04-09T15:41:50Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

                        {
-                            shutdown = true;
+                            // there was an error, we use the non-buffered execution path (this should be very rare)
+                            context.HandleEvents(socketEvent.Events);


important: epoll can return read, write or error. If it's error, we fall back to non-AIO code path

adamsitnik · 2020-04-09T15:43:15Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

+                    }
+
+                    // todo: perf: avoid the syscall by using a well known pattern that reads from the ring
+                    result = Interop.Sys.IoGetEvents(_aioContext.Ring, batchedCount, batchedCount, _aioEvents);


important: it's possible to avoid this sys call like @tmds does that in https://github.com/tmds/Tmds.LinuxAsync/blob/11233fc8fc80862533e319f4d22855347981aace/src/Tmds.LinuxAsync/EPollAsyncEngine.LinuxAio.cs#L203

it was not hot on the profile, so it was sitting on my todo list

adamsitnik · 2020-04-09T15:51:24Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

                    {
-                        throw new InternalException(err);
+                        shutdown = true;


the shutdown is implemented in a very interesting way: the epoll does not support canceling, but afaik we have some pipe where we write cancel request, which is represented by a file descriptior that is monitored by epool itself. So if there is new data in it, epoll receives a notification and we stop here.

adamsitnik · 2020-04-09T15:57:27Z

@antonfirsov I've added all the comments. If it makes it any easier for you, please feel free to squash all the commits and|or rewrite git history (there are some commits of shame here :D )

tmds · 2020-04-14T02:48:35Z

src/libraries/Common/src/Interop/Unix/System.Native/Interop.Aio.cs

+    internal static partial class Sys
+    {
+        [StructLayout(LayoutKind.Sequential)]
+        internal struct IoEvent


If you add structs that exactly match native structs, it would be nice to validate this with a shim struct and some asserts.
like is done here: https://github.com/dotnet/runtime/blob/6e83eafe49ce3c55e9f68ae669f24957e5b482f8/src/libraries/Native/Unix/System.Native/pal_networking.h#L290-L296, https://github.com/dotnet/runtime/blob/6e83eafe49ce3c55e9f68ae669f24957e5b482f8/src/libraries/Native/Unix/System.Native/pal_networking.c#L160-L165

@tmds you rather mean the C struct here I guess?

tmds · 2020-04-14T02:50:43Z

src/libraries/Common/src/Interop/Unix/System.Native/Interop.Aio.cs

+        }
+
+        [DllImport(Libraries.SystemNative, EntryPoint = "SystemNative_IsAioSupported")]
+        internal static extern bool IsAioSupported();


This can be implicitly done based on IoSetup return value.

tmds · 2020-04-14T03:08:36Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

@@ -362,6 +488,7 @@ private abstract class SendOperation : WriteOperation
        private sealed class BufferMemorySendOperation : SendOperation
        {
            public Memory<byte> Buffer;
+            internal MemoryHandle PinHandle;


These handles are only used while performing the operation against AIO. They could be in a List<MemoryHandle> that is populated during the use of AIO, instead of per operation.

With a single handle, multi buffer sends/receives can't be performed using AIO.

tmds · 2020-04-14T03:11:18Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

+
+                ioControlBlock.AioData = (ulong)id;
+                ioControlBlock.AioLioOpcode = Interop.Sys.IoControlBlockFlags.IOCB_CMD_PWRITE;
+                ioControlBlock.AioFildes = (uint)context._socket.DangerousGetHandle().ToInt32();


The handle reference count isn't incremented.

tmds · 2020-04-14T03:13:29Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

+                {
+                    ErrorCode = Map((Interop.Error)(-ioControlBlock.Res));
+
+                    if (CanRetry(ErrorCode))


It would be good to handle EINTR too.
I'm not sure if it can happen, but better be safe than sorry.
In case of EINTR, the operation should be tried again.

tmds · 2020-04-14T03:23:55Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

+                            _state = QueueState.Processing;
+                            nextOperation = _tail.Next;
+
+                            while (batchedCount < ioControlBlocks.Length && nextOperation.TryBatch(context, batchedCount, ref ioControlBlocks[batchedCount]))


Consider batching a single operation here.

tmds · 2020-04-14T03:35:17Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

+                    ReadOnlySpan<Interop.Sys.IoEvent> events = new ReadOnlySpan<Interop.Sys.IoEvent>(_aioEvents, batchedCount);
+                    for (int i = 0; i < events.Length; i++)
+                    {
+                        batchedOperations[events[i].Data].HandleBatchEvent(in events[i]);


Is there some code path that ensures other operations that are pending in the queue get executed?

Tom, I am not sure if I understand. We batch what we can and dispatch the rest, then here we handle only what was batched.

dispatch the rest

We can't dispatch the rest. The operations need to be handled sequentially.

We also need to take care of new operations being added to the queue while the batched operations are executing. So at the end of HandleBatchEvent we must check the queue.

antonfirsov

I resolved the conflicts in my branch, and currently in a phase of trying it out, will come back with more questions later.

antonfirsov · 2020-04-16T13:25:57Z

src/libraries/Common/src/Interop/Unix/System.Native/Interop.Aio.cs

+    internal static partial class Sys
+    {
+        [StructLayout(LayoutKind.Sequential)]
+        internal struct IoEvent


@tmds you rather mean the C struct here I guess?

antonfirsov · 2020-04-16T15:14:45Z

src/libraries/Native/Unix/System.Native/pal_networking.c

@@ -56,6 +56,10 @@
 #if HAVE_LINUX_CAN_H
 #include <linux/can.h>
 #endif
+#if HAVE_LINUX_AIO
+#include <linux/aio_abi.h>


I guess we do not support kernels where linux/aio_abi.h is not available, right?

antonfirsov · 2020-04-16T15:18:50Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

@@ -310,6 +310,110 @@ public void DoAbort()

            public abstract void InvokeCallback(bool allowPooling);

+            internal virtual bool TryBatch(SocketAsyncContext context, int id, ref Interop.Sys.IoControlBlock ioControlBlock) => false;
+
+            internal virtual void HandleBatchEvent(in Interop.Sys.IoEvent ioControlBlock)


It only works for readonly structs AFAIK.

adamsitnik · 2020-04-20T09:53:13Z

I guess we do not support kernels where linux/aio_abi.h is not available, right?

The file is in the Unix folder, I assumed that it can be compiled for macOS

…tnet#52769) Transition to GC Unsafe mode on every MONO_RT_EXTERNAL_ONLY function in reflection.c In particular, fix mono_reflection_type_from_name which is used in https://github.com/xamarin/xamarin-android/blob/681887ebdbd192ce7ce1cd02221d4939599ba762/src/monodroid/jni/embedded-assemblies.cc#L350 Fixes stack traces like ``` 05-14 08:06:12.848 31274 31274 F DEBUG : #00 pc 00000b99 [vdso] (__kernel_vsyscall+9) 05-14 08:06:12.848 31274 31274 F DEBUG : #1 pc 0005ad68 /apex/com.android.runtime/lib/bionic/libc.so (syscall+40) (BuildId: 6e3a0180fa6637b68c0d181c343e6806) 05-14 08:06:12.848 31274 31274 F DEBUG : #2 pc 00076511 /apex/com.android.runtime/lib/bionic/libc.so (abort+209) (BuildId: 6e3a0180fa6637b68c0d181c343e6806) 05-14 08:06:12.848 31274 31274 F DEBUG : #3 pc 0002afcd /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonodroid.so (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+141) (BuildId: 9726f32ad5f8fa5e7c5762baf2f6e3294da41cc1) 05-14 08:06:12.848 31274 31274 F DEBUG : #4 pc 00112c5d /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (eglib_log_adapter+141) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #5 pc 00020fdf /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (monoeg_g_logv+175) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #6 pc 0002113a /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (monoeg_g_log+42) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #7 pc 00128892 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_threads_transition_do_blocking+258) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #8 pc 0012a406 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_threads_enter_gc_safe_region_unbalanced_with_info+134) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #9 pc 0012a27e /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_threads_enter_gc_safe_region_internal+46) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #10 pc 000799a7 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_loader_lock+71) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #11 pc 000447a1 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_class_create_from_typedef+129) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #12 pc 0003c073 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_class_get_checked+99) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #13 pc 0003cc0f /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_class_from_name_checked_aux+735) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #14 pc 00037989 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_class_from_name_checked+73) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #15 pc 000cc5f4 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_reflection_get_type_internal+132) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : #16 pc 000c9bce /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_reflection_get_type_with_rootimage+126) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#17 pc 000ca204 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (_mono_reflection_get_type_from_info+292) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#18 pc 000ca06e /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_reflection_type_from_name_checked+334) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#19 pc 000c9f01 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonosgen-2.0.so (mono_reflection_type_from_name+49) (BuildId: b67e93dd750dafdd6f65f408b021b6a3a74868ac) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#20 pc 0001b40b /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonodroid.so (xamarin::android::internal::EmbeddedAssemblies::typemap_java_to_managed(char const*)+427) (BuildId: 9726f32ad5f8fa5e7c5762baf2f6e3294da41cc1) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#21 pc 0001b551 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonodroid.so (xamarin::android::internal::EmbeddedAssemblies::typemap_java_to_managed(_MonoString*)+113) (BuildId: 9726f32ad5f8fa5e7c5762baf2f6e3294da41cc1) 05-14 08:06:12.849 31274 31274 F DEBUG : dotnet#22 pc 000211a7 /data/app/~~rMrkpKmVPaBpM5jKb8fPAg==/com.microsoft.maui-JfRo8RWSDJaNtJuBa0y7_Q==/lib/x86/libmonodroid.so (xamarin::android::internal::MonodroidRuntime::typemap_java_to_managed(_MonoString*)+39) (BuildId: 9726f32ad5f8fa5e7c5762baf2f6e3294da41cc1) ```

…2915) * [build] Define NO_UNALIGNED_ACCESS for 32-bit arm platforms Possibly related to crashes on Android like this: ``` 05-18 10:59:07.466 17076 17076 F libc : Fatal signal 7 (SIGBUS), code 1 (BUS_ADRALN), fault addr 0xb9c95a41 in tid 17076 (simplehellomaui), pid 17076 (simplehellomaui) 05-18 10:59:07.501 17104 17104 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone 05-18 10:59:07.502 989 989 I tombstoned: received crash request for pid 17076 05-18 10:59:07.503 17104 17104 I crash_dump32: performing dump of process 17076 (target tid = 17076) 05-18 10:59:07.512 17104 17104 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 05-18 10:59:07.512 17104 17104 F DEBUG : Build fingerprint: 'google/crosshatch/crosshatch:11/RQ2A.210405.005/7181113:user/release-keys' 05-18 10:59:07.512 17104 17104 F DEBUG : Revision: 'MP1.0' 05-18 10:59:07.512 17104 17104 F DEBUG : ABI: 'arm' 05-18 10:59:07.515 17104 17104 F DEBUG : Timestamp: 2021-05-18 10:59:07+0200 05-18 10:59:07.515 17104 17104 F DEBUG : pid: 17076, tid: 17076, name: simplehellomaui >>> com.microsoft.simplehellomaui <<< 05-18 10:59:07.515 17104 17104 F DEBUG : uid: 10364 05-18 10:59:07.515 17104 17104 F DEBUG : signal 7 (SIGBUS), code 1 (BUS_ADRALN), fault addr 0xb9c95a41 05-18 10:59:07.515 17104 17104 F DEBUG : r0 bb4a5cd0 r1 b9c95a49 r2 00000000 r3 e94c7520 05-18 10:59:07.515 17104 17104 F DEBUG : r4 0000000c r5 00000000 r6 ff843c50 r7 ff843e70 05-18 10:59:07.515 17104 17104 F DEBUG : r8 b69547f8 r9 e99eac50 r10 00000000 r11 00000021 05-18 10:59:07.515 17104 17104 F DEBUG : ip e94c74f0 sp ff843c48 lr bb31e0dd pc bb3a4d24 05-18 10:59:07.531 709 709 E Layer : [Surface(name=Task=1)/@0x52e6b1a - animation-leash#0] No local sync point found 05-18 10:59:07.532 709 709 E Layer : [Surface(name=Task=1571)/@0x9c90165 - animation-leash#0] No local sync point found 05-18 10:59:07.706 17104 17104 F DEBUG : backtrace: 05-18 10:59:07.707 17104 17104 F DEBUG : #00 pc 000ddd24 /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mono_method_to_ir+9232) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #1 pc 000d7777 /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (inline_method+622) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #2 pc 000ec0a3 /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mono_method_to_ir+67470) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #3 pc 000cda6d /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mini_method_compile+2264) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #4 pc 000cf413 /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mono_jit_compile_method_inner+50) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #5 pc 000d1d7f /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mono_jit_compile_method_with_opt+1766) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #6 pc 0012d94d /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (common_call_trampoline+832) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #7 pc 0012d5cb /data/app/~~J4DFQ3c1v2YGrEurX7TNjg==/com.microsoft.simplehellomaui-_jGGPiZpZ3yT-QCTNDcgvQ==/lib/arm/libmonosgen-2.0.so (mono_magic_trampoline+62) (BuildId: d0a4e41a500357a621884b64f6ca8533b62a664b) 05-18 10:59:07.707 17104 17104 F DEBUG : #8 pc 0000006a <anonymous:b7986000> ``` * move to host/target sections

…et#53792) I have expanded the PerfMap format produced by Crossgen2 and R2RDump to produce metadata in form of pseudo-symbol records with high addresses. In this version I have implemented four metadata entries - output GUID, target OS, target architecture and perfmap format version number. I have verified for System.Private.CoreLib and for the composite framework that Crossgen2 and R2RDump produce identical metadata. To facilitate a smooth transition to the new perfmap format, in accordance with Juan's suggestion I have introduced a new command-line option to explicitly specify the perfmap format revision. As of today, 0 corresponds to the legacy Crossgen1-style output where the perfmap file name includes the {MVID} section, perfmap format #1 corresponds to current Crossgen2 with its new naming scheme. As of today there are no differences in the file content. Thanks Tomas

…otnet#63598) * Fix native frame unwind in syscall on arm64 for VS4Mac crash report. Add arm64 version of StepWithCompactNoEncoding for syscall leaf node wrappers that have compact encoding of 0. Fix ReadCompactEncodingRegister so it actually decrements the addr. Change StepWithCompactEncodingArm64 to match what MacOS libunwind does for framed and frameless stepping. arm64 can have frames with the same SP (but different IPs). Increment SP for this condition so createdump's unwind loop doesn't break out on the "SP not increasing" check and the frames are added to the thread frame list in the correct order. Add getting the unwind info for tail called functions like this: __ZL14PROCEndProcessPvji: 36630: f6 57 bd a9 stp x22, x21, [sp, #-48]! 36634: f4 4f 01 a9 stp x20, x19, [sp, #16] 36638: fd 7b 02 a9 stp x29, x30, [sp, dotnet#32] 3663c: fd 83 00 91 add x29, sp, dotnet#32 ... 367ac: e9 01 80 52 mov w9, #15 367b0: 7f 3e 02 71 cmp w19, dotnet#143 367b4: 20 01 88 1a csel w0, w9, w8, eq 367b8: 2e 00 00 94 bl _PROCAbort _TerminateProcess: -> 367bc: 22 00 80 52 mov w2, #1 367c0: 9c ff ff 17 b __ZL14PROCEndProcessPvji The IP (367bc) returns the (incorrect) frameless encoding with nothing on the stack (uses an incorrect LR to unwind). To fix this get the unwind info for PC -1 which points to PROCEndProcess with the correct unwind info. This matches how lldb unwinds this frame. Always address module segment to IP lookup list instead of checking the module regions. Strip pointer authentication bits on PC/LR.

* Fix the MacOS remote unwinder for VS4Mac The wrong module was being passed to the remote unwinder because the load bias for shared modules was being calculated incorrectly. Issue: dotnet#63309 * Fix native frame unwind in syscall on arm64 for VS4Mac crash report From PR in main: dotnet#63598 Add arm64 version of StepWithCompactNoEncoding for syscall leaf node wrappers that have compact encoding of 0. Fix ReadCompactEncodingRegister so it actually decrements the addr. Change StepWithCompactEncodingArm64 to match what MacOS libunwind does for framed and frameless stepping. arm64 can have frames with the same SP (but different IPs). Increment SP for this condition so createdump's unwind loop doesn't break out on the "SP not increasing" check and the frames are added to the thread frame list in the correct order. Add getting the unwind info for tail called functions like this: __ZL14PROCEndProcessPvji: 36630: f6 57 bd a9 stp x22, x21, [sp, #-48]! 36634: f4 4f 01 a9 stp x20, x19, [sp, #16] 36638: fd 7b 02 a9 stp x29, x30, [sp, dotnet#32] 3663c: fd 83 00 91 add x29, sp, dotnet#32 ... 367ac: e9 01 80 52 mov w9, #15 367b0: 7f 3e 02 71 cmp w19, dotnet#143 367b4: 20 01 88 1a csel w0, w9, w8, eq 367b8: 2e 00 00 94 bl _PROCAbort _TerminateProcess: -> 367bc: 22 00 80 52 mov w2, #1 367c0: 9c ff ff 17 b __ZL14PROCEndProcessPvji The IP (367bc) returns the (incorrect) frameless encoding with nothing on the stack (uses an incorrect LR to unwind). To fix this get the unwind info for PC -1 which points to PROCEndProcess with the correct unwind info. This matches how lldb unwinds this frame. Always address module segment to IP lookup list instead of checking the module regions. Strip pointer authentication bits on PC/LR.

This adds support for EnC on arm64. A couple of notes on the implementation compared to x64: - On x64 we get the fixed stack size from unwind info. However, for the frames we set up on arm64 for EnC it is not possible to extract the frame size from there because their prologs generally look like stp fp, lr, [sp,#-16]! mov fp, sp sub sp, sp, dotnet#144 with unwind codes like the following: set_fp; mov fp, sp save_fplr_x #1 (0x01); tp fp, lr, [sp, #-16]! As can be seen, it is not possible to get the fixed stack size from unwind info in this case. Instead we pass it through the GC info that already has a section for EnC data. - On arm64 the JIT is required to place the PSPSym at the same offset from caller-SP for both the main function and for funclets. Due to this we try to allocate the PSPSym as early as possible in the main function and we must take some care in funclets. However, this conflicts with the EnC frame header that the JIT uses to place values that must be preserved on EnC transitions. This is currently callee-saved registers and the MonitorAcquired boolean. Before this change we were allocating PSPSym above (before) the monitor acquired boolean, but we now have to allocate MonitorAcquired first, particularly because the size of the preserved header cannot change on EnC transitions, while the PSPSym can disappear or appear. This changes frame allocation slightly for synchronized functions.

These helpers are used to report names of things in warnings. The functional changes are: * For method parameters, use the parameter name if available (and only if not fallback to the #1 notation) * For property accessor methods, use the C# naming scheme, so for example Type.Property.get instead of Type.get_Property. Both of these changes are in preparation to bring NativeAOT closer in behavior to ILLink and the trim analyzers. For this I moved some of the helpers to the common shared code. Some unrelated code cleanup as well. Co-authored-by: Michal Strehovský <[email protected]>

* Initial implementation for contract customization fix build errors Move converter rooting to DefaultJsonTypeInfoResolver so that it can be used standalone Fix ConfigurationList.IsReadOnly Minor refactorings (#1) * Makes the following changes: * Move singleton initialization for DefaultTypeInfoResolver behind a static property. * Consolidate JsonSerializerContext & IJsonTypeInfoResolver values to a single field. * Move reflection fallback logic away from JsonSerializerContext and into JsonSerializerOptions * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.cs * remove testing of removed field Simplify the JsonTypeInfo.CreateObject implemenetation (#2) * Simplify the JsonTypeInfo.CreateObject implemenetation * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs Co-authored-by: Krzysztof Wicher <[email protected]> Co-authored-by: Krzysztof Wicher <[email protected]> Tests and fixes for JsonTypeInfoKind.None TypeInfo type mismatch tests Allow setting NumberHandling on JsonTypeInfoKind.None test resolver returning wrong type of options JsonTypeInfo/JsonPropertyInfo mutability tests rename test file Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver (#3) * Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver * address feedback Add simple test for using JsonTypeInfo<T> with APIs directly taking it fix and tests for untyped/typed CreateObject uncomment test cases, remove todo More tests and tiny fixes Add a JsonTypeInfoResolver.Combine test for JsonSerializerContext (#4) * Fix JsonTypeInfoResolver.Combine for JsonSerializerContext * Break up failing test Fix simple scenarios for combining contexts (#6) * Fix simple scenarios for combining contexts * feedback JsonSerializerContext combine test with different camel casing Remove unneeded virtual calls & branching when accessing Get & Set delegates (#7) JsonPropertyInfo tests everything minus ShouldSerialize & NumberHandling Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs throw InvalidOperationException rather than ArgumentNullException for source gen when PropertyInfo.Name is assigned through JsonPropertyInfoValues tests for duplicated property names and JsonPropertyInfo.NumberHandling Add tests for NumberHandling and failing tests for ShouldSerialize disable the failing test and add extra checks disable remainder of the failing ShouldSerialize tests, fix working one Fix ShouldSerialize and IgnoreCondition interop Add failing tests for CreateObject + parametrized constructors Fix CreateObject support for JsonConstructor types (#10) * Fix CreateObject support for JsonConstructor types * address feedback Make contexts more combinator friendly (#9) * Make contexts more combinator friendly * remove converter cache * redesign test to account for JsonConstructorAttribute * Combine unit tests * address feedback * Add acceptance tests for DataContract attributes & Specified pattern (#11) * Add private field serialization acceptance test (#13) * tests, PR feedback (#14) * PR feedback and extra tests * Shorten class name, remove incorrect check (not true for polimorphic cases) * Make parameter matching for custom properties map property Name with parameter (#16) * Test static initialization with JsonTypeInfo (dotnet#17) * Fix test failures and proper fix this time (dotnet#18) * Fix test failures and proper fix this time * reinstate ActiveIssueAttribute * PR feedback and adjust couple of tests which don't set TypeInfoResolver * fix IAsyncEnumerable tests * Lock JsonSerializerOptions in JsonTypeInfo.EnsureConfigured() Co-authored-by: Eirik Tsarpalis <[email protected]> Co-authored-by: Eirik Tsarpalis <[email protected]>

E.g., Update LSRA "Allocating Registers" table description. Dump nodes added during resolution, e.g.: ``` BB29 bottom (BB08->BB08): move V25 from STK to rdi (Critical) N001 ( 1, 1) [001174] ----------z t1174 = LCL_VAR int V25 cse4 rdi REG rdi ``` Dump more data in the LSRA block sequence data: ``` -BB03( 16 ) -BB04( 4 ) +BB03 ( 16 ) critical-in critical-out +BB04 ( 4 ) critical-out ``` When dumping various flow bitvectors, annotate the bitvectors better: ``` -BB25 in gen out -0000000000000000 -0000000000000003 CSE #1.c -0000000000000003 CSE #1.c +BB25 + in: 0000000000000000 +gen: 0000000000000003 CSE #1.c +out: 0000000000000003 CSE #1.c ``` Dump hoisting bitvectors using the sorting number: ``` - USEDEF (5)={V04 V00 V01 V02 V03} + USEDEF (5)={V00 V01 V02 V03 V04} ``` Also, fix various typos and formatting.

* WIP: add gRPC tests * Fix AOT and trimming * WIP * Implement IncludeNetworkSecurityConfig * Use IncludeNetworkSecurityConfig * Fix gRPC test * Avoid git checkout * Remove unnecessary code * WIP: start working on CI configuration * Remove WinHttpHandler * Fix problem with SSL * Change server host * Setup CI (#1) * Get Docker container building & exported via test build * Changes * Add missing pfx certificate * changes * cleanup Co-authored-by: Simon Rozsival <[email protected]> * Use tls * Update yml * Revert changes to the mono Android sample app * Bump android image version * Bump image version * Enable TLS * Remove hardcoded package versions * Update package versions * Update package versions * Rename pipeline * Move interop tests website dependencies versions to Versions.props * Add cred scan supression for the interop test server private key * Fix licenses * Remove dependencies * Fix path to Versions.props * Remove unnecessary dependency version * Fix building docker image * Change pfx password Co-authored-by: Jo Shields <[email protected]>

* switch to managed thread ID in Lock * fattening the lock * __declspec(selectany) * few tweaks * fairness * more room for thread ids * remove CurrentNativeThreadId * couple fixes * fix win-arm64 build * win-arm64 build , another try * Apply suggestions from code review Co-authored-by: Jan Kotas <[email protected]> * fix after renaming * do not report successful spin if thread has waited * keep extern and undo mangling of tls_CurrentThread in asm * use SyncTable indexer in less perf-sensitive places. * GetNewHashCode just delegate to shared random * Apply suggestions from code review Co-authored-by: Jan Kotas <[email protected]> * unchecked const conversion * some refactoring comments and typos * min number of spins in the backoff * moved CurrentManagedThreadIdUnchecked to ManagedThreadId * Use `-1` to report success and allow using element #1 in the SyncTable * use threadstatic for managed thread ID * check before calling RhGetProcessCpuCount * use 0 as default thread ID Co-authored-by: Jan Kotas <[email protected]>

…tnet#87189) This fixes a startup crash on Big Sur: > error: * Assertion at /Users/runner/work/1/s/src/mono/mono/utils/mono-hwcap-arm64.c:35, condition `res == 0' not met Because sysctl can't find some of these options: $ sysctl hw.optional.armv8_crc32 hw.optional.armv8_crc32: 1 $ sysctl hw.optional.arm.FEAT_RDM sysctl: unknown oid 'hw.optional.arm.FEAT_RDM' $ sysctl hw.optional.arm.FEAT_DotProd sysctl: unknown oid 'hw.optional.arm.FEAT_DotProd' $ sysctl hw.optional.arm.FEAT_SHA1 sysctl: unknown oid 'hw.optional.arm.FEAT_SHA1' $ sysctl hw.optional.arm.FEAT_SHA256 sysctl: unknown oid 'hw.optional.arm.FEAT_SHA256' $ sysctl hw.optional.arm.FEAT_AES sysctl: unknown oid 'hw.optional.arm.FEAT_AES' Full stack trace: * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1 * frame #0: 0x0000010ef37560 libmonosgen-2.0.dylib`monoeg_assertion_message frame #1: 0x0000010ef375cc libmonosgen-2.0.dylib`mono_assertion_message + 32 frame #2: 0x0000010ef40d6c libmonosgen-2.0.dylib`mono_hwcap_arch_init + 544 frame #3: 0x0000010ef54bd8 libmonosgen-2.0.dylib`mono_hwcap_init + 72 frame #4: 0x0000010ee14dc0 libmonosgen-2.0.dylib`parse_optimizations + 52 frame #5: 0x0000010edbed48 libmonosgen-2.0.dylib`mono_init frame #6: 0x0000010ee18968 libmonosgen-2.0.dylib`mono_jit_init_version frame #7: 0x0000010f48a300 libxamarin-dotnet-debug.dylib`xamarin_bridge_initialize + 216 frame #8: 0x0000010f4900a4 libxamarin-dotnet-debug.dylib`xamarin_main + 376

Fixes dotnet#95367. Relevant part of the JitDump: ``` Using `if true` assertions from pred BB02 Assertions in: #1 fgMorphTree BB04, STMT00021 (before) [000070] DA--------- * STORE_LCL_VAR ubyte V10 tmp9 [000057] ----------- \--* CAST int <- ubyte <- int [000006] ----------- \--* EQ int [000004] ----------- +--* LCL_VAR ref V02 tmp1 (last use) [000055] H---------- \--* CNS_INT(h) ref 'Frozen EmptyPartition`1<Int32> object' Assertion prop for index #1 in BB04: [000006] ----------- * EQ int GenTreeNode creates assertion: [000070] DA---+----- * STORE_LCL_VAR ubyte V10 tmp9 In BB04 New Local Constant Assertion: V10 == [0000000000000001], index = #2 fgMorphTree BB04, STMT00021 (after) [000070] DA---+----- * STORE_LCL_VAR ubyte V10 tmp9 [000055] H----+----- \--* CNS_INT(h) int ``` The JitDump is unfinished because the compiler crashes when trying to dump the last line. Clearly, the `CNS_INT` is no longer a handle at that point because we just bashed it to a constant 1.

adamsitnik and others added 21 commits February 11, 2020 08:55

remove duplicated reference

5650f77

add basic AIO methods

51b6a36

setup & destroy aio ring

6cd0d74

make it compile after switching to actual Linux machine

08181a9

implement a very simple batching

656c4c7

remove unnecesary unsafe

6d5bffc

can I handle empty reads like this?

f9d4de7

move TrySetRUnning to TryAsBatch to avoid memory unpinning issues

f6f219f

something works, but very slow

13a95bc

Merge remote-tracking branch 'upstream/master' into bufferSocketReads

771a169

move batching to context, make sure the lock is taken

f652825

change the way io events are handled

f5a6acb

commit what I have to be able to see nice diff on GH

bfad248

revert some changes to make reading the diff easier

f16c0af

all tests pass!

a446329

native AIO API improvements

0ba4dec

aioContext must be initialized before sending to iosetup

8e4ed97

11 is EAGAIN

7352277

IOCB_CMD_NOOP is not ignored by AIO as stated in the docs, we can't h…

fb60f59

…andle 0 reads using it

executing callback on epoll thread was a very bad idea

9cf5405

take advantage of data locality and initialize the fields in the orde…

98214a1

…r they are layout it memory

adamsitnik mentioned this pull request Feb 24, 2020

Suprising lack of AIO in traces tmds/Tmds.LinuxAsync#36

Closed

adamsitnik commented Apr 9, 2020

View reviewed changes

tmds reviewed Apr 14, 2020

View reviewed changes

antonfirsov reviewed Apr 16, 2020

View reviewed changes

adamsitnik closed this Feb 25, 2021

github-actions bot locked and limited conversation to collaborators Jan 10, 2024

		@@ -310,6 +310,110 @@ public void DoAbort()

		public abstract void InvokeCallback(bool allowPooling);

		internal virtual bool TryBatch(SocketAsyncContext context, int id, ref Interop.Sys.IoControlBlock ioControlBlock) => false;

AIO #1

AIO #1

Conversation

adamsitnik commented Feb 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmds Apr 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsitnik Apr 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsitnik commented Apr 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antonfirsov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsitnik commented Apr 20, 2020

tmds Apr 14, 2020 •

edited

Loading

adamsitnik Apr 9, 2020 •

edited

Loading