diff --git a/docs/design/coreclr/botr/README.md b/docs/design/coreclr/botr/README.md index 4c6e6cb5b3bf6a..6fcea274b2870b 100644 --- a/docs/design/coreclr/botr/README.md +++ b/docs/design/coreclr/botr/README.md @@ -30,6 +30,7 @@ Below is a table of contents. - [Mixed Mode Assemblies](mixed-mode.md) - [Guide For Porting](guide-for-porting.md) - [Vectors and Intrinsics](vectors-and-intrinsics.md) +- [Runtime Async Codegen](runtime-async-codegen.md) It may be possible that this table is not complete. You can get a complete list diff --git a/docs/design/coreclr/botr/clr-abi.md b/docs/design/coreclr/botr/clr-abi.md index 906bdbe3f47086..8298fd45b23e1f 100644 --- a/docs/design/coreclr/botr/clr-abi.md +++ b/docs/design/coreclr/botr/clr-abi.md @@ -678,3 +678,19 @@ MyStruct Test2() return default; } ``` + +# Interpreter ABI details + +The interpreter data stack is separately allocated from the normal "thread" stack, and it grows UP. The interpreter execution control stack is allocated on the "thread" stack, as a series of `InterpMethodContextFrame` values that are linked in a singly linked list onto an `InterpreterFrame` which is placed onto the Frame chain of the thread. `InterpMethodContextFrame` structures are always allocated in descending order so that a callee method's associated `InterpMethodContextFrame` is always located lower in memory compared to its caller or the containing `InterpreterFrame`. + +The base stack pointer within a method never changes, but when a function is called in the interpreter it will have a stack pointer which is associated with the set of arguments passed. In effect argument passing is done by giving a portion of the temporary args space of the caller function to the callee. + +All instructions and GC that address the stack pointer are relative to the current stack pointer, which does not move. This requires that implementations of the localloc instruction actually allocate the memory on the heap, and localloc'd memory is not actually tied to the data stack in any way. + +The stack pointer in all interpreter functions is always aligned on a `INTERP_STACK_ALIGNMENT` boundary. Currently this is a 16 byte alignment requirement. + +The stack elements are always aligned to at least `INTERP_STACK_SLOT_SIZE` and never more than `INTERP_STACK_ALIGNMENT` Given that today's implementation sets `INTERP_STACK_SLOT_SIZE` to 8 and `INTERP_STACK_ALIGNMENT` to 16, this implies all data on the stack is either aligned at an 8 or 16 byte alignment. + +Primitive types smaller than 4 bytes are always zero or sign extended to 4 bytes when on the stack. + +When a function is async it will have a continuation return. This return is not done using the data stack, but instead is done by setting the Continuation field in the `InterpreterFrame`. Thunks are responsible for setting/resetting this value as we enter/leave code compiled by the JIT. \ No newline at end of file diff --git a/docs/design/coreclr/botr/runtime-async-codegen.md b/docs/design/coreclr/botr/runtime-async-codegen.md new file mode 100644 index 00000000000000..7705390bb154bb --- /dev/null +++ b/docs/design/coreclr/botr/runtime-async-codegen.md @@ -0,0 +1,129 @@ +# Responsibilities of a code generator for implementing the Runtime Async feature + +This document describes the behaviors that a code generator must conform to to correctly make the runtime async feature work correctly. + +This document is NOT intended to describe the runtime-async feature. That is better described in the runtime-async specification. See (https://github.com/dotnet/runtime/blob/main/docs/design/specs/runtime-async.md). + + + +The general responsibilities of the runtime-async code generator + +1. Wrap the body of Task and ValueTask returning functions in try/finally blocks which set/reset the `ExecutionContext` and `SynchronizationContext`. + +2. Allow the async thunk logic to work. + +3. Generate Async Debug info (Not yet described in this document)f + + + +# Identifying calls to Runtime-Async methods that can be handled by runtime-async + +When compiling a call to a method that might be called in the optimized fashion, recognize the following sequence. + +``` +call[virt] +[ OPTIONAL ] +{ +[ OPTIONAL - Used for ValueTask based ConfigureAwait ] +{ +stloc X; +ldloca X +} +ldc.i4.0 / ldc.i4.1 +call[virt] (The virt instruction is used for ConfigureAwait on a Task based runtime async function) NI_System_Threading_Tasks_Task_ConfigureAwait +} +call One of the functions which matches NI_System_Runtime_CompilerServices_AsyncHelpers_Await +``` + +A search for this sequence is done if Method is known to be async. + +The dispatch to async functions save the `ExecutionContext` on suspension and restore it on resumption via `AsyncHelpers.CaptureExecutionContext` and `AsyncHelpers.RestoreExecutionContext` respectively + +If PREFIX_TASK_AWAIT_CONTINUE_ON_CAPTURED_CONTEXT, then continuation mode shall be ContinuationContextHandling::ContinueOnCapturedContext otherwise ContinuationContextHandling::ContinueOnThreadPool. + + + +# Non-optimized pattern + +It is also legal for code to have a simple direct usage of `AsyncHelpers.Await`, `AsyncHelpers.AwaitAwaiter`, `AsyncHelpers.UnsafeAwaitAwaiter` or `AsyncHelpers.TransparentAwait`. To support this these functions are marked as async even though they do not return a Task/ValueTask. + +Like other async calls the dispatch to these functions will save and restore the execution context on suspension/resumption. + +The dispatch to these functions will set continuation mode to ContinuationContextHandling::None + +# Calli of an async function + +The dispatch to these functions will save and restore the execution context only on async dispatch. + +# The System.Runtime.CompilerServices.AsyncHelpers::AsyncSuspend intrinsic + +When encountered, triggers the function to suspend immediately, and return the passed in Continuation. + +# Saving and restoring of contexts + +Capture the execution context before the suspension, and when the function resumes, call `AsyncHelpers.RestoreExecutionContext`. The context should be stored into the Continuation. The context may be captured by calling `AsyncHelpers.CaptureExecutionContext`. + +# ABI for async function handling + +There is an additional argument which is the Continuation. When calling a function normally, this is always set to 0. When resuming, this is set to the Continuation object. There is also an extra return argument. It is either 0 or a Continuation. If it is a continuation, then the calling function needs to suspend (if it is an async function), or generate a Task/ValueTask (if it is a async function wrapper). + +## Suspension path + +This is what is used in calls to async functions made from async functions. + +``` +bool didSuspend = false; // Needed for the context restore + +(result, continuation) = call func(NULL /\* Continuation argument \*/, args) +if (continuation != NULL) +{ + // Allocate new continuation + // Capture Locals + // Copy resumption details into continuation (Do things like call AsyncHelpers.CaptureContinuationContext or AsyncHelpers.CaptureExecutionContext as needed) + // Chain to continuation returned from called function + // IF in a function which saves the exec and sync contexts, and we haven't yet suspended, restore the old values. + // return. + + // Resumption point + + // Copy values out of continuation (including captured sync context and execution context locals) + // If the continuation may have an exception, check to see if its there, and if it is, throw it. Do this if CORINFO\_CONTINUATION\_HAS\_EXCEPTION is set. + // If the continuation has a return value, copy it out of the continuation. (CORINFO\_CONTINUATION\_HAS\_RESULT is set) +} +``` + +## Thunks path + +This is what is used in non-async functions when calling an async function. Generally used in the AsyncResumptionStub and in the Task returning thunk. +``` +(result, continuation) = call func(NULL /\* Continuation argument \*/, args) +place result onto IL evaluation stack +Place continuation into a local for access using the StubHelpers.AsyncCallContinuation() helper function. +``` + +Implement an intrinsic for StubHelpers.AsyncCallContinuation() which will load the most recent value stored into the continuation local. + +# Behavior of ContinuationContextHandling + +This only applies to calls which where ContinuationContextHandling is not ContinuationContextHandling::None. + +If set to ContinuationContextHandling::ContinueOnCapturedContext + +- The Continuation shall have an allocated data member for the captured context, and the CORINFO_CONTINUATION_HAS_CONTINUATION_CONTEXT flag shall be set on the continuation. + +- The Continuation will store the captured synchronization context. This is done by calling `AsyncHelpers.CaptureContinuationContext(ref newContinuation.ContinuationContext, ref newContinuation.Flags)` while filling in the `Continuation`. + +If set to ContinuationContextHandling::ContinueOnThreadPool +- The Continuation shall have the CORINFO_CONTINUATION_CONTINUE_ON_THREAD_POOL flag set + +# Exception handling behavior + +If an async function is called within a try block (In the jit hasTryIndex return true), set the CORINFO\_CONTINUATION\_HAS\_EXCEPTION bit on the Continuation and make it large enough. + +# Locals handling + +ByRef locals must not be captured. In fact, we should NULL out any locals which are ByRefs or ByRef-like. Currently we do not do this on synchronous execution, but logically possibly we should. + +# Saving and restoring the synchronization and execution contexts + +The code generator must save/restore the sync and execution contexts around the body of all Task/ValueTask methods when directly called with a null continuation context. The EE communicates when this is necessary with the `CORINFO_ASYNC_SAVE_CONTEXTS` flag returned through `getMethodInfo`. diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs index 54d116e0d20b5f..0045c9e53dbf3d 100644 --- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs +++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs @@ -6,6 +6,7 @@ using System.Diagnostics.CodeAnalysis; using System.Numerics; using System.Reflection; +using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Runtime.Serialization; using System.Runtime.Versioning; @@ -83,7 +84,7 @@ internal enum ContinuationFlags ContinueOnCapturedTaskScheduler = 64, } - // Keep in sync with dataAsyncResumeInfo in the JIT + // Keep in sync with CORINFO_AsyncResumeInfo in corinfo.h internal unsafe struct ResumeInfo { public delegate* Resume; @@ -144,6 +145,18 @@ public ref byte GetResultStorageOrNull() public static partial class AsyncHelpers { +#if FEATURE_INTERPRETER + [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "AsyncHelpers_ResumeInterpreterContinuation")] + private static partial void AsyncHelpers_ResumeInterpreterContinuation(ObjectHandleOnStack cont, ref byte resultStorage); + + internal static Continuation? ResumeInterpreterContinuation(Continuation cont, ref byte resultStorage) + { + ObjectHandleOnStack contHandle = ObjectHandleOnStack.Create(ref cont); + AsyncHelpers_ResumeInterpreterContinuation(contHandle, ref resultStorage); + return cont; + } +#endif + // This is the "magic" method on which other "Await" methods are built. // Calling this from an Async method returns the continuation to the caller thus // explicitly initiates suspension. diff --git a/src/coreclr/inc/corinfo.h b/src/coreclr/inc/corinfo.h index 7cd408af5905bb..9c22b3429f6ae8 100644 --- a/src/coreclr/inc/corinfo.h +++ b/src/coreclr/inc/corinfo.h @@ -1854,7 +1854,13 @@ enum { LCL_FINALLY_MARK = 0xFC }; // FC = "Finally Call" * when it generates code **********************************************************************************/ -typedef void* CORINFO_MethodPtr; // a generic method pointer +#ifdef TARGET_64BIT +typedef uint64_t TARGET_SIZE_T; +#else +typedef uint32_t TARGET_SIZE_T; +#endif + +typedef TARGET_SIZE_T CORINFO_MethodPtr; // a generic method pointer struct CORINFO_Object { @@ -1927,12 +1933,6 @@ struct CORINFO_RefArray : public CORINFO_Object CORINFO_Object* refElems[1]; // actually of variable size; }; -struct CORINFO_RefAny -{ - void * dataPtr; - CORINFO_CLASS_HANDLE type; -}; - // The jit assumes the CORINFO_VARARGS_HANDLE is a pointer to a subclass of this struct CORINFO_VarArgInfo { @@ -1940,6 +1940,18 @@ struct CORINFO_VarArgInfo // (The CORINFO_VARARGS_HANDLE counts as an arg) }; +// Note: Keep synchronized with AsyncHelpers.ResumeInfo +// Any changes to this are an R2R breaking change. Update the R2R verion as needed +struct CORINFO_AsyncResumeInfo +{ + // delegate* + TARGET_SIZE_T Resume; + // Pointer in main code for diagnostics. See comments on + // ICorDebugInfo::AsyncSuspensionPoint::DiagnosticNativeOffset and + // ResumeInfo.DiagnosticIP in SPC. + TARGET_SIZE_T DiagnosticIP; +}; + struct CORINFO_TYPE_LAYOUT_NODE { // Type handle if this is a SIMD type, i.e. for intrinsic types in diff --git a/src/coreclr/interpreter/compiler.cpp b/src/coreclr/interpreter/compiler.cpp index 1887032e93094d..dc0f2109e8bb93 100644 --- a/src/coreclr/interpreter/compiler.cpp +++ b/src/coreclr/interpreter/compiler.cpp @@ -795,11 +795,18 @@ int32_t* InterpCompiler::EmitCodeIns(int32_t *ip, InterpInst *ins, TArrayopcode; int32_t *startIp = ip; + *ip++ = opcode; // Set to true if the instruction was completely reverted. bool isReverted = false; + if (opcode == INTOP_HANDLE_CONTINUATION_SUSPEND) + { + // Capture meaningful start IP for async suspend diagnostics + ((InterpAsyncSuspendData*)GetDataItemAtIndex(ins->data[0]))->resumeInfo.DiagnosticIP = (size_t)startIp; + } + if (opcode == INTOP_SWITCH) { int32_t numLabels = ins->data [0]; @@ -1543,7 +1550,21 @@ void InterpCompiler::getEHinfo(CORINFO_METHOD_INFO* methodInfo, unsigned int ind ehClause->TryOffset = 0; ehClause->TryLength = (uint32_t)m_ILCodeSizeFromILHeader; ehClause->HandlerOffset = (uint32_t)m_synchronizedFinallyStartOffset; - ehClause->HandlerLength = (uint32_t)(m_synchronizedPostFinallyOffset - m_synchronizedFinallyStartOffset); + ehClause->HandlerLength = (uint32_t)(m_synchronizedOrAsyncPostFinallyOffset - m_synchronizedFinallyStartOffset); + ehClause->ClassToken = 0; + return; + } + } + else if (m_isAsyncMethodWithContextSaveRestore) + { + // Logically we append the async finally clause at the end of the EH clauses, as it holds all of the methodbody within its try region, and nested eh clauses are required to be before their containing clauses. + if (index == methodInfo->EHcount) + { + ehClause->Flags = CORINFO_EH_CLAUSE_FINALLY; + ehClause->TryOffset = 0; + ehClause->TryLength = (uint32_t)m_ILCodeSizeFromILHeader; + ehClause->HandlerOffset = (uint32_t)m_asyncFinallyStartOffset; + ehClause->HandlerLength = (uint32_t)(m_synchronizedOrAsyncPostFinallyOffset - m_asyncFinallyStartOffset); ehClause->ClassToken = 0; return; } @@ -1562,6 +1583,10 @@ unsigned int InterpCompiler::getEHcount(CORINFO_METHOD_INFO* methodInfo) { count++; } + else if (m_isAsyncMethodWithContextSaveRestore) + { + count++; + } } return count; } @@ -1741,7 +1766,9 @@ InterpCompiler::InterpCompiler(COMP_HANDLE compHnd, , m_nextCallAsyncContinuationVar(-1) , m_leavesTable(this) , m_dataItems(this) + , m_asyncSuspendDataItems(this) , m_globalVarsWithRefsStackTop(0) + , m_varIntervalMaps(this) #ifdef DEBUG , m_dumpScope(InterpConfig.InterpDump().contains(compHnd, methodInfo->ftn, compHnd->getMethodClass(methodInfo->ftn), &methodInfo->args)) , m_methodName(GetMallocAllocator()) @@ -1791,6 +1818,11 @@ InterpMethod* InterpCompiler::CompileMethod() INTERP_DUMP("Method is synchronized\n"); } + m_isAsyncMethodWithContextSaveRestore = m_methodInfo->options & CORINFO_ASYNC_SAVE_CONTEXTS; + if (m_isAsyncMethodWithContextSaveRestore) + { + INTERP_DUMP("Method is async with context save/restore\n"); + } CreateILVars(); GenerateCode(m_methodInfo); @@ -1940,6 +1972,7 @@ void InterpCompiler::CreateILVars() { bool hasThis = m_methodInfo->args.hasThis(); bool hasParamArg = m_methodInfo->args.hasTypeArg(); + bool hasContinuationArg = m_methodInfo->args.isAsyncCall(); bool hasThisPointerShadowCopyAsParamIndex = false; if (((m_methodInfo->options & CORINFO_GENERICS_CTXT_MASK) == CORINFO_GENERICS_CTXT_FROM_THIS) || (m_isSynchronized && hasThis)) { @@ -1955,7 +1988,7 @@ void InterpCompiler::CreateILVars() // add some starting extra space for new vars m_varsCapacity = m_numILVars + getEHcount(m_methodInfo) + 64; m_pVars = (InterpVar*)AllocTemporary0(m_varsCapacity * sizeof (InterpVar)); - m_varsSize = m_numILVars + hasParamArg + (hasThisPointerShadowCopyAsParamIndex ? 1 : 0) + getEHcount(m_methodInfo); + m_varsSize = m_numILVars + hasParamArg + hasContinuationArg + (hasThisPointerShadowCopyAsParamIndex ? 1 : 0) + (m_isAsyncMethodWithContextSaveRestore ? 2 : 0) + getEHcount(m_methodInfo); offset = 0; @@ -1981,8 +2014,12 @@ void InterpCompiler::CreateILVars() assert(interpType == InterpTypeO); assert(!hasParamArg); // We don't support both a param arg and a this pointer shadow copy m_paramArgIndex = m_numILVars; // The param arg is stored after the IL locals in the m_pVars array + INTERP_DUMP("Set m_paramArgIndex to %d for hasThisPointerShadowCopyAsParamIndex\n", m_paramArgIndex); } - CreateNextLocalVar(hasThisPointerShadowCopyAsParamIndex ? m_paramArgIndex : 0, argClass, interpType, &offset); + + int thisVar = hasThisPointerShadowCopyAsParamIndex ? m_paramArgIndex : 0; + CreateNextLocalVar(thisVar, argClass, interpType, &offset); + INTERP_DUMP("alloc this var(var %d) to offset %d\n", thisVar, m_pVars[thisVar].offset); argIndexOffset++; } @@ -1990,6 +2027,14 @@ void InterpCompiler::CreateILVars() { m_paramArgIndex = m_numILVars; // The param arg is stored after the IL locals in the m_pVars array CreateNextLocalVar(m_paramArgIndex, NULL, InterpTypeI, &offset); + INTERP_DUMP("alloc param var(var %d) to offset %d\n", m_paramArgIndex, m_pVars[m_paramArgIndex].offset); + } + + if (hasContinuationArg) + { + m_continuationArgIndex = (hasParamArg || hasThisPointerShadowCopyAsParamIndex) ? m_numILVars + 1 : m_numILVars; + CreateNextLocalVar(m_continuationArgIndex, NULL, InterpTypeO, &offset); + INTERP_DUMP("alloc continuation var(var %d) to offset %d\n", m_continuationArgIndex, m_pVars[m_continuationArgIndex].offset); } for (int i = argIndexOffset; i < numArgs; i++) @@ -1999,6 +2044,7 @@ void InterpCompiler::CreateILVars() InterpType interpType = GetInterpType(argCorType); sigArg = m_compHnd->getArgNext(sigArg); CreateNextLocalVar(i, argClass, interpType, &offset); + INTERP_DUMP("alloc arg var %d var (var %d) to offset %d\n", i, i, m_pVars[i].offset); } offset = ALIGN_UP_TO(offset, INTERP_STACK_ALIGNMENT); @@ -2013,6 +2059,7 @@ void InterpCompiler::CreateILVars() bool pinned = (argCorTypeWithFlags & CORINFO_TYPE_MOD_PINNED) == CORINFO_TYPE_MOD_PINNED; InterpType interpType = GetInterpType(argCorType); CreateNextLocalVar(index, argClass, interpType, &offset, pinned); + INTERP_DUMP("alloc il local var %d var (var %d) to offset %d\n", i, index, m_pVars[index].offset); sigArg = m_compHnd->getArgNext(sigArg); index++; } @@ -2030,9 +2077,28 @@ void InterpCompiler::CreateILVars() CORINFO_CLASS_HANDLE argClass = m_compHnd->getMethodClass(m_methodInfo->ftn); CreateNextLocalVar(0, argClass, InterpTypeO, &offset); m_shadowThisVar = index; + INTERP_DUMP("alloc shadow this var (var %d) to offset %d\n", m_shadowThisVar, m_pVars[0].offset); index++; // We need to account for the shadow copy in the variable index } + if (hasContinuationArg) + { + assert(index == m_continuationArgIndex); + index++; + } + + if (m_isAsyncMethodWithContextSaveRestore) + { + m_execContextVarIndex = index; + CreateNextLocalVar(index, NULL, InterpTypeO, &offset); + INTERP_DUMP("alloc ExecutableContextVar (var %d) to offset %d\n", m_execContextVarIndex, m_pVars[m_execContextVarIndex].offset); + index++; + m_syncContextVarIndex = index; + CreateNextLocalVar(index, NULL, InterpTypeO, &offset); + INTERP_DUMP("alloc SyncContextVar (var %d) to offset %d\n", m_syncContextVarIndex, m_pVars[m_syncContextVarIndex].offset); + index++; + } + offset = ALIGN_UP_TO(offset, INTERP_STACK_ALIGNMENT); m_ILLocalsSize = offset - m_ILLocalsOffset; @@ -2067,7 +2133,6 @@ void InterpCompiler::CreateNextLocalVar(int iArgToSet, CORINFO_CLASS_HANDLE argC *pOffset = ALIGN_UP_TO(*pOffset, align); m_pVars[iArgToSet].offset = *pOffset; m_pVars[iArgToSet].pinned = pinned; - INTERP_DUMP("alloc arg var %d to offset %d\n", iArgToSet, *pOffset); *pOffset += size; } @@ -2201,10 +2266,10 @@ void InterpCompiler::CreateBasicBlocks(CORINFO_METHOD_INFO* methodInfo) ip++; if (opcode == CEE_RET) { - if (m_isSynchronized && insOffset < m_ILCodeSizeFromILHeader) + if ((m_isSynchronized || m_isAsyncMethodWithContextSaveRestore) && insOffset < m_ILCodeSizeFromILHeader) { - // This is a ret instruction coming from the initial IL of a synchronized method. - CreateLeaveChainIslandBasicBlocks(methodInfo, insOffset, GetBB(m_synchronizedPostFinallyOffset)); + // This is a ret instruction coming from the initial IL of a synchronized or async method. + CreateLeaveChainIslandBasicBlocks(methodInfo, insOffset, GetBB(m_synchronizedOrAsyncPostFinallyOffset)); } } break; @@ -3012,6 +3077,37 @@ bool InterpCompiler::EmitNamedIntrinsicCall(NamedIntrinsic ni, bool nonVirtualCa switch (ni) { + case NI_System_Runtime_CompilerServices_AsyncHelpers_Await: + // AsyncHelpers.Await should never be expanded here. It is simply needed for recognition elsewhere. + return false; + case NI_System_Threading_Tasks_Task_ConfigureAwait: + // ConfigureAwait should never be expanded here. It is simply needed for recognition elsewhere. + return false; + case NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncSuspend: + if (!m_methodInfo->args.isAsyncCall()) + { + BADCODE("AsyncSuspend should only be used in async methods"); + } + if (m_methodInfo->args.retType != CORINFO_TYPE_VOID) + { + BADCODE("AsyncSuspend can only be used in async methods with void return type with today's implementation, it would need to emit the correct INTOP_RET* instruction.."); + } + AddIns(INTOP_SET_CONTINUATION); + m_pLastNewIns->SetSVar(m_pStackPointer[-1].var); + m_pStackPointer--; + AddIns(INTOP_RET_VOID); + return true; + + case NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncCallContinuation: + if (m_methodInfo->args.isAsyncCall()) + { + BADCODE("AsyncCallContinuation should not be used in async methods"); + } + AddIns(INTOP_GET_CONTINUATION); + PushStackType(StackTypeO, NULL); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + return true; + case NI_IsSupported_False: case NI_IsSupported_True: AddIns(INTOP_LDC_I4); @@ -3842,6 +3938,11 @@ static bool DisallowTailCall(CORINFO_SIG_INFO* callerSig, CORINFO_SIG_INFO* call { return true; } + if (callerSig->isAsyncCall()) + { + // Disallow tail calls from async methods for now + return true; + } return false; } @@ -3950,6 +4051,110 @@ void InterpCompiler::EmitCallsiteCallout(CorInfoIsAccessAllowedResult accessAllo } } +static OpcodePeepElement peepRuntimeAsyncCall[] = { + // Call or CallVirt at the start (at offset 0) + { 5, CEE_CALL }, + { 10, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitTask[] = { + // Call or CallVirt at the start (at offset 0) + // LDC_I4_0 or LDC_I4_1 goes here (at offset 5) + { 6, CEE_CALLVIRT}, + { 11, CEE_CALL }, + { 16, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_L_L[] = { + // Call or CallVirt at the start (at offset 0) + { 5, CEE_STLOC}, + { 9, CEE_LDLOCA }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 13 + { 14, CEE_CALL}, + { 19, CEE_CALL }, + { 24, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_S_L[] = { + // Call or CallVirt at the start (at offset 0) + { 5, CEE_STLOC_S }, + { 7, CEE_LDLOCA }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 11) + { 12, CEE_CALL }, + { 17, CEE_CALL }, + { 22, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_L_S[] = { + // Call or CallVirt at the start (at offset 0) + { 5, CEE_STLOC}, + { 9, CEE_LDLOCA_S }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 11) + { 12, CEE_CALL}, + { 17, CEE_CALL }, + { 22, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_S_S[] = { + // Call or CallVirt at the start (at offset 0) + { 5, CEE_STLOC_S}, + { 7, CEE_LDLOCA_S }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 9) + { 10, CEE_CALL}, + { 15, CEE_CALL }, + { 20, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_EXACT_S[] = { + // Call or CallVirt at the start (at offset 0) + // STLOC_0, STLOC_1, STLOC_2, or STLOC_3 goes here (at offset 5) + { 6, CEE_LDLOCA_S }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 8) + { 9, CEE_CALL}, + { 14, CEE_CALL }, + { 19, CEE_ILLEGAL } // End marker +}; + +static OpcodePeepElement peepRuntimeAsyncCallConfigureAwaitValueTask_EXACT_L[] = { + // Call or CallVirt at the start (at offset 0) + // STLOC_0, STLOC_1, STLOC_2, or STLOC_3 goes here (at offset 5) + { 6, CEE_LDLOCA }, + // LDC_I4_0 or LDC_I4_1 goes here (at offset 10) + { 11, CEE_CALL}, + { 16, CEE_CALL }, + { 19, CEE_ILLEGAL } // End marker +}; + +class InterpAsyncCallPeeps +{ + OpcodePeep peepCall = { peepRuntimeAsyncCall, &InterpCompiler::IsRuntimeAsyncCall, &InterpCompiler::ApplyRuntimeAsyncCall, "Call" }; + OpcodePeep peepCallConfigureAwaitTask = { peepRuntimeAsyncCallConfigureAwaitTask, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitTask, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitTask" }; + OpcodePeep peepCallConfigureAwaitValueTask_L_L = { peepRuntimeAsyncCallConfigureAwaitValueTask_L_L, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTask, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_L_L" }; + OpcodePeep peepCallConfigureAwaitValueTask_S_L = { peepRuntimeAsyncCallConfigureAwaitValueTask_S_L, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTask, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_S_L" }; + OpcodePeep peepCallConfigureAwaitValueTask_L_S = { peepRuntimeAsyncCallConfigureAwaitValueTask_L_S, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTask, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_L_S" }; + OpcodePeep peepCallConfigureAwaitValueTask_S_S = { peepRuntimeAsyncCallConfigureAwaitValueTask_S_S, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTask, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_S_S" }; + OpcodePeep peepCallConfigureAwaitValueTask_EXACT_S = { peepRuntimeAsyncCallConfigureAwaitValueTask_EXACT_S, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTaskExactStLoc, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_EXACT_S" }; + OpcodePeep peepCallConfigureAwaitValueTask_EXACT_L = { peepRuntimeAsyncCallConfigureAwaitValueTask_EXACT_L, &InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTaskExactStLoc, &InterpCompiler::ApplyRuntimeAsyncCall, "CallConfigureAwaitValueTask_EXACT_L" }; + +public: + OpcodePeep* Peeps[9] = { + &peepCall, + &peepCallConfigureAwaitTask, + &peepCallConfigureAwaitValueTask_L_L, + &peepCallConfigureAwaitValueTask_S_L, + &peepCallConfigureAwaitValueTask_L_S, + &peepCallConfigureAwaitValueTask_S_S, + &peepCallConfigureAwaitValueTask_EXACT_S, + &peepCallConfigureAwaitValueTask_EXACT_L, + NULL }; + + + bool FindAndApplyPeep(InterpCompiler* compiler) + { + return compiler->FindAndApplyPeep(Peeps); + } +} AsyncCallPeeps; + void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool readonly, bool tailcall, bool newObj, bool isCalli) { uint32_t token = getU4LittleEndian(m_ip + 1); @@ -3970,17 +4175,17 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re m_ip += 5; return; } - else if (token == INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED) + else if (token == INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC) { - if (!m_isSynchronized) - NO_WAY("INTERP_CALL_SYNCHRONIZED_MONITOR_EXIT used in a non-synchronized method"); + if (!m_isSynchronized && !m_isAsyncMethodWithContextSaveRestore) + NO_WAY("INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC used in a non-synchronized or async method"); - INTERP_DUMP("INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED with synchronized ret val var V%d\n", (int)m_synchronizedRetValVarIndex); + INTERP_DUMP("INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC with synchronized ret val var V%d\n", (int)m_synchronizedOrAsyncRetValVarIndex); - if (m_synchronizedRetValVarIndex != -1) + if (m_synchronizedOrAsyncRetValVarIndex != -1) { // If the function returns a value, and we've processed a ret instruction, we'll have a var to load - EmitLoadVar(m_synchronizedRetValVarIndex); + EmitLoadVar(m_synchronizedOrAsyncRetValVarIndex); } else { @@ -3996,6 +4201,62 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re m_ip += 5; return; } + else if (token == INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD) + { + if (!m_isAsyncMethodWithContextSaveRestore) + { + NO_WAY("INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD used in a non-async method"); + } + + AddIns(INTOP_CZERO_I); + m_pLastNewIns->SetSVar(m_continuationArgIndex); + PushInterpType(InterpTypeI4, NULL); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + int32_t isStartedArg = m_pStackPointer[-1].var; + m_pStackPointer--; + + AddIns(INTOP_MOV_P); + m_pLastNewIns->SetSVar(m_execContextVarIndex); + PushInterpType(InterpTypeO, NULL); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + int32_t execContextAddressVar = m_pStackPointer[-1].var; + m_pStackPointer--; + + AddIns(INTOP_MOV_P); + m_pLastNewIns->SetSVar(m_syncContextVarIndex); + PushInterpType(InterpTypeO, NULL); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + int32_t syncContextAddressVar = m_pStackPointer[-1].var; + m_pStackPointer--; + + // Create a new dummy var to serve as the dVar of the call + // FIXME Consider adding special dVar type (ex -1), that is + // resolved to null offset. The opcode shouldn't really write to it + PushStackType(StackTypeI4, NULL); + m_pStackPointer--; + int32_t dVar = m_pStackPointer[0].var; + + CORINFO_ASYNC_INFO asyncInfo; + m_compHnd->getAsyncInfo(&asyncInfo); + + AddIns(INTOP_CALL); + m_pLastNewIns->data[0] = GetMethodDataItemIndex(asyncInfo.restoreContextsMethHnd); + m_pLastNewIns->SetDVar(dVar); + m_pLastNewIns->SetSVar(CALL_ARGS_SVAR); + + m_pLastNewIns->flags |= INTERP_INST_FLAG_CALL; + m_pLastNewIns->info.pCallInfo = (InterpCallInfo*)AllocMemPool0(sizeof (InterpCallInfo)); + int32_t numArgs = 3; + int *callArgs = (int*) AllocMemPool((numArgs + 1) * sizeof(int)); + callArgs[0] = isStartedArg; + callArgs[1] = execContextAddressVar; + callArgs[2] = syncContextAddressVar; + callArgs[3] = CALL_ARGS_TERMINATOR; + m_pLastNewIns->info.pCallInfo->pCallArgs = callArgs; + + m_ip += 5; + return; + } bool isVirtual = (*m_ip == CEE_CALLVIRT); bool isDelegateInvoke = false; @@ -4008,6 +4269,7 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re int callIFunctionPointerVar = -1; void* calliCookie = NULL; + ContinuationContextHandling continuationContextHandling = ContinuationContextHandling::None; if (isCalli) { // Suppress uninitialized use warning. @@ -4027,10 +4289,22 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re callIFunctionPointerVar = m_pStackPointer[-1].var; m_pStackPointer--; calliCookie = m_compHnd->GetCookieForInterpreterCalliSig(&callInfo.sig); + m_ip += 5; } else { - ResolveToken(token, newObj ? CORINFO_TOKENKIND_NewObj : CORINFO_TOKENKIND_Method, &resolvedCallToken); + if (!newObj && m_methodInfo->args.isAsyncCall() && AsyncCallPeeps.FindAndApplyPeep(this)) + { + resolvedCallToken = m_resolvedAsyncCallToken; + continuationContextHandling = m_currentContinuationContextHandling; + } + else + { + ResolveToken(token, newObj ? CORINFO_TOKENKIND_NewObj : CORINFO_TOKENKIND_Method, &resolvedCallToken); + continuationContextHandling = ContinuationContextHandling::None; + m_ip += 5; + } + CORINFO_CALLINFO_FLAGS flags = (CORINFO_CALLINFO_FLAGS)(CORINFO_CALLINFO_ALLOWINSTPARAM | CORINFO_CALLINFO_SECURITYCHECKS | CORINFO_CALLINFO_DISALLOW_STUB); if (isVirtual) @@ -4043,6 +4317,11 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re BADCODE("Vararg methods are not supported in interpreted code"); } + if (continuationContextHandling != ContinuationContextHandling::None && !callInfo.sig.isAsyncCall()) + { + BADCODE("We're trying to emit an async call, but the async resolved context didn't find one"); + } + if (isJmp) { if (callInfo.sig.numArgs != m_methodInfo->args.numArgs || @@ -4059,11 +4338,11 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re if (callInfo.methodFlags & CORINFO_FLG_INTRINSIC) { NamedIntrinsic ni = GetNamedIntrinsic(m_compHnd, m_methodHnd, callInfo.hMethod); - if ((ni == NI_System_StubHelpers_NextCallReturnAddress) && (m_ip[5] == CEE_POP)) + if ((ni == NI_System_StubHelpers_NextCallReturnAddress) && (m_ip[0] == CEE_POP)) { // Call to System.StubHelpers.NextCallReturnAddress followed by CEE_POP is a special case that we handle // as a no-op. - m_ip += 6; // Skip the call and the CEE_POP + m_ip += 1; // Skip the call and the CEE_POP return; } @@ -4075,12 +4354,13 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re ni == NI_System_StubHelpers_GetStubContext || ni == NI_System_StubHelpers_NextCallReturnAddress || ni == NI_System_Runtime_CompilerServices_RuntimeHelpers_SetNextCallGenericContext || - ni == NI_System_Runtime_CompilerServices_RuntimeHelpers_SetNextCallAsyncContinuation); + ni == NI_System_Runtime_CompilerServices_RuntimeHelpers_SetNextCallAsyncContinuation || + ni == NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncCallContinuation || + ni == NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncSuspend); if ((InterpConfig.InterpMode() == 3) || isMustExpand) { if (EmitNamedIntrinsicCall(ni, callInfo.kind == CORINFO_CALL, resolvedCallToken.hClass, callInfo.hMethod, callInfo.sig)) { - m_ip += 5; return; } } @@ -4144,12 +4424,19 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re int numArgs = numArgsFromStack + (newObjThisArgLocation == 0); int extraParamArgLocation = INT_MAX; + int continuationArgLocation = INT_MAX; if (callInfo.sig.hasTypeArg()) { extraParamArgLocation = callInfo.sig.hasThis() ? 1 : 0; numArgs++; } + if (callInfo.sig.isAsyncCall()) + { + continuationArgLocation = callInfo.sig.hasTypeArg() ? extraParamArgLocation + 1 : (callInfo.sig.hasThis() ? 1 : 0); + numArgs++; + } + int *callArgs = (int*) AllocMemPool((numArgs + 1) * sizeof(int)); CORINFO_ARG_LIST_HANDLE args; args = callInfo.sig.args; @@ -4171,6 +4458,11 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re // This is the extra type argument, which is not on the logical IL stack // Skip it for now. We will fill it in later. } + else if (iActualArg == continuationArgLocation) + { + // This is the async continuation argument, which is not on the logical IL stack + // Skip it for now. We will fill it in later. + } else if (iActualArg == newObjThisArgLocation) { // This is the newObj arg type argument, which is not on the logical IL stack @@ -4358,6 +4650,17 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re callArgs[extraParamArgLocation] = contextParamVar; } + if (continuationArgLocation != INT_MAX) + { + PushStackType(StackTypeI, NULL); + m_pStackPointer--; + int32_t continuationArg = m_pStackPointer[0].var; + + AddIns(INTOP_LDNULL); + m_pLastNewIns->SetDVar(continuationArg); + callArgs[continuationArgLocation] = continuationArg; + } + // Process dVar int32_t dVar; if (newObjDVar != -1) @@ -4574,7 +4877,7 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re break; case CORINFO_VIRTUALCALL_LDVIRTFTN: - if ((callInfo.sig.sigInst.methInstCount != 0) || (m_compHnd->getMethodAttribs(callInfo.hMethod) & CORINFO_FLG_SHAREDINST)) + if ((callInfo.sig.sigInst.methInstCount != 0) || (m_compHnd->getClassAttribs(m_compHnd->getMethodClass(callInfo.hMethod)) & CORINFO_FLG_SHAREDINST)) { assert(extraParamArgLocation == INT_MAX); // We should not have a type argument for the ldvirtftn path since we don't know @@ -4612,37 +4915,626 @@ void InterpCompiler::EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool re break; } - m_pLastNewIns->SetDVar(dVar); - m_pLastNewIns->SetSVar(CALL_ARGS_SVAR); + m_pLastNewIns->SetDVar(dVar); + m_pLastNewIns->SetSVar(CALL_ARGS_SVAR); + + m_pLastNewIns->flags |= INTERP_INST_FLAG_CALL; + m_pLastNewIns->info.pCallInfo = (InterpCallInfo*)AllocMemPool0(sizeof (InterpCallInfo)); + m_pLastNewIns->info.pCallInfo->pCallArgs = callArgs; + + if (injectRet) + { + // Jmp to PInvoke was converted to normal pinvoke, so we need to inject a ret after the call + InterpType retType = GetInterpType(m_methodInfo->args.retType); + switch (retType) + { + case InterpTypeVT: + { + AddIns(INTOP_RET_VT); + m_pLastNewIns->SetSVar(dVar); + m_pLastNewIns->data[0] = m_pVars[dVar].size; + break; + } + case InterpTypeVoid: + AddIns(INTOP_RET_VOID); + break; + default: + AddIns(GetRetForType(retType)); + m_pLastNewIns->SetSVar(dVar); + break; + } + } + + if (callInfo.sig.isAsyncCall() && m_methodInfo->args.isAsyncCall()) // Async2 functions may need to suspend + { + EmitSuspend(callInfo, continuationContextHandling); + } +} + +void SetSlotToTrue(TArray &gcRefMap, int32_t slotOffset) +{ + assert(slotOffset % sizeof(void*) == 0); + int32_t slotIndex = slotOffset / sizeof(void*); + while (gcRefMap.GetSize() <= slotIndex) + { + int32_t growSize = gcRefMap.GetSize(); + if (growSize < 16) + growSize = 16; + gcRefMap.GrowBy(growSize); + } + gcRefMap.Set(slotIndex, true); +} + +void InterpCompiler::EmitSuspend(const CORINFO_CALL_INFO &callInfo, ContinuationContextHandling ContinuationContextHandling) +{ + CORINFO_LOOKUP_KIND kindForAllocationContinuation; + m_compHnd->getLocationOfThisType(m_methodHnd, &kindForAllocationContinuation); + + bool needsKeepAlive = kindForAllocationContinuation.needsRuntimeLookup && kindForAllocationContinuation.runtimeLookupKind != CORINFO_LOOKUP_THISOBJ; + + // Compute the number of EH clauses that overlap with this BB. + if (m_pCBB->enclosingTryBlockCount == -1) + { + int32_t enclosingTryBlockCount = 0; + for (unsigned int i = 0; i < getEHcount(m_methodInfo); i++) + { + CORINFO_EH_CLAUSE clause; + getEHinfo(m_methodInfo, i, &clause); + if (clause.TryOffset <= (uint32_t)m_pCBB->ilOffset && (clause.TryOffset + clause.TryLength) > (uint32_t)m_pCBB->ilOffset) + { + enclosingTryBlockCount++; + } + } + m_pCBB->enclosingTryBlockCount = enclosingTryBlockCount; + } + bool needsEHHandling = m_pCBB->enclosingTryBlockCount > (m_isAsyncMethodWithContextSaveRestore ? 1 : 0); + + bool captureContinuationContext = ContinuationContextHandling == ContinuationContextHandling::ContinueOnCapturedContext; + + // 1. For each IL var that is live across the await/continuation point. Add it to the list of live vars + // 2. For each stack var that is live other than the return value from the call + // - Create a new stack var, and move the value to it with the appopriate MOV intop. Then replace the var on the stack with the new var. + // - Put the new var in the list of live vars. + // 3. Iterate the list of live vars that we just produced. If any of them are GC Byref types or structure types which are byreflike, add them to a list of vars to be zeroed and remove them from the list of live vars. + // 4. Walk the list of live vars and assume that each one is laid out sequentially in memory where each var is aligned at at least INTERP_STACK_SLOT_SIZE. Based on that assumption build a bool array of what locations in memory are GC references. + // 5. using the getContinuationType api on m_compHnd and the bool array from step 4, get the continuation type handle. + // 6. Build an InterpAsyncSuspendData structure based on the list of live vars and the list of vars to be zeroed. + // 7. Emit the INTOP_HANDLE_CONTINUATION instruction with the index of a pointer to the InterpAsyncSuspendData structure. + + TArray liveVars(GetMemPoolAllocator()); + TArray varsToZero(GetMemPoolAllocator()); + + // Step 2: Handle live stack vars (excluding return value) + int32_t stackDepth = (int32_t)(m_pStackPointer - m_pStackBase); + int32_t returnValueVar = -1; + if (stackDepth > 0 && callInfo.sig.retType != CORINFO_TYPE_VOID) + { + returnValueVar = m_pStackPointer[-1].var; + liveVars.Add(returnValueVar); + } + + // Step 1: Collect live IL vars + for (int32_t i = 0; i < m_continuationArgIndex; i++) + { + // Check if this IL var is live across the continuation point + // For simplicity, we consider all IL vars as potentially live + // A more sophisticated implementation would use dataflow analysis + assert(m_pVars[i].ILGlobal); + liveVars.Add(i); + } + + for (int32_t i = 0; i < stackDepth; i++) + { + int32_t stackVar = m_pStackBase[i].var; + if (stackVar != returnValueVar) + { + // Create a new var and emit a move instruction + InterpType interpType = m_pVars[stackVar].interpType; + CORINFO_CLASS_HANDLE clsHnd = m_pVars[stackVar].clsHnd; + int32_t size = m_pVars[stackVar].size; + int32_t newVar = CreateVarExplicit(interpType, clsHnd, size); + + // Emit move from old var to new var + AddIns(InterpGetMovForType(interpType, false)); + m_pLastNewIns->SetSVar(stackVar); + m_pLastNewIns->SetDVar(newVar); + if (interpType == InterpTypeVT) + { + m_pLastNewIns->data[0] = size; + } + + // Replace the var on the stack with the new var + m_pStackBase[i].var = newVar; + liveVars.Add(newVar); + } + } + + // Step 3: Identify byref and byref-like types to be zeroed + for (int32_t i = 0; i < liveVars.GetSize(); i++) + { + int32_t var = liveVars.Get(i); + InterpType interpType = m_pVars[var].interpType; + CORINFO_CLASS_HANDLE clsHnd = m_pVars[var].clsHnd; + + bool shouldZero = false; + if (interpType == InterpTypeByRef) + { + shouldZero = true; + } + else if (interpType == InterpTypeVT && clsHnd != NULL) + { + // Check if this is a byref-like struct + DWORD classAttribs = m_compHnd->getClassAttribs(clsHnd); + if (classAttribs & CORINFO_FLG_BYREF_LIKE) + { + shouldZero = true; + } + } + + if (shouldZero) + { + varsToZero.Add(var); + liveVars.RemoveAt(i); + i--; // Adjust index after removal + } + } + + // Step 4: Build GC reference map + + // Calculate number of pointer-sized slots + TArray objRefSlots(GetMemPoolAllocator()); + + // Fill in the GC reference map + int32_t currentOffset = 0; + int32_t returnValueDataStartOffset = 0; + for (int32_t i = -3; i < liveVars.GetSize(); i++) + { + int32_t var; + + if (i == -3) + { + if (!needsEHHandling) + continue; + INTERP_DUMP("Allocate EH at offset %d\n", currentOffset); + SetSlotToTrue(objRefSlots, currentOffset); + currentOffset += sizeof(void*); // Align to pointer size to match the expected layout + continue; + } + if (i == -2) + { + if (!captureContinuationContext) + continue; + INTERP_DUMP("Allocate ContinuationContext at offset %d\n", currentOffset); + SetSlotToTrue(objRefSlots, currentOffset); + currentOffset += sizeof(void*); // Align to pointer size to match the expected layout + continue; + } + if (i == -1) + { + returnValueDataStartOffset = currentOffset; + // Handle return value first + if (returnValueVar == -1) + continue; + var = returnValueVar; + INTERP_DUMP("returnValueVar is %d\n", returnValueVar); + } + else + { + var = liveVars.Get(i); + if (var == returnValueVar) + continue; + } + InterpType interpType = m_pVars[var].interpType; + CORINFO_CLASS_HANDLE clsHnd = m_pVars[var].clsHnd; + + int32_t alignUNUSED; + int32_t size = GetInterpTypeStackSize(clsHnd, interpType, &alignUNUSED); + + // Since we're representing copying out of the continuation into interpreter stack frames + // which are always aligned to INTERP_STACK_SLOT_SIZE, we need to ensure that each + // var takes up at least that much space. + size = ALIGN_UP_TO(size, INTERP_STACK_SLOT_SIZE); + + INTERP_DUMP("Allocate var %d at data offset %d (size %d) (offsetFromStartOfReturnValue = %d)\n", var, currentOffset, size, currentOffset - returnValueDataStartOffset); + + if (interpType == InterpTypeO) + { + // Mark as GC reference + SetSlotToTrue(objRefSlots, currentOffset); + } + else if (interpType == InterpTypeVT) + { + assert(clsHnd != NULL); + InterpreterStackMap* stackMap = GetInterpreterStackMap(clsHnd); + + for (unsigned j = 0; j < stackMap->m_slotCount; j++) + { + InterpreterStackMapSlot slotInfo = stackMap->m_slots[j]; + SetSlotToTrue(objRefSlots, currentOffset + slotInfo.m_offsetBytes); + } + } + + currentOffset += size; + } + + int32_t execContextOffset = 0; + { + // Mark ExecContext pointer as a GC reference + execContextOffset = currentOffset; + INTERP_DUMP("Allocate ExecutableContext at offset %d\n", currentOffset); + SetSlotToTrue(objRefSlots, currentOffset); + currentOffset += sizeof(void*); + } + + int32_t keepAliveOffset = 0; + if (needsKeepAlive) + { + // Mark keep-alive pointer as a GC reference + keepAliveOffset = currentOffset; + INTERP_DUMP("Allocate KeepAlive at offset %d\n", currentOffset); + SetSlotToTrue(objRefSlots, currentOffset); + currentOffset += sizeof(void*); + } + + // Step 5: Get continuation type handle + assert((int32_t)(currentOffset / sizeof(void*)) <= objRefSlots.GetSize()); + CORINFO_CLASS_HANDLE continuationTypeHnd = m_compHnd->getContinuationType( + currentOffset, + objRefSlots.GetUnderlyingArray(), + (currentOffset / sizeof(void*)) * sizeof(bool) + ); + + // Step 6: Build InterpAsyncSuspendData structure + if (m_asyncResumeFuncPtr == NULL) + { + m_compHnd->getAsyncResumptionStub(&m_asyncResumeFuncPtr); + assert(m_asyncResumeFuncPtr != NULL); + } + + InterpAsyncSuspendData* suspendData = (InterpAsyncSuspendData*)AllocMethodData(sizeof(InterpAsyncSuspendData)); + m_asyncSuspendDataItems.Add(suspendData); + CORINFO_ASYNC_INFO asyncInfo; + m_compHnd->getAsyncInfo(&asyncInfo); + + GetDataForHelperFtn(CORINFO_HELP_ALLOC_CONTINUATION); + suspendData->continuationTypeHnd = continuationTypeHnd; + AllocateIntervalMapData_ForVars(&suspendData->liveLocalsIntervals, liveVars); + AllocateIntervalMapData_ForVars(&suspendData->zeroedLocalsIntervals, varsToZero); + + int32_t flags = 0; + if (returnValueVar != -1) + { + flags |= CORINFO_CONTINUATION_HAS_RESULT; + } + + if (captureContinuationContext) + { + flags |= CORINFO_CONTINUATION_HAS_CONTINUATION_CONTEXT; + } + + if (ContinuationContextHandling == ContinuationContextHandling::ContinueOnThreadPool) + { + flags |= CORINFO_CONTINUATION_CONTINUE_ON_THREAD_POOL; + } + + if (needsEHHandling) + { + flags |= CORINFO_CONTINUATION_HAS_EXCEPTION; + } + + suspendData->flags = (CorInfoContinuationFlags)flags; + + suspendData->offsetIntoContinuationTypeForExecutionContext = execContextOffset + OFFSETOF__CORINFO_Continuation__data; + suspendData->keepAliveOffset = keepAliveOffset + OFFSETOF__CORINFO_Continuation__data; + suspendData->captureSyncContextMethod = asyncInfo.captureContinuationContextMethHnd; + suspendData->restoreExecutionContextMethod = asyncInfo.restoreExecutionContextMethHnd; + suspendData->restoreContextsMethod = asyncInfo.restoreContextsMethHnd; + suspendData->resumeInfo.Resume = (size_t)m_asyncResumeFuncPtr; + suspendData->resumeInfo.DiagnosticIP = (size_t)NULL; + suspendData->methodStartIP = 0; // This is filled in by logic later in emission once we know the final address of the method + suspendData->continuationArgOffset = m_pVars[m_continuationArgIndex].offset; + suspendData->asyncMethodReturnType = NULL; + switch (m_methodInfo->args.retType) + { + case CORINFO_TYPE_VALUECLASS: + suspendData->asyncMethodReturnType = m_methodInfo->args.retTypeClass; + suspendData->asyncMethodReturnTypePrimitiveSize = 0; + break; + case CORINFO_TYPE_STRING: + case CORINFO_TYPE_CLASS: + suspendData->asyncMethodReturnType = m_compHnd->getBuiltinClass(CLASSID_SYSTEM_OBJECT); + suspendData->asyncMethodReturnTypePrimitiveSize = 0; + break; + case CORINFO_TYPE_BYTE: + case CORINFO_TYPE_UBYTE: + suspendData->asyncMethodReturnTypePrimitiveSize = 1; + break; + case CORINFO_TYPE_SHORT: + case CORINFO_TYPE_USHORT: + suspendData->asyncMethodReturnTypePrimitiveSize = 2; + break; + case CORINFO_TYPE_INT: + case CORINFO_TYPE_UINT: + suspendData->asyncMethodReturnTypePrimitiveSize = 4; + break; + case CORINFO_TYPE_LONG: + case CORINFO_TYPE_ULONG: + suspendData->asyncMethodReturnTypePrimitiveSize = 8; + break; + case CORINFO_TYPE_FLOAT: + suspendData->asyncMethodReturnTypePrimitiveSize = 4; + break; + case CORINFO_TYPE_DOUBLE: + suspendData->asyncMethodReturnTypePrimitiveSize = 8; + break; + case CORINFO_TYPE_BOOL: + suspendData->asyncMethodReturnTypePrimitiveSize = 1; + break; + case CORINFO_TYPE_CHAR: + suspendData->asyncMethodReturnTypePrimitiveSize = 2; + break; + case CORINFO_TYPE_BYREF: + BADCODE("ByRef return types not supported for async methods"); + break; + case CORINFO_TYPE_NATIVEINT: + case CORINFO_TYPE_NATIVEUINT: + suspendData->asyncMethodReturnTypePrimitiveSize = sizeof(void*); + break; + + case CORINFO_TYPE_VOID: + suspendData->asyncMethodReturnType = NULL; + suspendData->asyncMethodReturnTypePrimitiveSize = 0; + break; + case CORINFO_TYPE_PTR: + suspendData->asyncMethodReturnTypePrimitiveSize = sizeof(void*); + break; + case CORINFO_TYPE_REFANY: + BADCODE("TypedReference return types not supported for async methods"); + break; + + case CORINFO_TYPE_VAR: + BADCODE("VAR return types should have been translated before we get here on async methods"); + break; + default: + // Other types are not supported + BADCODE("Unsupported async method return type"); + break; + } + + // Step 7: Emit the INTOP_HANDLE_CONTINUATION instruction + + CorInfoHelpFunc helperFuncForAllocatingContinuation = CORINFO_HELP_ALLOC_CONTINUATION; + int handleContinuationOpcode = INTOP_HANDLE_CONTINUATION; + int32_t varGenericContext = -1; + + if (needsKeepAlive) + { + handleContinuationOpcode = INTOP_HANDLE_CONTINUATION_GENERIC; + switch (kindForAllocationContinuation.runtimeLookupKind) + { + case CORINFO_LOOKUP_CLASSPARAM: + helperFuncForAllocatingContinuation = CORINFO_HELP_ALLOC_CONTINUATION_CLASS; + varGenericContext = getParamArgIndex(); + break; + case CORINFO_LOOKUP_METHODPARAM: + { + helperFuncForAllocatingContinuation = CORINFO_HELP_ALLOC_CONTINUATION_METHOD; + varGenericContext = getParamArgIndex(); + break; + } + default: + assert(0); + break; + } + } + + AddIns(handleContinuationOpcode); + int32_t suspendDataIndex = GetDataItemIndex(suspendData); + m_pLastNewIns->data[0] = suspendDataIndex; + m_pLastNewIns->data[1] = GetDataForHelperFtn(helperFuncForAllocatingContinuation); + PushInterpType(InterpTypeO, NULL); + int32_t varAllocatedContinuation = m_pStackPointer[-1].var; + m_pStackPointer--; + + if (handleContinuationOpcode == INTOP_HANDLE_CONTINUATION_GENERIC) + { + m_pLastNewIns->SetSVar(varGenericContext); + } + m_pLastNewIns->SetDVar(varAllocatedContinuation); + + AddIns(INTOP_CAPTURE_CONTEXT_ON_SUSPEND); + m_pLastNewIns->data[0] = suspendDataIndex; + m_pLastNewIns->SetSVar(varAllocatedContinuation); + PushInterpType(InterpTypeI, NULL); + int32_t unusedRetVal = m_pStackPointer[-1].var; + m_pStackPointer--; + m_pLastNewIns->SetDVar(unusedRetVal); + + if (m_isAsyncMethodWithContextSaveRestore) + { + // Save the current execution context into the continuation + AddIns(INTOP_RESTORE_CONTEXTS_ON_SUSPEND); + m_pLastNewIns->data[0] = suspendDataIndex; + m_pLastNewIns->SetSVars3(varAllocatedContinuation, m_continuationArgIndex, m_execContextVarIndex /* We know the sync context immediately follows */); + PushInterpType(InterpTypeO, NULL); + varAllocatedContinuation = m_pStackPointer[-1].var; + m_pStackPointer--; + m_pLastNewIns->SetDVar(varAllocatedContinuation); + } + + // Add return pad for allocation helper call. This will fill in the continuation with the needed data, and actually suspend the method. + AddIns(INTOP_HANDLE_CONTINUATION_SUSPEND); + m_pLastNewIns->data[0] = suspendDataIndex; + m_pLastNewIns->SetSVar(varAllocatedContinuation); + + // Add location to resume to. The implementation of this opcode will: + // - restore the data captured + // - If there is an exception, throw it + // - if there is a captured exec context, call the restoration function. + AddIns(INTOP_HANDLE_CONTINUATION_RESUME); + m_pLastNewIns->data[0] = suspendDataIndex; + + // Once we've resumed, if the return type is a primitive integral type which + // isn't an I4/U4 but is represented as such on the evaluation stack, sign/zero + // extend it to the proper size. + switch (callInfo.sig.retType) + { + case CORINFO_TYPE_UBYTE: + case CORINFO_TYPE_BOOL: + EmitConv(&m_pStackPointer[-1], StackTypeI4, INTOP_CONV_U1_I4); + break; + case CORINFO_TYPE_BYTE: + EmitConv(&m_pStackPointer[-1], StackTypeI4, INTOP_CONV_I1_I4); + break; + case CORINFO_TYPE_USHORT: + case CORINFO_TYPE_CHAR: + EmitConv(&m_pStackPointer[-1], StackTypeI4, INTOP_CONV_U2_I4); + break; + case CORINFO_TYPE_SHORT: + EmitConv(&m_pStackPointer[-1], StackTypeI4, INTOP_CONV_I2_I4); + break; + default: + // Other types do not need extension + break; + } +} + +InterpIntervalMapEntry InterpCompiler::ComputeNextIntervalMapEntry_ForVars(const TArray &vars, int32_t *pNextIndex) +{ + if (*pNextIndex == vars.GetSize()) + { + InterpIntervalMapEntry endEntry; + endEntry.startOffset = 0; + endEntry.countBytes = 0; + return endEntry; + } + + InterpIntervalMapEntry entry; + entry.startOffset = vars.Get((*pNextIndex)++); + int32_t lastOffset = entry.startOffset; + + while (*pNextIndex < vars.GetSize() && vars.Get(*pNextIndex) == (lastOffset + 1)) + { + lastOffset++; + (*pNextIndex)++; + } + + entry.countBytes = lastOffset - entry.startOffset + 1; + return entry; +} + +void InterpCompiler::AllocateIntervalMapData_ForVars(InterpIntervalMapEntry** ppIntervalMap, const TArray &vars) +{ + int32_t intervalCount = 1; + int32_t nextIndex = 0; + while (ComputeNextIntervalMapEntry_ForVars(vars, &nextIndex).countBytes != 0) + { + intervalCount++; + } + nextIndex = 0; + + size_t size = sizeof(InterpIntervalMapEntry) * intervalCount; + *ppIntervalMap = (InterpIntervalMapEntry*)AllocMemPool0(size); + for (int32_t intervalMapIndex = 0; intervalMapIndex < intervalCount; intervalMapIndex++) + { + (*ppIntervalMap)[intervalMapIndex] = ComputeNextIntervalMapEntry_ForVars(vars, &nextIndex); + } + + assert((*ppIntervalMap)[intervalCount - 1].countBytes == 0); + + m_varIntervalMaps.Add(ppIntervalMap); +} + +void InterpCompiler::GetVarSizeAndOffset(const InterpIntervalMapEntry* pVarIntervalMap, int32_t entryIndex, int32_t internalIndex, uint32_t* pVarSize, uint32_t* pVarOffset) +{ + int32_t var = pVarIntervalMap[entryIndex].startOffset + internalIndex; + + *pVarOffset = m_pVars[var].offset; + *pVarSize = ALIGN_UP_TO(m_pVars[var].size, INTERP_STACK_SLOT_SIZE); +} + +bool IncrementVarIntervalMapIndices(const InterpIntervalMapEntry* pVarIntervalMap, int32_t *pEntryIndex, int32_t *pInternalIndex) +{ + if (pVarIntervalMap[*pEntryIndex].countBytes == 0) + { + // Fully incremented + return false; + } + + (*pInternalIndex)++; + if (*pInternalIndex >= (int32_t)pVarIntervalMap[*pEntryIndex].countBytes) + { + (*pEntryIndex)++; + *pInternalIndex = 0; + if (pVarIntervalMap[*pEntryIndex].countBytes == 0) + { + // Fully incremented + return false; + } + } + + return true; +} + +InterpIntervalMapEntry InterpCompiler::ComputeNextIntervalMapEntry_ForOffsets(const InterpIntervalMapEntry* pVarIntervalMap, int32_t *pNextIndex, int32_t *pInternalIndex) +{ + if (pVarIntervalMap[*pNextIndex].countBytes == 0) + { + InterpIntervalMapEntry endEntry; + endEntry.startOffset = 0; + endEntry.countBytes = 0; + return endEntry; + } + + InterpIntervalMapEntry entry; + GetVarSizeAndOffset(pVarIntervalMap, *pNextIndex, *pInternalIndex, &entry.countBytes, &entry.startOffset); + + while (IncrementVarIntervalMapIndices(pVarIntervalMap, pNextIndex, pInternalIndex)) + { + uint32_t nextOffsetIfMerging = entry.startOffset + entry.countBytes; + uint32_t nextOffset; + uint32_t nextSize; + GetVarSizeAndOffset(pVarIntervalMap, *pNextIndex, *pInternalIndex, &nextSize, &nextOffset); + if (nextOffset != nextOffsetIfMerging) + { + break; + } + entry.countBytes += nextSize; + } + + return entry; +} + +void InterpCompiler::ConvertToIntervalMapData_ForOffsets(InterpIntervalMapEntry** ppIntervalMap) +{ + const InterpIntervalMapEntry* const pVarIntervalMap = *ppIntervalMap; + + int32_t intervalCount = 1; + int32_t nextIndex = 0; + int32_t nextIndexInternal = 0; + while (ComputeNextIntervalMapEntry_ForOffsets(pVarIntervalMap, &nextIndex, &nextIndexInternal).countBytes != 0) + { + intervalCount++; + } + nextIndex = 0; + nextIndexInternal = 0; + + size_t size = sizeof(InterpIntervalMapEntry) * intervalCount; + *ppIntervalMap = (InterpIntervalMapEntry*)AllocMethodData(size); + for (int32_t intervalMapIndex = 0; intervalMapIndex < intervalCount; intervalMapIndex++) + { + (*ppIntervalMap)[intervalMapIndex] = ComputeNextIntervalMapEntry_ForOffsets(pVarIntervalMap, &nextIndex, &nextIndexInternal); + } - m_pLastNewIns->flags |= INTERP_INST_FLAG_CALL; - m_pLastNewIns->info.pCallInfo = (InterpCallInfo*)AllocMemPool0(sizeof (InterpCallInfo)); - m_pLastNewIns->info.pCallInfo->pCallArgs = callArgs; + assert((*ppIntervalMap)[intervalCount - 1].countBytes == 0); +} - if (injectRet) +void InterpCompiler::UpdateLocalIntervalMaps() +{ + for (int32_t i = 0; i < m_varIntervalMaps.GetSize(); i++) { - // Jmp to PInvoke was converted to normal pinvoke, so we need to inject a ret after the call - InterpType retType = GetInterpType(m_methodInfo->args.retType); - switch (retType) - { - case InterpTypeVT: - { - AddIns(INTOP_RET_VT); - m_pLastNewIns->SetSVar(dVar); - m_pLastNewIns->data[0] = m_pVars[dVar].size; - break; - } - case InterpTypeVoid: - AddIns(INTOP_RET_VOID); - break; - default: - AddIns(GetRetForType(retType)); - m_pLastNewIns->SetSVar(dVar); - break; - } + ConvertToIntervalMapData_ForOffsets(m_varIntervalMaps.Get(i)); } - - m_ip += 5; } static int32_t GetStindForType(InterpType interpType) @@ -5145,7 +6037,7 @@ static OpcodePeepElement peepBoxIsInstUnboxAnyOpcodes[] = { { 15, CEE_ILLEGAL } // End marker }; -OpcodePeepElement peepTypeValueTypeOpcodesOpcodes[] = { +static OpcodePeepElement peepTypeValueTypeOpcodesOpcodes[] = { { 0, CEE_LDTOKEN }, { 5, CEE_CALL }, { 10, CEE_CALL }, @@ -5204,62 +6096,268 @@ class InterpILOpcodePeeps bool FindAndApplyPeep(InterpCompiler* compiler) { - const uint8_t* ip = compiler->m_ip; + return compiler->FindAndApplyPeep(Peeps); + } +} ILOpcodePeeps; + + +bool InterpCompiler::IsRuntimeAsyncCall(const uint8_t* ip, OpcodePeepElement* pattern, void** ppComputedInfo) +{ + CORINFO_RESOLVED_TOKEN awaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[0].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &awaitResolvedToken); + if (!m_compHnd->isIntrinsic(awaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, awaitResolvedToken.hMethod) != NI_System_Runtime_CompilerServices_AsyncHelpers_Await) + { + return false; + } + + ResolveToken(getU4LittleEndian(ip + 1), CORINFO_TOKENKIND_Await, &m_resolvedAsyncCallToken); + if (m_resolvedAsyncCallToken.hMethod == NULL) + { + return false; + } + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnCapturedContext; + return true; +} + +bool InterpCompiler::IsRuntimeAsyncCallConfigureAwaitTask(const uint8_t* ip, OpcodePeepElement* pattern, void** ppComputedInfo) +{ + switch (*(ip + pattern[2].offsetIntoPeep - 1)) + { + case CEE_LDC_I4_0: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnThreadPool; + break; + case CEE_LDC_I4_1: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnCapturedContext; + break; + default: + return false; + } + + uint32_t stLocVar = 0; + uint32_t ldlocaVar = 0; + + switch (*(ip + pattern[0].offsetIntoPeep)) + { + case CEE_STLOC_S: + stLocVar = (ip + pattern[0].offsetIntoPeep)[1]; + break; + default: + // Must be STLOC + stLocVar = getU2LittleEndian(ip + pattern[0].offsetIntoPeep + 2); + break; + } + + switch (*(ip + pattern[1].offsetIntoPeep)) + { + case CEE_LDLOCA_S: + ldlocaVar = (ip + pattern[1].offsetIntoPeep)[1]; + break; + default: + // Must be LDLOCA + ldlocaVar = getU2LittleEndian(ip + pattern[1].offsetIntoPeep + 2); + break; + } + + if (ldlocaVar != stLocVar) + { + return false; + } + + CORINFO_RESOLVED_TOKEN configureAwaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[2].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &configureAwaitResolvedToken); + if (!m_compHnd->isIntrinsic(configureAwaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, configureAwaitResolvedToken.hMethod) != NI_System_Threading_Tasks_Task_ConfigureAwait) + { + return false; + } + + CORINFO_RESOLVED_TOKEN awaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[3].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &awaitResolvedToken); + if (!m_compHnd->isIntrinsic(awaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, awaitResolvedToken.hMethod) != NI_System_Runtime_CompilerServices_AsyncHelpers_Await) + { + return false; + } + + ResolveToken(getU4LittleEndian(ip + 1), CORINFO_TOKENKIND_Await, &m_resolvedAsyncCallToken); + if (m_resolvedAsyncCallToken.hMethod == NULL) + { + return false; + } + return true; +} + +bool InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTaskExactStLoc(const uint8_t* ip, OpcodePeepElement* pattern, void** ppComputedInfo) +{ + switch (*(ip + pattern[1].offsetIntoPeep - 1)) + { + case CEE_LDC_I4_0: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnThreadPool; + break; + case CEE_LDC_I4_1: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnCapturedContext; + break; + default: + return false; + } + + uint32_t stLocVar = 0; + uint32_t ldlocaVar = 0; + + switch (*(ip + pattern[0].offsetIntoPeep - 1)) + { + case CEE_STLOC_0: + stLocVar = 0; + break; + case CEE_STLOC_1: + stLocVar = 1; + break; + case CEE_STLOC_2: + stLocVar = 2; + break; + case CEE_STLOC_3: + stLocVar = 3; + break; + default: + return false; + } + + switch (*(ip + pattern[1].offsetIntoPeep)) + { + case CEE_LDLOCA_S: + ldlocaVar = (ip + pattern[1].offsetIntoPeep)[1]; + break; + default: + // Must be LDLOCA + ldlocaVar = getU2LittleEndian(ip + pattern[1].offsetIntoPeep + 2); + break; + } + + if (ldlocaVar != stLocVar) + { + return false; + } + + CORINFO_RESOLVED_TOKEN configureAwaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[1].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &configureAwaitResolvedToken); + if (!m_compHnd->isIntrinsic(configureAwaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, configureAwaitResolvedToken.hMethod) != NI_System_Threading_Tasks_Task_ConfigureAwait) + { + return false; + } + + CORINFO_RESOLVED_TOKEN awaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[2].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &awaitResolvedToken); + if (!m_compHnd->isIntrinsic(awaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, awaitResolvedToken.hMethod) != NI_System_Runtime_CompilerServices_AsyncHelpers_Await) + { + return false; + } + + ResolveToken(getU4LittleEndian(ip + 1), CORINFO_TOKENKIND_Await, &m_resolvedAsyncCallToken); + if (m_resolvedAsyncCallToken.hMethod == NULL) + { + return false; + } + return true; +} + +bool InterpCompiler::IsRuntimeAsyncCallConfigureAwaitValueTask(const uint8_t* ip, OpcodePeepElement* pattern, void** ppComputedInfo) +{ + switch (*(ip + pattern[0].offsetIntoPeep - 1)) + { + case CEE_LDC_I4_0: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnThreadPool; + break; + case CEE_LDC_I4_1: + m_currentContinuationContextHandling = ContinuationContextHandling::ContinueOnCapturedContext; + break; + default: + return false; + } + + CORINFO_RESOLVED_TOKEN configureAwaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[0].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &configureAwaitResolvedToken); + if (!m_compHnd->isIntrinsic(configureAwaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, configureAwaitResolvedToken.hMethod) != NI_System_Threading_Tasks_Task_ConfigureAwait) + { + return false; + } + + CORINFO_RESOLVED_TOKEN awaitResolvedToken; + ResolveToken(getU4LittleEndian(ip + pattern[1].offsetIntoPeep + 1), CORINFO_TOKENKIND_Method, &awaitResolvedToken); + if (!m_compHnd->isIntrinsic(awaitResolvedToken.hMethod) || + GetNamedIntrinsic(m_compHnd, m_methodHnd, awaitResolvedToken.hMethod) != NI_System_Runtime_CompilerServices_AsyncHelpers_Await) + { + return false; + } + + ResolveToken(getU4LittleEndian(ip + 1), CORINFO_TOKENKIND_Await, &m_resolvedAsyncCallToken); + if (m_resolvedAsyncCallToken.hMethod == NULL) + { + return false; + } + return true; +} + +bool InterpCompiler::FindAndApplyPeep(OpcodePeep* Peeps[]) +{ + const uint8_t* ip = m_ip; + + for (int i = 0; Peeps[i] != NULL; i++) + { + OpcodePeep *peep = Peeps[i]; + bool skipToNextPeep = false; - for (int i = 0; Peeps[i] != NULL; i++) + // Check to see if peep applies to the current block just looking at the IL streams + // starting at the current ip. + for (int iPeepOpCode = 0; peep->pattern[iPeepOpCode].opcode != CEE_ILLEGAL; iPeepOpCode++) { - OpcodePeep *peep = Peeps[i]; - bool skipToNextPeep = false; + const uint8_t *ipForOpcode = ip + peep->pattern[iPeepOpCode].offsetIntoPeep; + int32_t insOffset = (int32_t)(ipForOpcode - m_pILCode); - // Check to see if peep applies to the current block just looking at the IL streams - // starting at the current ip. - for (int iPeepOpCode = 0; peep->pattern[iPeepOpCode].opcode != CEE_ILLEGAL; iPeepOpCode++) + if (ipForOpcode >= m_pILCode + m_ILCodeSize) { - const uint8_t *ipForOpcode = ip + peep->pattern[iPeepOpCode].offsetIntoPeep; - int32_t insOffset = (int32_t)(ipForOpcode - compiler->m_pILCode); - - if (ipForOpcode >= compiler->m_pILCode + compiler->m_ILCodeSize) - { - // Ran off the end of the IL code - skipToNextPeep = true; - break; - } - InterpBasicBlock *pNewBB = compiler->m_ppOffsetToBB[insOffset]; - if (pNewBB != NULL && compiler->m_pCBB != pNewBB) - { - // Ran into a different basic block - skipToNextPeep = true; - break; - } + // Ran off the end of the IL code + skipToNextPeep = true; + break; + } + InterpBasicBlock *pNewBB = m_ppOffsetToBB[insOffset]; + if (pNewBB != NULL && m_pCBB != pNewBB) + { + // Ran into a different basic block + skipToNextPeep = true; + break; + } - if (peep->pattern[iPeepOpCode].opcode != CEEDecodeOpcode(&ipForOpcode)) - { - // Opcode does not match - skipToNextPeep = true; - break; - } + if (peep->pattern[iPeepOpCode].opcode != CEEDecodeOpcode(&ipForOpcode)) + { + // Opcode does not match + skipToNextPeep = true; + break; } - if (skipToNextPeep) - continue; + } + if (skipToNextPeep) + continue; - // The IL opcode pattern matches, now to check the more non-opcode conditions - void* computedInfo; - if ((compiler->*(peep->CheckIfTokensAllowPeepToBeUsedFunc))(ip, peep->pattern, &computedInfo)) + // The IL opcode pattern matches, now to check the more non-opcode conditions + void* computedInfo; + if ((this->*(peep->CheckIfTokensAllowPeepToBeUsedFunc))(ip, peep->pattern, &computedInfo)) + { + INTERP_DUMP("Applying peep: %s\n", peep->Name); + // The peep applies, so apply it + int skip = (this->*(peep->ApplyPeepFunc))(ip, peep->pattern, computedInfo); + if (skip == -1) { - INTERP_DUMP("Applying peep: %s\n", peep->Name); - // The peep applies, so apply it - int skip = (compiler->*(peep->ApplyPeepFunc))(ip, peep->pattern, computedInfo); - if (skip == -1) - { - skip = (int)peep->GetPeepSize(); - } - compiler->m_ip = ip + skip; - return true; + skip = (int)peep->GetPeepSize(); } + m_ip = ip + skip; + return true; } - return false; } -} ILOpcodePeeps; + return false; +} bool InterpCompiler::IsTypeEqualityCheckPeep(const uint8_t* ip, OpcodePeepElement* pattern, void** ppComputedInfo) { @@ -5898,7 +6996,7 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) { INTERP_DUMP("Synchronized method - adding extra IL opcodes for monitor enter/exit\n"); - // Synchronized methods are methods are implemented by adding a try/finally in the method + // Synchronized methods are methods implemented by adding a try/finally in the method // which takes the lock on entry and releases it on exit. To integrate this into the interpreter, we actually // add a set of extra "IL" opcodes at the end of the method which do the monitor finally and actual return // logic. We also add a variable to hold the flag indicating the lock was taken. @@ -5911,14 +7009,14 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) CEE_ENDFINALLY}; uint8_t opCodesForSynchronizedMethodEpilog[] = { CEE_CALL, - getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED, 0), - getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED, 1), - getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED, 2), - getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED, 3), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 0), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 1), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 2), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 3), CEE_RET }; m_synchronizedFinallyStartOffset = m_ILCodeSize; - m_synchronizedPostFinallyOffset = m_ILCodeSize + sizeof(opCodesForSynchronizedMethodFinally); + m_synchronizedOrAsyncPostFinallyOffset = m_ILCodeSize + sizeof(opCodesForSynchronizedMethodFinally); int32_t newILCodeSize = m_ILCodeSize + sizeof(opCodesForSynchronizedMethodFinally) + sizeof(opCodesForSynchronizedMethodEpilog); // We need to realloc the IL code buffer to hold the extra opcodes @@ -5928,7 +7026,49 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) memcpy(newILCode, m_pILCode, m_ILCodeSize); } memcpy(newILCode + m_synchronizedFinallyStartOffset, opCodesForSynchronizedMethodFinally, sizeof(opCodesForSynchronizedMethodFinally)); - memcpy(newILCode + m_synchronizedPostFinallyOffset, opCodesForSynchronizedMethodEpilog, sizeof(opCodesForSynchronizedMethodEpilog)); + memcpy(newILCode + m_synchronizedOrAsyncPostFinallyOffset, opCodesForSynchronizedMethodEpilog, sizeof(opCodesForSynchronizedMethodEpilog)); + m_pILCode = newILCode; + m_ILCodeSize = newILCodeSize; + + // Update the MethodInfo so that the various bits of logic in the compiler always get the updated IL code + m_methodInfo->ILCodeSize = newILCodeSize; + m_methodInfo->ILCode = m_pILCode; + } + else if (m_isAsyncMethodWithContextSaveRestore) + { + INTERP_DUMP("Synchronized method - adding extra IL opcodes for async save/restore\n"); + + // Async methods are methods implemented by adding a try/finally in the method + // which takes the lock on entry and releases it on exit. To integrate this into the interpreter, we actually + // add a set of extra "IL" opcodes at the end of the method which do the monitor finally and actual return + // logic. + + uint8_t opCodesForAsyncMethodFinally[] = { CEE_CALL, + getLittleEndianByte(INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD, 0), + getLittleEndianByte(INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD, 1), + getLittleEndianByte(INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD, 2), + getLittleEndianByte(INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD, 3), + CEE_ENDFINALLY}; + + uint8_t opCodesForAsyncMethodEpilog[] = { CEE_CALL, + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 0), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 1), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 2), + getLittleEndianByte(INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC, 3), + CEE_RET }; + + m_asyncFinallyStartOffset = m_ILCodeSize; + m_synchronizedOrAsyncPostFinallyOffset = m_ILCodeSize + sizeof(opCodesForAsyncMethodFinally); + int32_t newILCodeSize = m_ILCodeSize + sizeof(opCodesForAsyncMethodFinally) + sizeof(opCodesForAsyncMethodEpilog); + // We need to realloc the IL code buffer to hold the extra opcodes + + uint8_t* newILCode = (uint8_t*)AllocMemPool(newILCodeSize); + if (m_ILCodeSize != 0) + { + memcpy(newILCode, m_pILCode, m_ILCodeSize); + } + memcpy(newILCode + m_asyncFinallyStartOffset, opCodesForAsyncMethodFinally, sizeof(opCodesForAsyncMethodFinally)); + memcpy(newILCode + m_synchronizedOrAsyncPostFinallyOffset, opCodesForAsyncMethodEpilog, sizeof(opCodesForAsyncMethodEpilog)); m_pILCode = newILCode; m_ILCodeSize = newILCodeSize; @@ -5974,6 +7114,12 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) // adding an opcode. AddIns(INTOP_SAFEPOINT); + if (m_continuationArgIndex != -1) + { + AddIns(INTOP_CHECK_FOR_CONTINUATION); + m_pLastNewIns->SetSVar(m_continuationArgIndex); + } + CORJIT_FLAGS corJitFlags; DWORD jitFlagsSize = m_compHnd->getJitFlags(&corJitFlags, sizeof(corJitFlags)); assert(jitFlagsSize == sizeof(corJitFlags)); @@ -6066,6 +7212,51 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) m_pLastNewIns->SetSVars2(syncObjVar, m_synchronizedFlagVarIndex); } + if (m_isAsyncMethodWithContextSaveRestore) + { + // Load the address of the execution context/sync context locals, and call AsyncHelpers.CaptureContexts + AddIns(INTOP_LDLOCA); + m_pLastNewIns->SetSVar(m_execContextVarIndex); + PushInterpType(InterpTypeByRef, NULL); + m_pStackPointer[-1].SetAsLocalVariableAddress(); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + int32_t execContextAddressVar = m_pStackPointer[-1].var; + m_pStackPointer--; + + AddIns(INTOP_LDLOCA); + m_pLastNewIns->SetSVar(m_syncContextVarIndex); + PushInterpType(InterpTypeByRef, NULL); + m_pStackPointer[-1].SetAsLocalVariableAddress(); + m_pLastNewIns->SetDVar(m_pStackPointer[-1].var); + int32_t syncContextAddressVar = m_pStackPointer[-1].var; + m_pStackPointer--; + + CORINFO_ASYNC_INFO asyncInfo; + + m_compHnd->getAsyncInfo(&asyncInfo); + + // Create a new dummy var to serve as the dVar of the call + // FIXME Consider adding special dVar type (ex -1), that is + // resolved to null offset. The opcode shouldn't really write to it + PushStackType(StackTypeI4, NULL); + m_pStackPointer--; + int32_t dVar = m_pStackPointer[0].var; + + AddIns(INTOP_CALL); + m_pLastNewIns->data[0] = GetMethodDataItemIndex(asyncInfo.captureContextsMethHnd); + m_pLastNewIns->SetDVar(dVar); + m_pLastNewIns->SetSVar(CALL_ARGS_SVAR); + + m_pLastNewIns->flags |= INTERP_INST_FLAG_CALL; + m_pLastNewIns->info.pCallInfo = (InterpCallInfo*)AllocMemPool0(sizeof (InterpCallInfo)); + int32_t numArgs = 2; + int *callArgs = (int*) AllocMemPool((numArgs + 1) * sizeof(int)); + callArgs[0] = execContextAddressVar; + callArgs[1] = syncContextAddressVar; + callArgs[2] = CALL_ARGS_TERMINATOR; + m_pLastNewIns->info.pCallInfo->pCallArgs = callArgs; + } + linkBBlocks = true; needsRetryEmit = false; @@ -6436,24 +7627,24 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) m_pStackPointer[-1].BashStackTypeToI_ForLocalVariableAddress(); } - if (m_isSynchronized && m_currentILOffset < m_ILCodeSizeFromILHeader) + if ((m_isSynchronized || m_isAsyncMethodWithContextSaveRestore) && m_currentILOffset < m_ILCodeSizeFromILHeader) { - // We are in a synchronized method, but we need to go through the finally first + // We are in a synchronized/async method, but we need to go through the finally/fault first int32_t ilOffset = (int32_t)(m_ip - m_pILCode); - int32_t target = m_synchronizedPostFinallyOffset; + int32_t target = m_synchronizedOrAsyncPostFinallyOffset; if (retType != InterpTypeVoid) { CheckStackExact(1); - if (m_synchronizedRetValVarIndex == -1) + if (m_synchronizedOrAsyncRetValVarIndex == -1) { PushInterpType(retType, m_pVars[m_pStackPointer[-1].var].clsHnd); - m_synchronizedRetValVarIndex = m_pStackPointer[-1].var; + m_synchronizedOrAsyncRetValVarIndex = m_pStackPointer[-1].var; m_pStackPointer--; - INTERP_DUMP("Created synchronized ret val var V%d\n", m_synchronizedRetValVarIndex); + INTERP_DUMP("Created ret val var V%d\n", m_synchronizedOrAsyncRetValVarIndex); } - INTERP_DUMP("Store to synchronized ret val var V%d\n", m_synchronizedRetValVarIndex); - EmitStoreVar(m_synchronizedRetValVarIndex); + INTERP_DUMP("Store to ret val var V%d\n", m_synchronizedOrAsyncRetValVarIndex); + EmitStoreVar(m_synchronizedOrAsyncRetValVarIndex); } else { @@ -6465,6 +7656,12 @@ void InterpCompiler::GenerateCode(CORINFO_METHOD_INFO* methodInfo) break; } + if (m_methodInfo->args.isAsyncCall()) + { + // We're doing a standard return. Set the continuation return to NULL. + AddIns(INTOP_SET_CONTINUATION_NULL); + } + if (retType == InterpTypeVoid) { CheckStackExact(0); @@ -9083,6 +10280,14 @@ void InterpCompiler::UnlinkUnreachableBBlocks() } } +void InterpCompiler::UpdateWithFinalMethodByteCodeAddress(InterpByteCodeStart *pByteCodeStart) +{ + for (int32_t i = 0; i < m_asyncSuspendDataItems.GetSize(); i++) + { + m_asyncSuspendDataItems.Get(i)->methodStartIP = pByteCodeStart; + } +} + void InterpCompiler::PrintClassName(CORINFO_CLASS_HANDLE cls) { char className[100]; @@ -9327,6 +10532,31 @@ void InterpCompiler::PrintHelperFtn(int32_t _data) } } +void PrintLocalIntervals(InterpIntervalMapEntry* pIntervals) +{ + printf("["); + while(pIntervals->countBytes != 0) + { + printf("{%d, %d},", pIntervals->startOffset, pIntervals->startOffset + pIntervals->countBytes - 1); + pIntervals++; + } + printf("]"); +} + +void InterpCompiler::PrintInterpAsyncSuspendData(InterpAsyncSuspendData* pSuspendInfo) +{ + printf(" AsyncSuspendData["); + printf("continuationTypeHnd=%p", pSuspendInfo->continuationTypeHnd); + printf(", flags=%d", pSuspendInfo->flags); + printf(", offsetIntoContinuationTypeForExecutionContext=%d", pSuspendInfo->offsetIntoContinuationTypeForExecutionContext); + printf(", liveLocalsIntervals="); + PrintLocalIntervals(pSuspendInfo->liveLocalsIntervals); + printf(", zeroedLocalsIntervals="); + PrintLocalIntervals(pSuspendInfo->zeroedLocalsIntervals); + printf(", keepAliveOffset=%d", pSuspendInfo->keepAliveOffset); + printf("]"); +} + void InterpCompiler::PrintInsData(InterpInst *ins, int32_t insOffset, const int32_t *pData, int32_t opcode) { switch (g_interpOpArgType[opcode]) { @@ -9447,6 +10677,18 @@ void InterpCompiler::PrintInsData(InterpInst *ins, int32_t insOffset, const int3 printf(", %d", pData[1]); break; } + case InterpOpHandleContinuation: + { + PrintInterpAsyncSuspendData((InterpAsyncSuspendData*)GetDataItemAtIndex(pData[0])); + printf(", "); + PrintHelperFtn(pData[1]); + break; + } + case InterpOpHandleContinuationPt2: + { + PrintInterpAsyncSuspendData((InterpAsyncSuspendData*)GetDataItemAtIndex(pData[0])); + break; + } default: assert(0); break; diff --git a/src/coreclr/interpreter/compiler.h b/src/coreclr/interpreter/compiler.h index ef06a7188b5d36..91bf06a59d5b87 100644 --- a/src/coreclr/interpreter/compiler.h +++ b/src/coreclr/interpreter/compiler.h @@ -342,6 +342,9 @@ struct InterpBasicBlock // Number of catch, filter or finally clauses that overlap with this basic block. int32_t overlappingEHClauseCount; + // Number of try blocks that enclose this basic block. + int32_t enclosingTryBlockCount; + InterpBasicBlock(int32_t index) : InterpBasicBlock(index, 0) { } InterpBasicBlock(int32_t index, int32_t ilOffset) @@ -367,6 +370,7 @@ struct InterpBasicBlock isFinallyCallIsland = false; clauseVarIndex = -1; overlappingEHClauseCount = 0; + enclosingTryBlockCount = -1; } }; @@ -541,6 +545,7 @@ class InterpCompiler friend class InterpIAllocator; friend class InterpGcSlotAllocator; friend class InterpILOpcodePeeps; + friend class InterpAsyncCallPeeps; private: CORINFO_METHOD_HANDLE m_methodHnd; @@ -614,6 +619,8 @@ class InterpCompiler // from the interpreter code header during execution. TArray m_dataItems; + TArray m_asyncSuspendDataItems; + InterpDataItemIndexMap m_genericLookupToDataItemIndex; int32_t GetDataItemIndex(void* data) { @@ -764,13 +771,22 @@ class InterpCompiler int32_t m_paramArgIndex = -1; // Index of the type parameter argument in the m_pVars array. // For each catch or filter clause, we create a variable that holds the exception object. // This is the index of the first such variable. + int32_t m_continuationArgIndex = -1; // Index of the continuation argument in the m_pVars array for async methods. int32_t m_clauseVarsIndex = 0; + + int32_t m_synchronizedOrAsyncPostFinallyOffset = -1; // If the method is synchronized/async, this is the offset of the instruction after the finally which does the actual return bool m_isSynchronized = false; int32_t m_synchronizedFlagVarIndex = -1; // If the method is synchronized, this is the index of the argument that flag indicating if the lock was taken - int32_t m_synchronizedRetValVarIndex = -1; // If the method is synchronized, ret instructions are replaced with a store to this var and a leave to an epilog instruction. + int32_t m_synchronizedOrAsyncRetValVarIndex = -1; // If the method is synchronized, ret instructions are replaced with a store to this var and a leave to an epilog instruction. int32_t m_synchronizedFinallyStartOffset = -1; // If the method is synchronized, this is the offset of the start of the finally epilog - int32_t m_synchronizedPostFinallyOffset = -1; // If the method is synchronized, this is the offset of the instruction after the finally which does the actual return + + int32_t m_execContextVarIndex = -1; // If the method is async, this is the var index of the ExecutionContext local + int32_t m_syncContextVarIndex = -1; // If the method is async, this is the var index of the SynchronizationContext local + + void *m_asyncResumeFuncPtr = NULL; + bool m_isAsyncMethodWithContextSaveRestore = false; + int32_t m_asyncFinallyStartOffset = -1; // If the method is async, this is the offset of the start of the fault handler bool m_shadowCopyOfThisPointerActuallyNeeded = false; bool m_shadowCopyOfThisPointerHasVar = false; @@ -807,6 +823,8 @@ class InterpCompiler void ConvertFloatingPointStackEntryToStackType(StackInfo* entry, StackType type); // Opcode peeps + bool FindAndApplyPeep(OpcodePeep* Peeps[]); + bool IsStoreLoadPeep(const uint8_t* ip, OpcodePeepElement* peep, void** computedInfo); int ApplyStoreLoadPeep(const uint8_t* ip, OpcodePeepElement* peep, void* computedInfo); @@ -834,6 +852,21 @@ class InterpCompiler bool IsTypeValueTypePeep(const uint8_t* ip, OpcodePeepElement* peep, void** outComputedInfo); int ApplyTypeValueTypePeep(const uint8_t* ip, OpcodePeepElement* peep, void* computedInfo); + enum class ContinuationContextHandling : uint8_t + { + ContinueOnCapturedContext, + ContinueOnThreadPool, + None + }; + bool IsRuntimeAsyncCall(const uint8_t* ip, OpcodePeepElement* peep, void** computedInfo); + bool IsRuntimeAsyncCallConfigureAwaitTask(const uint8_t* ip, OpcodePeepElement* peep, void** computedInfo); + bool IsRuntimeAsyncCallConfigureAwaitValueTask(const uint8_t* ip, OpcodePeepElement* peep, void** computedInfo); + bool IsRuntimeAsyncCallConfigureAwaitValueTaskExactStLoc(const uint8_t* ip, OpcodePeepElement* peep, void** computedInfo); + + int ApplyRuntimeAsyncCall(const uint8_t* ip, OpcodePeepElement* peep, void* computedInfo) { return -1; } + ContinuationContextHandling m_currentContinuationContextHandling = ContinuationContextHandling::None; + CORINFO_RESOLVED_TOKEN m_resolvedAsyncCallToken; + // Code emit void EmitConv(StackInfo *sp, StackType type, InterpOpcode convOp); void EmitLoadVar(int var); @@ -843,6 +876,7 @@ class InterpCompiler void EmitShiftOp(int32_t opBase); void EmitCompareOp(int32_t opBase); void EmitCall(CORINFO_RESOLVED_TOKEN* pConstrainedToken, bool readonly, bool tailcall, bool newObj, bool isCalli); + void EmitSuspend(const CORINFO_CALL_INFO &callInfo, ContinuationContextHandling ContinuationContextHandling); void EmitCalli(bool isTailCall, void* calliCookie, int callIFunctionPointerVar, CORINFO_SIG_INFO* callSiteSig); bool EmitNamedIntrinsicCall(NamedIntrinsic ni, bool nonVirtualCall, CORINFO_CLASS_HANDLE clsHnd, CORINFO_METHOD_HANDLE method, CORINFO_SIG_INFO sig); void EmitLdind(InterpType type, CORINFO_CLASS_HANDLE clsHnd, int32_t offset); @@ -873,6 +907,15 @@ class InterpCompiler void EndActiveCall(InterpInst *call); void CompactActiveVars(int32_t *current_offset); + TArray m_varIntervalMaps; + InterpIntervalMapEntry ComputeNextIntervalMapEntry_ForVars(const TArray &vars, int32_t *pNextIndex); + void AllocateIntervalMapData_ForVars(InterpIntervalMapEntry** ppIntervalMap, const TArray &vars); + + void GetVarSizeAndOffset(const InterpIntervalMapEntry* pVarIntervalMap, int32_t entryIndex, int32_t internalIndex, uint32_t* pVarSize, uint32_t* pVarOffset); + InterpIntervalMapEntry ComputeNextIntervalMapEntry_ForOffsets(const InterpIntervalMapEntry* pVarIntervalMap, int32_t *pNextIndex, int32_t *pInternalIndex); + void ConvertToIntervalMapData_ForOffsets(InterpIntervalMapEntry** ppIntervalMap); + void UpdateLocalIntervalMaps(); + // Passes int32_t* m_pMethodCode; int32_t m_methodCodeSize; // code size measured in int32_t slots, instead of bytes @@ -905,6 +948,7 @@ class InterpCompiler void PrintInsData(InterpInst *ins, int32_t offset, const int32_t *pData, int32_t opcode); void PrintCompiledCode(); void PrintCompiledIns(const int32_t *ip, const int32_t *start); + void PrintInterpAsyncSuspendData(InterpAsyncSuspendData* pSuspendInfo); #ifdef DEBUG InterpDumpScope m_dumpScope; TArray m_methodName; @@ -931,6 +975,7 @@ class InterpCompiler InterpMethod* CompileMethod(); void BuildGCInfo(InterpMethod *pInterpMethod); void BuildEHInfo(); + void UpdateWithFinalMethodByteCodeAddress(InterpByteCodeStart *pByteCodeStart); int32_t* GetCode(int32_t *pCodeSize); }; diff --git a/src/coreclr/interpreter/compileropt.cpp b/src/coreclr/interpreter/compileropt.cpp index fa7d297802d71d..2636afdb468be9 100644 --- a/src/coreclr/interpreter/compileropt.cpp +++ b/src/coreclr/interpreter/compileropt.cpp @@ -453,4 +453,6 @@ void InterpCompiler::AllocOffsets() m_globalVarsWithRefsStackTop = globalVarsWithRefsStackTop; m_totalVarsStackSize = ALIGN_UP_TO(finalVarsStackSize, INTERP_STACK_ALIGNMENT); + + UpdateLocalIntervalMaps(); } diff --git a/src/coreclr/interpreter/eeinterp.cpp b/src/coreclr/interpreter/eeinterp.cpp index 49af0d423fbe9e..72b02b6274fe82 100644 --- a/src/coreclr/interpreter/eeinterp.cpp +++ b/src/coreclr/interpreter/eeinterp.cpp @@ -126,6 +126,7 @@ CorJitResult CILInterp::compileMethod(ICorJitInfo* compHnd, *(InterpMethod**)args.hotCodeBlockRW = pMethod; memcpy ((uint8_t*)args.hotCodeBlockRW + sizeof(InterpMethod*), pIRCode, IRCodeSize * sizeof(int32_t)); + compiler.UpdateWithFinalMethodByteCodeAddress((InterpByteCodeStart*)args.hotCodeBlock); *entryAddress = (uint8_t*)args.hotCodeBlock; *nativeSizeOfCode = sizeOfCode; diff --git a/src/coreclr/interpreter/inc/interpretershared.h b/src/coreclr/interpreter/inc/interpretershared.h index cae71c625da081..ab7aab3303c6ce 100644 --- a/src/coreclr/interpreter/inc/interpretershared.h +++ b/src/coreclr/interpreter/inc/interpretershared.h @@ -22,8 +22,12 @@ struct InterpHelperData { uint32_t accessType : 3; }; -#ifndef INTERPRETER_COMPILER_INTERNAL +#ifdef INTERPRETER_COMPILER_INTERNAL +#define COMPILER_SHARED_TYPE(compilerType, vmType, fieldName) compilerType fieldName +#else +#define COMPILER_SHARED_TYPE(compilerType, vmType, fieldName) vmType fieldName class MethodDesc; +class MethodTable; #endif struct CallStubHeader; @@ -33,11 +37,7 @@ struct InterpMethod #if DEBUG InterpMethod *self; #endif -#ifdef INTERPRETER_COMPILER_INTERNAL - CORINFO_METHOD_HANDLE methodHnd; -#else - DPTR(MethodDesc) methodDesc; -#endif + COMPILER_SHARED_TYPE(CORINFO_METHOD_HANDLE, DPTR(MethodDesc), methodHnd); int32_t argsSize; int32_t allocaSize; void** pDataItems; @@ -188,4 +188,31 @@ enum class CalliFlags : int32_t PInvoke = 1 << 2, // The call is a PInvoke call }; +struct InterpIntervalMapEntry +{ + uint32_t startOffset; + uint32_t countBytes; // If count is 0 then this is the end marker. +}; + +struct InterpAsyncSuspendData +{ + CORINFO_AsyncResumeInfo resumeInfo; + + COMPILER_SHARED_TYPE(CORINFO_CLASS_HANDLE, DPTR(MethodTable), continuationTypeHnd); + + InterpIntervalMapEntry* zeroedLocalsIntervals; // This will be used for the locals we need to keep live. + InterpIntervalMapEntry* liveLocalsIntervals; // Following the end of this struct is the array of InterpIntervalMapEntry for live locals + CorInfoContinuationFlags flags; + int32_t offsetIntoContinuationTypeForExecutionContext; + int32_t keepAliveOffset; // Only needed if we have a generic context to keep alive + InterpByteCodeStart* methodStartIP; + COMPILER_SHARED_TYPE(CORINFO_CLASS_HANDLE, DPTR(MethodTable), asyncMethodReturnType); + int32_t asyncMethodReturnTypePrimitiveSize; // 0 if not primitive, otherwise size in bytes + int32_t continuationArgOffset; + + COMPILER_SHARED_TYPE(CORINFO_METHOD_HANDLE, DPTR(MethodDesc), captureSyncContextMethod); + COMPILER_SHARED_TYPE(CORINFO_METHOD_HANDLE, DPTR(MethodDesc), restoreExecutionContextMethod); + COMPILER_SHARED_TYPE(CORINFO_METHOD_HANDLE, DPTR(MethodDesc), restoreContextsMethod); +}; + #endif diff --git a/src/coreclr/interpreter/inc/intops.def b/src/coreclr/interpreter/inc/intops.def index 75b29c618aab49..6b8a382cb9ad9c 100644 --- a/src/coreclr/interpreter/inc/intops.def +++ b/src/coreclr/interpreter/inc/intops.def @@ -315,6 +315,8 @@ OPDEF(INTOP_SHL_I8, "shl.i8", 4, 1, 2, InterpOpNoArgs) OPDEF(INTOP_SHR_I4, "shr.i4", 4, 1, 2, InterpOpNoArgs) OPDEF(INTOP_SHR_I8, "shr.i8", 4, 1, 2, InterpOpNoArgs) +OPDEF(INTOP_CZERO_I, "czero.i", 4, 1, 1, InterpOpNoArgs) // Set dvar to 1 if operand is non-zero otherwise 0 (produces a 32bit int) + OPDEF(INTOP_CEQ_I4, "ceq.i4", 4, 1, 2, InterpOpNoArgs) OPDEF(INTOP_CEQ_I8, "ceq.i8", 4, 1, 2, InterpOpNoArgs) OPDEF(INTOP_CEQ_R4, "ceq.r4", 4, 1, 2, InterpOpNoArgs) @@ -433,6 +435,17 @@ OPDEF(INTOP_THROW_PNSE, "throw.pnse", 1, 0, 0, InterpOpNoArgs) OPDEF(INTOP_LOAD_FRAMEVAR, "load.framevar", 2, 1, 0, InterpOpNoArgs) +OPDEF(INTOP_SET_CONTINUATION_NULL, "set.continuation.null", 1, 0, 0, InterpOpNoArgs) +OPDEF(INTOP_SET_CONTINUATION, "set.continuation", 2, 0, 1, InterpOpNoArgs) +OPDEF(INTOP_GET_CONTINUATION, "get.continuation", 2, 1, 0, InterpOpNoArgs) +OPDEF(INTOP_HANDLE_CONTINUATION, "handle.continuation", 4, 1, 0, InterpOpHandleContinuation) +OPDEF(INTOP_HANDLE_CONTINUATION_GENERIC, "handle.continuation.generic", 5, 1, 1, InterpOpHandleContinuation) +OPDEF(INTOP_HANDLE_CONTINUATION_SUSPEND, "handle.continuation.suspend", 3, 0, 1, InterpOpHandleContinuationPt2) +OPDEF(INTOP_HANDLE_CONTINUATION_RESUME, "handle.continuation.resume", 2, 0, 0, InterpOpHandleContinuationPt2) +OPDEF(INTOP_CHECK_FOR_CONTINUATION, "check.for.continuation", 3, 0, 1, InterpOpNoArgs) +OPDEF(INTOP_CAPTURE_CONTEXT_ON_SUSPEND, "capture.context.on.suspend", 4, 1, 1, InterpOpHandleContinuationPt2) +OPDEF(INTOP_RESTORE_CONTEXTS_ON_SUSPEND, "restore.contexts.on.suspend", 6, 1, 3, InterpOpHandleContinuationPt2) + // Intrinsics OPDEF(INTOP_COMPARE_EXCHANGE_U1, "compare.exchange.u1", 5, 1, 3, InterpOpNoArgs) OPDEF(INTOP_COMPARE_EXCHANGE_U2, "compare.exchange.u2", 5, 1, 3, InterpOpNoArgs) diff --git a/src/coreclr/interpreter/intops.h b/src/coreclr/interpreter/intops.h index 98cd7127a7c6e4..fc894ae3e448c0 100644 --- a/src/coreclr/interpreter/intops.h +++ b/src/coreclr/interpreter/intops.h @@ -30,6 +30,8 @@ typedef enum InterpOpPointerHelperFtn, InterpOpPointerInt, InterpOpGenericLookupInt, + InterpOpHandleContinuation, + InterpOpHandleContinuationPt2, } InterpOpArgType; extern const uint8_t g_interpOpLen[]; @@ -127,6 +129,7 @@ inline double getR8LittleEndian(const uint8_t* ptr) // We use a couple of special "intrinsic" tokens to represent these operations. // These are recognized by our implementation of the CALL opcode. const uint32_t INTERP_CALL_SYNCHRONIZED_MONITOR_EXIT = 0xFFFFFFFE; -const uint32_t INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED = 0xFFFFFFFF; +const uint32_t INTERP_LOAD_RETURN_VALUE_FOR_SYNCHRONIZED_OR_ASYNC = 0xFFFFFFFF; +const uint32_t INTERP_RESTORE_CONTEXTS_FOR_ASYNC_METHOD = 0xFFFFFFFD; #endif diff --git a/src/coreclr/interpreter/intrinsics.cpp b/src/coreclr/interpreter/intrinsics.cpp index cd09858f3eb8ff..8ee940d97cad21 100644 --- a/src/coreclr/interpreter/intrinsics.cpp +++ b/src/coreclr/interpreter/intrinsics.cpp @@ -115,6 +115,15 @@ NamedIntrinsic GetNamedIntrinsic(COMP_HANDLE compHnd, CORINFO_METHOD_HANDLE comp else if (!strcmp(methodName, "SetNextCallAsyncContinuation")) return NI_System_Runtime_CompilerServices_RuntimeHelpers_SetNextCallAsyncContinuation; } + else if (!strcmp(className, "AsyncHelpers")) + { + if (!strcmp(methodName, "AsyncSuspend")) + return NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncSuspend; + else if (!strcmp(methodName, "AsyncCallContinuation")) + return NI_System_Runtime_CompilerServices_AsyncHelpers_AsyncCallContinuation; + else if (!strcmp(methodName, "Await")) + return NI_System_Runtime_CompilerServices_AsyncHelpers_Await; + } } else if (!strcmp(namespaceName, "System.Runtime.InteropServices")) { @@ -151,6 +160,15 @@ NamedIntrinsic GetNamedIntrinsic(COMP_HANDLE compHnd, CORINFO_METHOD_HANDLE comp return NI_System_Threading_Volatile_WriteBarrier; } } + else if (!strcmp(namespaceName, "System.Threading.Tasks")) + { + if (!strcmp(methodName, "ConfigureAwait")) + { + if (!strcmp(className, "Task`1") || !strcmp(className, "Task") || + !strcmp(className, "ValueTask`1") || !strcmp(className, "ValueTask")) + return NI_System_Threading_Tasks_Task_ConfigureAwait; + } + } return NI_Illegal; } diff --git a/src/coreclr/jit/codegencommon.cpp b/src/coreclr/jit/codegencommon.cpp index c5f2cf20db5da2..0289c0ef2c1374 100644 --- a/src/coreclr/jit/codegencommon.cpp +++ b/src/coreclr/jit/codegencommon.cpp @@ -5727,7 +5727,7 @@ CORINFO_FIELD_HANDLE CodeGen::genEmitAsyncResumeInfo(unsigned stateNum) emitter::dataSection* dataSection; UNATIVE_OFFSET baseOffs = genEmitAsyncResumeInfoTable(&dataSection); - return compiler->eeFindJitDataOffs(baseOffs + stateNum * sizeof(emitter::dataAsyncResumeInfo)); + return compiler->eeFindJitDataOffs(baseOffs + stateNum * sizeof(CORINFO_AsyncResumeInfo)); } //------------------------------------------------------------------------ diff --git a/src/coreclr/jit/emit.cpp b/src/coreclr/jit/emit.cpp index 0ef56334dfacc9..f18b84a19ca2b9 100644 --- a/src/coreclr/jit/emit.cpp +++ b/src/coreclr/jit/emit.cpp @@ -8023,7 +8023,7 @@ void emitter::emitAsyncResumeTable(unsigned numEntries, UNATIVE_OFFSET* dataSecO emitEnsureDataSectionAlignment(TARGET_POINTER_SIZE); UNATIVE_OFFSET secOffs = emitConsDsc.dsdOffs; - unsigned emittedSize = sizeof(emitter::dataAsyncResumeInfo) * numEntries; + unsigned emittedSize = sizeof(CORINFO_AsyncResumeInfo) * numEntries; emitConsDsc.dsdOffs += emittedSize; dataSection* secDesc = (dataSection*)emitGetMem(roundUp(sizeof(dataSection) + numEntries * sizeof(emitLocation))); @@ -8560,9 +8560,9 @@ void emitter::emitOutputDataSec(dataSecDsc* sec, BYTE* dst) { JITDUMP(" section %u, size %u, async resume info\n", secNum++, dscSize); - size_t numElems = dscSize / sizeof(emitter::dataAsyncResumeInfo); + size_t numElems = dscSize / sizeof(CORINFO_AsyncResumeInfo); - emitter::dataAsyncResumeInfo* aDstRW = (emitter::dataAsyncResumeInfo*)dstRW; + CORINFO_AsyncResumeInfo* aDstRW = (CORINFO_AsyncResumeInfo*)dstRW; for (size_t i = 0; i < numElems; i++) { emitLocation* emitLoc = &((emitLocation*)dsc->dsCont)[i]; @@ -8735,7 +8735,7 @@ void emitter::emitDispDataSec(dataSecDsc* section, BYTE* dst) { if (i > 0) { - sprintf_s(label, ArrLen(label), "RWD%02zu", i * sizeof(dataAsyncResumeInfo)); + sprintf_s(label, ArrLen(label), "RWD%02zu", i * sizeof(CORINFO_AsyncResumeInfo)); printf(labelFormat, label); } diff --git a/src/coreclr/jit/emit.h b/src/coreclr/jit/emit.h index 46d4dc2b3b6b80..0dc7710dc32124 100644 --- a/src/coreclr/jit/emit.h +++ b/src/coreclr/jit/emit.h @@ -3501,17 +3501,6 @@ class emitter /* The following logic keeps track of initialized data sections */ /************************************************************************/ - // Note: Keep synchronized with AsyncHelpers.ResumeInfo - struct dataAsyncResumeInfo - { - // delegate* - target_size_t Resume; - // Pointer in main code for diagnostics. See comments on - // ICorDebugInfo::AsyncSuspensionPoint::DiagnosticNativeOffset and - // ResumeInfo.DiagnosticIP in SPC. - target_size_t DiagnosticIP; - }; - /* One of these is allocated for every blob of initialized data */ struct dataSection diff --git a/src/coreclr/pal/inc/unixasmmacrosamd64.inc b/src/coreclr/pal/inc/unixasmmacrosamd64.inc index 1d3d11c7d24ab0..90c8947e754297 100644 --- a/src/coreclr/pal/inc/unixasmmacrosamd64.inc +++ b/src/coreclr/pal/inc/unixasmmacrosamd64.inc @@ -298,6 +298,7 @@ C_FUNC(\Name\()_End): __PWTB_StackAlloc = __PWTB_FloatArgumentRegisters + 8 * 16 + 8 // 8 floating point registers __PWTB_TransitionBlock = __PWTB_StackAlloc + __PWTB_ArgumentRegisters = __PWTB_TransitionBlock .if \stackAllocOnEntry >= 4*8 .error "Max supported stackAllocOnEntry is 3*8" diff --git a/src/coreclr/vm/amd64/AsmHelpers.asm b/src/coreclr/vm/amd64/AsmHelpers.asm index 772fe72643da8e..83e53a633ded54 100644 --- a/src/coreclr/vm/amd64/AsmHelpers.asm +++ b/src/coreclr/vm/amd64/AsmHelpers.asm @@ -587,6 +587,8 @@ HaveInterpThreadContext: lea rax, [rsp + __PWTB_TransitionBlock] ; Copy the arguments to the interpreter stack, invoke the InterpExecMethod and load the return value call qword ptr [r11] + ; Fill in the ContinuationContext register + mov rcx, [rsp + __PWTB_ArgumentRegisters] EPILOG_WITH_TRANSITION_BLOCK_RETURN @@ -1114,6 +1116,8 @@ END_PROLOGUE mov r11, rcx ; The routines list mov r10, rdx ; interpreter stack args call qword ptr [r11] + mov rdx, [rbp + 48] + mov [rdx], rcx mov rsp, rbp pop rbp ret @@ -1129,6 +1133,8 @@ END_PROLOGUE mov r10, rdx ; interpreter stack args mov rcx, r8 ; return buffer call qword ptr [r11] + mov rdx, [rbp + 48] + mov [rdx], rcx mov rsp, rbp pop rbp ret @@ -1144,6 +1150,8 @@ END_PROLOGUE mov r10, rdx ; interpreter stack args mov rdx, r8 ; return buffer call qword ptr [r11] + mov rdx, [rbp + 48] + mov [rdx], rcx mov rsp, rbp pop rbp ret @@ -1161,6 +1169,8 @@ END_PROLOGUE mov r11, rcx ; The routines list mov r10, rdx ; interpreter stack args call qword ptr [r11] + mov rdx, [rbp + 48] + mov [rdx], rcx mov r8, [rbp - 8] movsd real8 ptr [r8], xmm0 mov rsp, rbp @@ -1179,6 +1189,8 @@ END_PROLOGUE mov r11, rcx ; The routines list mov r10, rdx ; interpreter stack args call qword ptr [r11] + mov rdx, [rbp + 48] + mov [rdx], rcx mov r8, [rbp - 8] mov qword ptr [r8], rax mov rsp, rbp diff --git a/src/coreclr/vm/amd64/asmconstants.h b/src/coreclr/vm/amd64/asmconstants.h index 893f3b35458047..a16c20bdc18f12 100644 --- a/src/coreclr/vm/amd64/asmconstants.h +++ b/src/coreclr/vm/amd64/asmconstants.h @@ -586,7 +586,7 @@ ASMCONSTANTS_C_ASSERT(OFFSETOF__Thread__m_pInterpThreadContext == offsetof(Threa #define OFFSETOF__InterpThreadContext__pStackPointer 0x10 ASMCONSTANTS_C_ASSERT(OFFSETOF__InterpThreadContext__pStackPointer == offsetof(InterpThreadContext, pStackPointer)) -#define OFFSETOF__CallStubHeader__Routines 0x10 +#define OFFSETOF__CallStubHeader__Routines 0x18 ASMCONSTANTS_C_ASSERT(OFFSETOF__CallStubHeader__Routines == offsetof(CallStubHeader, Routines)) #ifdef TARGET_UNIX diff --git a/src/coreclr/vm/amd64/asmhelpers.S b/src/coreclr/vm/amd64/asmhelpers.S index 74cf5256b5873d..e8bc73c45c7095 100644 --- a/src/coreclr/vm/amd64/asmhelpers.S +++ b/src/coreclr/vm/amd64/asmhelpers.S @@ -482,6 +482,8 @@ LOCAL_LABEL(HaveInterpThreadContext): // rbx contains IR bytecode address // Copy the arguments to the interpreter stack, invoke the InterpExecMethod and load the return value call qword ptr [r11] + // Fill in the ContinuationContext register + mov rcx, [rsp + __PWTB_ArgumentRegisters + 3*8] EPILOG_WITH_TRANSITION_BLOCK_RETURN @@ -1733,14 +1735,15 @@ LEAF_END Load_XMM7, _TEXT NESTED_ENTRY CallJittedMethodRetVoid, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp - alloc_stack 0x10 - save_reg_postrsp r10, 0 + push_register r8 + push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov r10, [rsp] + mov r8, [rbp - 8] + mov [r8], rcx mov rsp, rbp pop rbp ret @@ -1749,15 +1752,16 @@ NESTED_END CallJittedMethodRetVoid, _TEXT NESTED_ENTRY CallJittedMethodRetBuffRDI, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp - alloc_stack 0x10 - save_reg_postrsp r10, 0 + push_register r8 + push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args mov rdi, rdx // return buffer call qword ptr [r11] - mov r10, [rsp] + mov r8, [rbp - 8] + mov [r8], rcx mov rsp, rbp pop rbp ret @@ -1766,15 +1770,16 @@ NESTED_END CallJittedMethodRetBuffRDI, _TEXT NESTED_ENTRY CallJittedMethodRetBuffRSI, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp - alloc_stack 0x10 - save_reg_postrsp r10, 0 + push_register r8 + push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args mov rsi, rdx // return buffer call qword ptr [r11] - mov r10, [rsp] + mov r8, [rbp - 8] + mov [r8], rcx mov rsp, rbp pop rbp ret @@ -1783,15 +1788,16 @@ NESTED_END CallJittedMethodRetBuffRSI, _TEXT NESTED_ENTRY CallJittedMethodRetDouble, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - //pop rdx - mov rdx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rdx, [rbp - 16] movsd real8 ptr [rdx], xmm0 mov rsp, rbp pop rbp @@ -1801,14 +1807,16 @@ NESTED_END CallJittedMethodRetDouble, _TEXT NESTED_ENTRY CallJittedMethodRetI8, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov rdx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rdx, [rbp - 16] mov qword ptr [rdx], rax mov rsp, rbp pop rbp @@ -1818,14 +1826,16 @@ NESTED_END CallJittedMethodRetI8, _TEXT NESTED_ENTRY CallJittedMethodRetI8I8, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov rcx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rcx, [rbp - 16] mov qword ptr [rcx], rax mov qword ptr [rcx + 8], rdx mov rsp, rbp @@ -1836,14 +1846,16 @@ NESTED_END CallJittedMethodRetI8I8, _TEXT NESTED_ENTRY CallJittedMethodRetI8Double, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov rcx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rcx, [rbp - 16] mov qword ptr [rcx], rax movsd real8 ptr [rcx + 8], xmm0 mov rsp, rbp @@ -1854,14 +1866,16 @@ NESTED_END CallJittedMethodRetI8Double, _TEXT NESTED_ENTRY CallJittedMethodRetDoubleI8, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov rcx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rcx, [rbp - 16] movsd real8 ptr [rcx], xmm0 mov qword ptr [rcx + 8], rax mov rsp, rbp @@ -1872,14 +1886,16 @@ NESTED_END CallJittedMethodRetDoubleI8, _TEXT NESTED_ENTRY CallJittedMethodRetDoubleDouble, _TEXT, NoHandler push_nonvol_reg rbp mov rbp, rsp + push_register r8 push_register rdx - push_register rax // align END_PROLOGUE sub rsp, rcx // total stack space mov r11, rdi // The routines list mov r10, rsi // interpreter stack args call qword ptr [r11] - mov rcx, [rbp - 8] + mov r8, [rbp - 8] + mov [r8], rcx + mov rcx, [rbp - 16] movsd real8 ptr [rcx], xmm0 movsd real8 ptr [rcx + 8], xmm1 mov rsp, rbp diff --git a/src/coreclr/vm/arm64/asmconstants.h b/src/coreclr/vm/arm64/asmconstants.h index c98dde2892ff84..40c77f77c923ae 100644 --- a/src/coreclr/vm/arm64/asmconstants.h +++ b/src/coreclr/vm/arm64/asmconstants.h @@ -318,7 +318,7 @@ ASMCONSTANTS_C_ASSERT(OFFSETOF__Thread__m_pInterpThreadContext == offsetof(Threa #define OFFSETOF__InterpThreadContext__pStackPointer 0x10 ASMCONSTANTS_C_ASSERT(OFFSETOF__InterpThreadContext__pStackPointer == offsetof(InterpThreadContext, pStackPointer)) -#define OFFSETOF__CallStubHeader__Routines 0x10 +#define OFFSETOF__CallStubHeader__Routines 0x18 ASMCONSTANTS_C_ASSERT(OFFSETOF__CallStubHeader__Routines == offsetof(CallStubHeader, Routines)) #define SIZEOF__TransitionBlock 0xb0 diff --git a/src/coreclr/vm/arm64/asmhelpers.S b/src/coreclr/vm/arm64/asmhelpers.S index 93abb4d13fe60f..046482ea6e996c 100644 --- a/src/coreclr/vm/arm64/asmhelpers.S +++ b/src/coreclr/vm/arm64/asmhelpers.S @@ -713,7 +713,7 @@ LOCAL_LABEL(NoManagedThreadOrCallStub): SAVE_FLOAT_ARGUMENT_REGISTERS sp, __PWTB_FloatArgumentRegisters #endif - add x0, sp, #__PWTB_TransitionBlock + 16 + add x0, sp, #__PWTB_TransitionBlock mov x1, x19 bl C_FUNC(GetInterpThreadContextWithPossiblyMissingThreadOrCallStub) mov x11, x0 @@ -741,6 +741,8 @@ LOCAL_LABEL(HaveInterpThreadContext): // Copy the arguments to the interpreter stack, invoke the InterpExecMethod and load the return value ldr x11, [x10], #8 blr x11 + // Fill in the ContinuationContext register + ldr x2, [sp, #(__PWTB_ArgumentRegister_FirstArg + 16)] EPILOG_WITH_TRANSITION_BLOCK_RETURN @@ -2298,16 +2300,20 @@ LEAF_END Load_Q1_Q2_Q3_Q4_Q5_Q6_Q7 // X0 - routines array // X1 - interpreter stack args location -// X2 - stack arguments size (properly aligned) +// X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVoid, _TEXT, NoHandler - PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -16 + PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 + str x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 + ldr x4, [fp, #16] + str x2, [x4] EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 16 + EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN NESTED_END CallJittedMethodRetVoid, _TEXT @@ -2315,16 +2321,20 @@ NESTED_END CallJittedMethodRetVoid, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetBuff, _TEXT, NoHandler - PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -16 + PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 + str x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 mov x8, x2 ldr x11, [x10], #8 blr x11 + ldr x4, [fp, #16] + str x2, [x4] EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 16 + EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN NESTED_END CallJittedMethodRetBuff, _TEXT @@ -2332,16 +2342,19 @@ NESTED_END CallJittedMethodRetBuff, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetI8, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str x0, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str x0, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2351,16 +2364,19 @@ NESTED_END CallJittedMethodRetI8, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2I8, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp x0, x1, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp x0, x1, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2370,16 +2386,19 @@ NESTED_END CallJittedMethodRet2I8, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetDouble, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str d0, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2389,16 +2408,19 @@ NESTED_END CallJittedMethodRetDouble, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Double, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp d0, d1, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2408,17 +2430,20 @@ NESTED_END CallJittedMethodRet2Double, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Double, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2], #16 - str d2, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp d0, d1, [x9], #16 + str d2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2428,17 +2453,20 @@ NESTED_END CallJittedMethodRet3Double, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Double, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2], #16 - stp d2, d3, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp d0, d1, [x9], #16 + stp d2, d3, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2448,16 +2476,19 @@ NESTED_END CallJittedMethodRet4Double, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetFloat, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str s0, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str s0, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2467,16 +2498,19 @@ NESTED_END CallJittedMethodRetFloat, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Float, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp s0, s1, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2486,17 +2520,20 @@ NESTED_END CallJittedMethodRet2Float, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Float, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2], #8 - str s2, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp s0, s1, [x9], #8 + str s2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2506,17 +2543,20 @@ NESTED_END CallJittedMethodRet3Float, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Float, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2], #8 - stp s2, s3, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + stp s0, s1, [x9], #8 + stp s2, s3, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2526,16 +2566,19 @@ NESTED_END CallJittedMethodRet4Float, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVector64, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str d0, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2545,17 +2588,20 @@ NESTED_END CallJittedMethodRetVector64, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Vector64, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2565,18 +2611,21 @@ NESTED_END CallJittedMethodRet2Vector64, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Vector64, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2], #8 - str d2, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9], #8 + str d2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2586,19 +2635,22 @@ NESTED_END CallJittedMethodRet3Vector64, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Vector64, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2], #8 - str d2, [x2], #8 - str d3, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9], #8 + str d2, [x9], #8 + str d3, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2608,16 +2660,19 @@ NESTED_END CallJittedMethodRet4Vector64, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVector128, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str q0, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2627,17 +2682,20 @@ NESTED_END CallJittedMethodRetVector128, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Vector128, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2647,18 +2705,21 @@ NESTED_END CallJittedMethodRet2Vector128, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Vector128, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2], #16 - str q2, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9], #16 + str q2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN @@ -2668,19 +2729,22 @@ NESTED_END CallJittedMethodRet3Vector128, _TEXT // X1 - interpreter stack args location // X2 - interpreter stack return value location // X3 - stack arguments size (properly aligned) +// X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Vector128, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32 - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2], #16 - str q2, [x2], #16 - str q3, [x2] + ldr x9, [fp, #24] + str x2, [x9] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9], #16 + str q2, [x9], #16 + str q3, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32 EPILOG_RETURN diff --git a/src/coreclr/vm/arm64/asmhelpers.asm b/src/coreclr/vm/arm64/asmhelpers.asm index c501e0b50fa24a..281ca3bd0e85bc 100644 --- a/src/coreclr/vm/arm64/asmhelpers.asm +++ b/src/coreclr/vm/arm64/asmhelpers.asm @@ -1071,7 +1071,7 @@ JIT_PollGCRarePath cbnz x11, HaveInterpThreadContext NoManagedThreadOrCallStub - add x0, sp, #__PWTB_TransitionBlock + 16 + add x0, sp, #__PWTB_TransitionBlock mov x1, x19 bl GetInterpThreadContextWithPossiblyMissingThreadOrCallStub mov x11, x0 @@ -1089,6 +1089,8 @@ HaveInterpThreadContext ; Copy the arguments to the interpreter stack, invoke the InterpExecMethod and load the return value ldr x11, [x10], #8 blr x11 + ; Fill in the ContinuationContext register + ldr x2, [sp, #(__PWTB_ArgumentRegister_FirstArg + 16)] EPILOG_WITH_TRANSITION_BLOCK_RETURN @@ -2526,16 +2528,20 @@ CopyLoop ; X0 - routines array ; X1 - interpreter stack args location - ; X2 - stack arguments size (properly aligned) + ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVoid - PROLOG_SAVE_REG_PAIR fp, lr, #-16! + PROLOG_SAVE_REG_PAIR fp, lr, #-32! + str x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 + ldr x4, [fp, #16] + str x2, [x4] EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR fp, lr, #16! + EPILOG_RESTORE_REG_PAIR fp, lr, #32! ret lr NESTED_END CallJittedMethodRetVoid @@ -2543,16 +2549,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetBuff - PROLOG_SAVE_REG_PAIR fp, lr, #-16! + PROLOG_SAVE_REG_PAIR fp, lr, #-32! + str x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 mov x8, x2 ldr x11, [x10], #8 blr x11 + ldr x4, [fp, #16] + str x2, [x4] EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR fp, lr, #16! + EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN NESTED_END CallJittedMethodRetBuff @@ -2560,16 +2570,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetI8 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str x0, [x2] + ldr x9, [fp, #16] + str x0, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2579,16 +2592,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2I8 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp x0, x1, [x2] + ldr x9, [fp, #16] + stp x0, x1, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2598,16 +2614,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetDouble PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2] + ldr x9, [fp, #16] + str d0, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2617,16 +2636,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Double PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2] + ldr x9, [fp, #16] + stp d0, d1, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2636,17 +2658,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Double PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2], #16 - str d2, [x2] + ldr x9, [fp, #16] + stp d0, d1, [x9], #16 + str d2, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2656,17 +2681,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Double PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp d0, d1, [x2], #16 - stp d2, d3, [x2] + ldr x9, [fp, #16] + stp d0, d1, [x9], #16 + stp d2, d3, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2676,16 +2704,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetFloat PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str s0, [x2] + ldr x9, [fp, #16] + str s0, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2695,16 +2726,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Float PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2] + ldr x9, [fp, #16] + stp s0, s1, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2714,17 +2748,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Float PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2], #8 - str s2, [x2] + ldr x9, [fp, #16] + stp s0, s1, [x9], #8 + str s2, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2734,17 +2771,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Float PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - stp s0, s1, [x2], #8 - stp s2, s3, [x2] + ldr x9, [fp, #16] + stp s0, s1, [x9], #8 + stp s2, s3, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2754,16 +2794,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVector64 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2] + ldr x9, [fp, #16] + str d0, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2773,17 +2816,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Vector64 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2793,18 +2839,21 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Vector64 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2], #8 - str d2, [x2] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9], #8 + str d2, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2814,19 +2863,22 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Vector64 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str d0, [x2], #8 - str d1, [x2], #8 - str d2, [x2], #8 - str d3, [x2] + ldr x9, [fp, #16] + str d0, [x9], #8 + str d1, [x9], #8 + str d2, [x9], #8 + str d3, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2836,16 +2888,19 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVector128 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2] + ldr x9, [fp, #16] + str q0, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2855,17 +2910,20 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Vector128 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2875,18 +2933,21 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet3Vector128 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2], #16 - str q2, [x2] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9], #16 + str q2, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN @@ -2896,19 +2957,22 @@ CopyLoop ; X1 - interpreter stack args location ; X2 - interpreter stack return value location ; X3 - stack arguments size (properly aligned) + ; X4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet4Vector128 PROLOG_SAVE_REG_PAIR fp, lr, #-32! - str x2, [fp, #16] + stp x2, x4, [fp, #16] sub sp, sp, x3 mov x10, x0 mov x9, x1 ldr x11, [x10], #8 blr x11 - ldr x2, [fp, #16] - str q0, [x2], #16 - str q1, [x2], #16 - str q2, [x2], #16 - str q3, [x2] + ldr x9, [fp, #16] + str q0, [x9], #16 + str q1, [x9], #16 + str q2, [x9], #16 + str q3, [x9] + ldr x9, [fp, #24] + str x2, [x9] EPILOG_STACK_RESTORE EPILOG_RESTORE_REG_PAIR fp, lr, #32! EPILOG_RETURN diff --git a/src/coreclr/vm/callstubgenerator.cpp b/src/coreclr/vm/callstubgenerator.cpp index ca105f15c90f05..a0a5f345a58979 100644 --- a/src/coreclr/vm/callstubgenerator.cpp +++ b/src/coreclr/vm/callstubgenerator.cpp @@ -1968,35 +1968,35 @@ PCODE CallStubGenerator::GetFPReg32RangeRoutine(int x1, int x2) } #endif // TARGET_ARM64 -extern "C" void CallJittedMethodRetVoid(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetDouble(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetI8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRetVoid(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetDouble(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetI8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRetVoid(); extern "C" void InterpreterStubRetDouble(); extern "C" void InterpreterStubRetI8(); #ifdef TARGET_AMD64 #ifdef TARGET_WINDOWS -extern "C" void CallJittedMethodRetBuffRCX(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetBuffRDX(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRetBuffRCX(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetBuffRDX(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRetBuffRCX(); extern "C" void InterpreterStubRetBuffRDX(); #else // TARGET_WINDOWS -extern "C" void CallJittedMethodRetBuffRDI(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetBuffRSI(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRetBuffRDI(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetBuffRSI(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRetBuffRDI(); extern "C" void InterpreterStubRetBuffRSI(); #endif // TARGET_WINDOWS #else // TARGET_AMD64 -extern "C" void CallJittedMethodRetBuff(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRetBuff(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRetBuff(); #endif // TARGET_AMD64 #ifdef UNIX_AMD64_ABI -extern "C" void CallJittedMethodRetI8I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetI8Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetDoubleI8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetDoubleDouble(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRetI8I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetI8Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetDoubleI8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetDoubleDouble(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRetI8I8(); extern "C" void InterpreterStubRetI8Double(); extern "C" void InterpreterStubRetDoubleI8(); @@ -2004,22 +2004,22 @@ extern "C" void InterpreterStubRetDoubleDouble(); #endif #ifdef TARGET_ARM64 -extern "C" void CallJittedMethodRet2I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet2Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet3Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet4Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetFloat(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet2Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet3Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet4Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetVector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet2Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet3Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet4Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRetVector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet2Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet3Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); -extern "C" void CallJittedMethodRet4Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize); +extern "C" void CallJittedMethodRet2I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet2Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet3Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet4Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetFloat(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet2Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet3Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet4Float(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetVector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet2Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet3Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet4Vector64(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetVector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet2Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet3Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet4Vector128(PCODE *routines, int8_t *pArgs, int8_t *pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRet2I8(); extern "C" void InterpreterStubRet2Double(); extern "C" void InterpreterStubRet3Double(); @@ -2039,10 +2039,10 @@ extern "C" void InterpreterStubRet4Vector128(); #endif // TARGET_ARM64 #if defined(TARGET_RISCV64) -extern "C" void CallJittedMethodRet2I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRet2Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetFloatInt(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); -extern "C" void CallJittedMethodRetIntFloat(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); +extern "C" void CallJittedMethodRet2I8(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRet2Double(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetFloatInt(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); +extern "C" void CallJittedMethodRetIntFloat(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuation); extern "C" void InterpreterStubRet2I8(); extern "C" void InterpreterStubRet2Double(); extern "C" void InterpreterStubRetFloatInt(); @@ -2273,15 +2273,15 @@ CallStubHeader *CallStubGenerator::GenerateCallStub(MethodDesc *pMD, AllocMemTra S_SIZE_T finalStubSize(sizeof(CallStubHeader) + m_routineIndex * sizeof(PCODE)); void *pHeaderStorage = pamTracker->Track(pLoaderAllocator->GetHighFrequencyHeap()->AllocMem(finalStubSize)); - CallStubHeader *pHeader = new (pHeaderStorage) CallStubHeader(m_routineIndex, pRoutines, ALIGN_UP(m_totalStackSize, STACK_ALIGN_SIZE), m_pInvokeFunction); + CallStubHeader *pHeader = new (pHeaderStorage) CallStubHeader(m_routineIndex, pRoutines, ALIGN_UP(m_totalStackSize, STACK_ALIGN_SIZE), sig.IsAsyncCall(), m_pInvokeFunction); return pHeader; } struct CachedCallStubKey { - CachedCallStubKey(int32_t hashCode, int numRoutines, PCODE *pRoutines, int totalStackSize, CallStubHeader::InvokeFunctionPtr pInvokeFunction) - : HashCode(hashCode), NumRoutines(numRoutines), TotalStackSize(totalStackSize), Invoke(pInvokeFunction), Routines(pRoutines) + CachedCallStubKey(int32_t hashCode, int numRoutines, PCODE *pRoutines, int totalStackSize, bool hasContinuationRet, CallStubHeader::InvokeFunctionPtr pInvokeFunction) + : HashCode(hashCode), NumRoutines(numRoutines), TotalStackSize(totalStackSize), HasContinuationRet(hasContinuationRet), Invoke(pInvokeFunction), Routines(pRoutines) { } @@ -2289,7 +2289,7 @@ struct CachedCallStubKey { LIMITED_METHOD_CONTRACT; - if (HashCode != other.HashCode || NumRoutines != other.NumRoutines || TotalStackSize != other.TotalStackSize || Invoke != other.Invoke) + if (HashCode != other.HashCode || NumRoutines != other.NumRoutines || TotalStackSize != other.TotalStackSize || Invoke != other.Invoke || HasContinuationRet != other.HasContinuationRet) return false; for (int i = 0; i < NumRoutines; i++) @@ -2303,15 +2303,16 @@ struct CachedCallStubKey const int32_t HashCode = 0; const int NumRoutines = 0; const int TotalStackSize = 0; + const bool HasContinuationRet = false; const CallStubHeader::InvokeFunctionPtr Invoke = NULL; // Pointer to the invoke function const PCODE *Routines; }; struct CachedCallStub { - CachedCallStub(int32_t hashCode, int numRoutines, PCODE *pRoutines, int totalStackSize, CallStubHeader::InvokeFunctionPtr pInvokeFunction) : + CachedCallStub(int32_t hashCode, int numRoutines, PCODE *pRoutines, int totalStackSize, bool hasContinuationRet, CallStubHeader::InvokeFunctionPtr pInvokeFunction) : HashCode(hashCode), - Header(numRoutines, pRoutines, totalStackSize, pInvokeFunction) + Header(numRoutines, pRoutines, totalStackSize, hasContinuationRet, pInvokeFunction) { } @@ -2325,6 +2326,7 @@ struct CachedCallStub Header.NumRoutines, &Header.Routines[0], Header.TotalStackSize, + Header.HasContinuationRet, Header.Invoke); } @@ -2371,12 +2373,14 @@ CallStubHeader *CallStubGenerator::GenerateCallStubForSig(MetaSig &sig) } hashState.Add(m_totalStackSize); hashState.AddPointer((void*)m_pInvokeFunction); + hashState.Add(sig.IsAsyncCall() ? 1 : 0); CachedCallStubKey cachedHeaderKey( hashState.ToHashCode(), m_routineIndex, pRoutines, ALIGN_UP(m_totalStackSize, STACK_ALIGN_SIZE), + sig.IsAsyncCall(), m_pInvokeFunction); CrstHolder lockHolder(&s_callStubCrst); @@ -2396,7 +2400,7 @@ CallStubHeader *CallStubGenerator::GenerateCallStubForSig(MetaSig &sig) // We only need to allocate the actual pRoutines array, and then we can just use the cachedHeader we already constructed size_t finalCachedCallStubSize = sizeof(CachedCallStub) + m_routineIndex * sizeof(PCODE); void* pHeaderStorage = amTracker.Track(SystemDomain::GetGlobalLoaderAllocator()->GetHighFrequencyHeap()->AllocMem(S_SIZE_T(finalCachedCallStubSize))); - CachedCallStub *pHeader = new (pHeaderStorage) CachedCallStub(cachedHeaderKey.HashCode, m_routineIndex, pRoutines, ALIGN_UP(m_totalStackSize, STACK_ALIGN_SIZE), m_pInvokeFunction); + CachedCallStub *pHeader = new (pHeaderStorage) CachedCallStub(cachedHeaderKey.HashCode, m_routineIndex, pRoutines, ALIGN_UP(m_totalStackSize, STACK_ALIGN_SIZE), sig.IsAsyncCall(), m_pInvokeFunction); s_callStubCache->Add(pHeader); amTracker.SuppressRelease(); @@ -2491,6 +2495,18 @@ void CallStubGenerator::ComputeCallStub(MetaSig &sig, PCODE *pRoutines) interpreterStackOffset += INTERP_STACK_SLOT_SIZE; } + if (argIt.HasAsyncContinuation()) + { +#if LOG_COMPUTE_CALL_STUB + printf("argIt.HasAsyncContinuation\n"); +#endif + // In the Interpreter calling convention the argument after the param type is the async continuation + ArgLocDesc asyncContinuationLocDesc; + argIt.GetAsyncContinuationLoc(&asyncContinuationLocDesc); + ProcessArgument(NULL, asyncContinuationLocDesc, pRoutines); + interpreterStackOffset += INTERP_STACK_SLOT_SIZE; + } + int ofs; while ((ofs = argIt.GetNextOffset()) != TransitionBlock::InvalidOffset) { diff --git a/src/coreclr/vm/callstubgenerator.h b/src/coreclr/vm/callstubgenerator.h index 84d46929361259..dab828a43a5a50 100644 --- a/src/coreclr/vm/callstubgenerator.h +++ b/src/coreclr/vm/callstubgenerator.h @@ -13,25 +13,27 @@ class AllocMemTracker; // stack, invokes the target method, and translates the return value back to the interpreter stack. struct CallStubHeader { - typedef void (*InvokeFunctionPtr)(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize); + typedef void (*InvokeFunctionPtr)(PCODE *routines, int8_t*pArgs, int8_t*pRet, int totalStackSize, PTR_PTR_Object pContinuationRet); // Number of routines in the Routines array. The last one is the target method to call. int NumRoutines; // Total stack size used for the arguments. int TotalStackSize; + bool HasContinuationRet; // Indicates whether the stub supports returning a continuation // This is a pointer to a helper function that invokes the target method. There are several // versions of this function, depending on the return type of the target method. InvokeFunctionPtr Invoke; // This is an array of routines that translate the arguments from the interpreter stack to the CPU registers and native stack. PCODE Routines[0]; - CallStubHeader(int numRoutines, PCODE *pRoutines, int totalStackSize, InvokeFunctionPtr pInvokeFunction) + CallStubHeader(int numRoutines, PCODE *pRoutines, int totalStackSize, bool hasContinuationRet, InvokeFunctionPtr pInvokeFunction) { LIMITED_METHOD_CONTRACT; NumRoutines = numRoutines; TotalStackSize = totalStackSize; Invoke = pInvokeFunction; + HasContinuationRet = hasContinuationRet; memcpy(Routines, pRoutines, NumRoutines * sizeof(PCODE)); } diff --git a/src/coreclr/vm/corelib.h b/src/coreclr/vm/corelib.h index 4fef8092eabb32..549df6f8113c8f 100644 --- a/src/coreclr/vm/corelib.h +++ b/src/coreclr/vm/corelib.h @@ -744,11 +744,21 @@ DEFINE_METHOD(ASYNC_HELPERS, RESTORE_CONTEXTS, RestoreContexts, No DEFINE_METHOD(ASYNC_HELPERS, RESTORE_CONTEXTS_ON_SUSPENSION, RestoreContextsOnSuspension, NoSig) DEFINE_METHOD(ASYNC_HELPERS, ASYNC_CALL_CONTINUATION, AsyncCallContinuation, NoSig) +#ifdef FEATURE_INTERPRETER +DEFINE_METHOD(ASYNC_HELPERS, RESUME_INTERPRETER_CONTINUATION, ResumeInterpreterContinuation, NoSig) +#endif + #ifdef TARGET_BROWSER DEFINE_METHOD(ASYNC_HELPERS, HANDLE_ASYNC_ENTRYPOINT, HandleAsyncEntryPoint, SM_TaskOfInt_RetInt) DEFINE_METHOD(ASYNC_HELPERS, HANDLE_ASYNC_ENTRYPOINT_VOID, HandleAsyncEntryPoint, SM_Task_RetVoid) #endif // TARGET_BROWSER +DEFINE_CLASS_U(CompilerServices, Continuation, ContinuationObject) +DEFINE_FIELD_U(Next, ContinuationObject, Next) +DEFINE_FIELD_U(ResumeInfo, ContinuationObject, ResumeInfo) +DEFINE_FIELD_U(Flags, ContinuationObject, Flags) +DEFINE_FIELD_U(State, ContinuationObject, State) + DEFINE_CLASS(SPAN_HELPERS, System, SpanHelpers) DEFINE_METHOD(SPAN_HELPERS, MEMSET, Fill, SM_RefByte_Byte_UIntPtr_RetVoid) DEFINE_METHOD(SPAN_HELPERS, MEMZERO, ClearWithoutReferences, SM_RefByte_UIntPtr_RetVoid) @@ -974,6 +984,11 @@ DEFINE_FIELD_U(_DONT_USE_InternalThread, ThreadBaseObject, m_InternalThread) DEFINE_FIELD_U(_priority, ThreadBaseObject, m_Priority) DEFINE_FIELD_U(_isDead, ThreadBaseObject, m_IsDead) DEFINE_FIELD_U(_isThreadPool, ThreadBaseObject, m_IsThreadPool) +DEFINE_FIELD_U(_executionContext, ThreadBaseObject, m_ExecutionContext) + + +DEFINE_CLASS_U(Threading, ExecutionContext, ExecutionContextObject) +DEFINE_FIELD_U(m_isFlowSuppressed, ExecutionContextObject, m_isFlowSuppressed) DEFINE_CLASS(DIRECTONTHREADLOCALDATA, Threading, Thread+DirectOnThreadLocalData) diff --git a/src/coreclr/vm/frames.h b/src/coreclr/vm/frames.h index 972b17ecb9ca00..5491e238918411 100644 --- a/src/coreclr/vm/frames.h +++ b/src/coreclr/vm/frames.h @@ -2478,6 +2478,7 @@ class InterpreterFrame : public FramedMethodFrame #if defined(HOST_AMD64) && defined(HOST_WINDOWS) , m_SSP(0) #endif + , m_continuation(NULL) { WRAPPER_NO_CONTRACT; Push(); @@ -2548,6 +2549,28 @@ class InterpreterFrame : public FramedMethodFrame m_isFaulting = isFaulting; } + void GcScanRoots_Impl(promote_func *fn, ScanContext* sc) + { + fn(dac_cast(dac_cast(&m_continuation)), sc, 0); + } + + void SetContinuation(OBJECTREF continuation) + { + LIMITED_METHOD_CONTRACT; + m_continuation = OBJECTREFToObject(continuation); + } + + OBJECTREF GetContinuation() + { + LIMITED_METHOD_CONTRACT; + return ObjectToOBJECTREF(m_continuation); + } + + PTR_PTR_Object GetContinuationPtr() + { + LIMITED_METHOD_CONTRACT; + return dac_cast(dac_cast(&m_continuation)); + } private: // The last known topmost interpreter frame in the InterpExecMethod belonging to // this InterpreterFrame. @@ -2561,6 +2584,7 @@ class InterpreterFrame : public FramedMethodFrame #ifndef TARGET_WASM TADDR m_SP; #endif // TARGET_WASM + PTR_Object m_continuation; }; #endif // FEATURE_INTERPRETER diff --git a/src/coreclr/vm/interpexec.cpp b/src/coreclr/vm/interpexec.cpp index ed673cf571728f..ead5ed940c43f6 100644 --- a/src/coreclr/vm/interpexec.cpp +++ b/src/coreclr/vm/interpexec.cpp @@ -9,6 +9,8 @@ #include "frames.h" #include "virtualcallstub.h" #include "comdelegate.h" +#include "gchelpers.inl" +#include "arraynative.inl" // for numeric_limits #include @@ -23,11 +25,11 @@ #endif // !TARGET_WASM // Call invoker helpers provided by platform. -void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target); +void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet); void InvokeUnmanagedMethod(MethodDesc *targetMethod, int8_t *pArgs, int8_t *pRet, PCODE callTarget); -void InvokeCalliStub(PCODE ftn, void* cookie, int8_t *pArgs, int8_t *pRet); +void InvokeCalliStub(PCODE ftn, void* cookie, int8_t *pArgs, int8_t *pRet, Object** pContinuationRet); void InvokeUnmanagedCalli(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet); -void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target); +void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet); extern "C" PCODE CID_VirtualOpenDelegateDispatch(TransitionBlock * pTransitionBlock); // Filter to ignore SEH exceptions representing C++ exceptions. @@ -93,7 +95,7 @@ void InvokeUnmanagedMethodWithTransition(MethodDesc *targetMethod, int8_t *stack { GCX_PREEMP_NO_DTOR(); // WASM-TODO: Handle unmanaged calling conventions - InvokeManagedMethod(pParam->targetMethod, pParam->pArgs, pParam->pRet, pParam->callTarget); + InvokeManagedMethod(pParam->targetMethod, pParam->pArgs, pParam->pRet, pParam->callTarget, NULL); GCX_PREEMP_NO_DTOR_END(); } PAL_EXCEPT_FILTER(IgnoreCppExceptionFilter) @@ -205,7 +207,7 @@ MethodDesc* GetTargetPInvokeMethodDesc(PCODE target) return NULL; } -void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target) +void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet) { CONTRACTL { @@ -243,15 +245,21 @@ void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE tar } } - pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize); + Object* continuationUnused = NULL; + if (!pHeader->HasContinuationRet) + { + pContinuationRet = &continuationUnused; + } + + pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize, pContinuationRet); } void InvokeUnmanagedMethod(MethodDesc *targetMethod, int8_t *pArgs, int8_t *pRet, PCODE callTarget) { - InvokeManagedMethod(targetMethod, pArgs, pRet, callTarget); + InvokeManagedMethod(targetMethod, pArgs, pRet, callTarget, NULL); } -void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target) +void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet) { CONTRACTL { @@ -276,7 +284,14 @@ void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, in memcpy(actualCallStub, stubHeaderTemplate, templateSize); CallStubHeader *pHeader = (CallStubHeader*)actualCallStub; pHeader->SetTarget(target); // The method to call - pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize); + + Object* continuationUnused = NULL; + if (!pHeader->HasContinuationRet) + { + pContinuationRet = &continuationUnused; + } + + pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize, pContinuationRet); } void InvokeUnmanagedCalli(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet) @@ -298,11 +313,12 @@ void InvokeUnmanagedCalli(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet) memcpy(actualCallStub, stubHeaderTemplate, templateSize); CallStubHeader *pHeader = (CallStubHeader*)actualCallStub; pHeader->SetTarget(ftn); // The method to call + Object* continuationUnused = NULL; - pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize); + pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize, &continuationUnused); } -void InvokeCalliStub(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet) +void InvokeCalliStub(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet, Object** pContinuationRet) { CONTRACTL { @@ -321,7 +337,14 @@ void InvokeCalliStub(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet) memcpy(actualCallStub, stubHeaderTemplate, templateSize); CallStubHeader *pHeader = (CallStubHeader*)actualCallStub; pHeader->SetTarget(ftn); // The method to call - pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize); + + Object* continuationUnused = NULL; + if (!pHeader->HasContinuationRet) + { + pContinuationRet = &continuationUnused; + } + + pHeader->Invoke(pHeader->Routines, pArgs, pRet, pHeader->TotalStackSize, pContinuationRet); } void* GetCookieForCalliSig(MetaSig metaSig) @@ -344,7 +367,7 @@ CallStubHeader *CreateNativeToInterpreterCallStub(InterpMethod* pInterpMethod) GCX_PREEMP(); AllocMemTracker amTracker; - pHeader = callStubGenerator.GenerateCallStub(pInterpMethod->methodDesc, &amTracker, false /* interpreterToNative */); + pHeader = callStubGenerator.GenerateCallStub(pInterpMethod->methodHnd, &amTracker, false /* interpreterToNative */); if (InterlockedCompareExchangeT(&pInterpMethod->pCallStub, pHeader, NULL) == NULL) { @@ -385,7 +408,7 @@ void DBG_PrintInterpreterStack() InterpMethodContextFrame* cxtFrame = pInterpFrame->GetTopInterpMethodContextFrame(); while (cxtFrame != NULL) { - MethodDesc* currentMD = cxtFrame->startIp->Method->methodDesc; + MethodDesc* currentMD = cxtFrame->startIp->Method->methodHnd; size_t irOffset = ((size_t)cxtFrame->ip - (size_t)(&cxtFrame->startIp[1])) / sizeof(size_t); fprintf(stderr, "%4d) %s::%s, IR_%04zx\n", @@ -404,6 +427,7 @@ typedef void* (*HELPER_FTN_BOX_UNBOX)(MethodTable*, void*); typedef Object* (*HELPER_FTN_NEWARR)(MethodTable*, intptr_t); typedef void* (*HELPER_FTN_P_PP)(void*, void*); typedef void (*HELPER_FTN_V_PPP)(void*, void*, void*); +typedef void* (*HELPER_FTN_P_PPIP)(void*, void*, int32_t, void*); typedef void (*HELPER_FTN_V_PP)(void*, void*); InterpThreadContext::InterpThreadContext() @@ -730,6 +754,93 @@ void* DoGenericLookup(void* genericVarAsPtr, InterpGenericLookup* pLookup) return result; } +void AsyncHelpers_ResumeInterpreterContinuation(QCall::ObjectHandleOnStack cont, uint8_t* resultStorage) +{ + QCALL_CONTRACT; + + BEGIN_QCALL; + + GCX_COOP(); + + Thread *pThread = GetThread(); + InterpThreadContext *threadContext = pThread->GetInterpThreadContext(); + if (threadContext == nullptr || threadContext->pStackStart == nullptr) + { + COMPlusThrow(kOutOfMemoryException); + } + int8_t *sp = threadContext->pStackPointer; + + // This construct ensures that the InterpreterFrame is always stored at a higher address than the + // InterpMethodContextFrame. This is important for the stack walking code. + struct Frames + { + InterpMethodContextFrame interpMethodContextFrame = {0}; + InterpreterFrame interpreterFrame; + + Frames(TransitionBlock* pTransitionBlock) + : interpreterFrame(pTransitionBlock, &interpMethodContextFrame) + { + } + } + frames(NULL); + + CONTINUATIONREF contRef = (CONTINUATIONREF)ObjectToOBJECTREF(cont.Get()); + NULL_CHECK(contRef); + + // We are working with an interpreter async continuation, move things around to get the InterpAsyncSuspendData + size_t resumeDataOffset = offsetof(InterpAsyncSuspendData, resumeInfo); + InterpAsyncSuspendData* pSuspendData = (InterpAsyncSuspendData*)(void*)(((uint8_t*)contRef->GetResumeInfo()) - resumeDataOffset); + _ASSERTE(&pSuspendData->resumeInfo.Resume == contRef->GetResumeInfo()); + + // All the incoming args and locals should be zeroed out, except for the continuation argument + InterpMethod *pMethod = pSuspendData->methodStartIP->Method; + memset(sp, 0, pMethod->allocaSize); + *(CONTINUATIONREF*)(sp + pSuspendData->continuationArgOffset) = contRef; + + frames.interpMethodContextFrame.startIp = pSuspendData->methodStartIP; + frames.interpMethodContextFrame.pStack = sp; + + // Figure out the exact sizes to work with, since we may be copying into a Task + // and not only into another interpreter continuation + int32_t returnValueSize = pSuspendData->asyncMethodReturnTypePrimitiveSize; + if (pSuspendData->asyncMethodReturnType != NULL) + { + if (pSuspendData->asyncMethodReturnType->IsValueType()) + returnValueSize = pSuspendData->asyncMethodReturnType->GetNumInstanceFieldBytes(); + else + returnValueSize = sizeof(OBJECTREF); + } + + void* returnValueLocation = alloca(returnValueSize < (int32_t)sizeof(StackVal) ? (int32_t)sizeof(StackVal) : returnValueSize); + memset(returnValueLocation, 0, returnValueSize); + + frames.interpMethodContextFrame.pRetVal = (int8_t*)returnValueLocation; + + InterpExecMethod(&frames.interpreterFrame, &frames.interpMethodContextFrame, threadContext); + + if (frames.interpreterFrame.GetContinuation() == NULL) + { + // We had a normal return, so copy out the return value + if (returnValueSize > 0) + { + if (pSuspendData->asyncMethodReturnType != NULL && pSuspendData->asyncMethodReturnType->ContainsGCPointers()) + { + // GC refs need to be written with write barriers + memmoveGCRefs(resultStorage, returnValueLocation, returnValueSize); + } + else + { + memcpyNoGCRefs(resultStorage, returnValueLocation, returnValueSize); + } + } + } + + cont.Set(frames.interpreterFrame.GetContinuation()); + frames.interpreterFrame.Pop(); + + END_QCALL; +} + void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFrame *pFrame, InterpThreadContext *pThreadContext, ExceptionClauseArgs *pExceptionClauseArgs) { CONTRACTL @@ -1895,6 +2006,10 @@ void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFr ip += 4; \ } while (0) + case INTOP_CZERO_I: + LOCAL_VAR(ip[1], int32_t) = LOCAL_VAR(ip[2], intptr_t) != 0; + ip += 4; + break; case INTOP_CEQ_I4: LOCAL_VAR(ip[1], int32_t) = LOCAL_VAR(ip[2], int32_t) == LOCAL_VAR(ip[3], int32_t); ip += 4; @@ -2512,7 +2627,7 @@ void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFr goto CALL_INTERP_METHOD; } #endif // FEATURE_PORTABLE_ENTRYPOINTS - InvokeCalliStub(calliFunctionPointer, cookie, callArgsAddress, returnValueAddress); + InvokeCalliStub(calliFunctionPointer, cookie, callArgsAddress, returnValueAddress, pInterpreterFrame->GetContinuationPtr()); } break; @@ -2675,7 +2790,7 @@ void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFr // Save current execution state for when we return from called method pFrame->ip = ip; - InvokeDelegateInvokeMethod(targetMethod, callArgsAddress, returnValueAddress, targetAddress); + InvokeDelegateInvokeMethod(targetMethod, callArgsAddress, returnValueAddress, targetAddress, pInterpreterFrame->GetContinuationPtr()); break; } @@ -2735,7 +2850,7 @@ void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFr { // If we didn't get the interpreter code pointer setup, then this is a method we need to invoke as a compiled method. // Interpreter-FIXME: Implement tailcall via helpers, see https://github.com/dotnet/runtime/blob/main/docs/design/features/tailcalls-with-helpers.md - InvokeManagedMethod(targetMethod, callArgsAddress, returnValueAddress, (PCODE)NULL); + InvokeManagedMethod(targetMethod, callArgsAddress, returnValueAddress, (PCODE)NULL, pInterpreterFrame->GetContinuationPtr()); break; } } @@ -3618,6 +3733,312 @@ do \ case INTOP_THROW_PNSE: COMPlusThrow(kPlatformNotSupportedException); break; + + case INTOP_HANDLE_CONTINUATION_GENERIC: + case INTOP_HANDLE_CONTINUATION: + { + int32_t helperOffset = *ip == INTOP_HANDLE_CONTINUATION_GENERIC ? 4 : 3; + int32_t ipAdjust = *ip == INTOP_HANDLE_CONTINUATION_GENERIC ? 5 : 4; + int32_t suspendDataOffset = *ip == INTOP_HANDLE_CONTINUATION_GENERIC ? 3 : 2; + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[suspendDataOffset]]; + + // Zero out locals that need zeroing + InterpIntervalMapEntry *pZeroLocalsEntry = pAsyncSuspendData->zeroedLocalsIntervals; + while (pZeroLocalsEntry->countBytes != 0) + { + memset(LOCAL_VAR_ADDR(pZeroLocalsEntry->startOffset, uint8_t), 0, pZeroLocalsEntry->countBytes); + pZeroLocalsEntry++; + } + + if (pInterpreterFrame->GetContinuation() == NULL) + { + // No continuation to handle. + LOCAL_VAR(ip[1], void*) = NULL; // We don't allocate a continuation here + ip += ipAdjust; // (4 or 5 for this opcode) + break; + } + MethodTable *pContinuationType = pAsyncSuspendData->continuationTypeHnd; + + MethodDesc *pILTargetMethod = NULL; + HELPER_FTN_P_PP helperFtn = NULL; + HELPER_FTN_P_PPIP helperFtnGeneric = NULL; + if (*ip == INTOP_HANDLE_CONTINUATION_GENERIC) + { + helperFtnGeneric = GetPossiblyIndirectHelper(pMethod, ip[helperOffset], &pILTargetMethod); + } + else + { + helperFtn = GetPossiblyIndirectHelper(pMethod, ip[helperOffset], &pILTargetMethod); + } + if (pILTargetMethod != NULL) + { + returnOffset = ip[1]; + callArgsOffset = pMethod->allocaSize; + + // Pass argument to the target method + LOCAL_VAR(callArgsOffset, OBJECTREF) = pInterpreterFrame->GetContinuation(); + LOCAL_VAR(callArgsOffset + INTERP_STACK_SLOT_SIZE, MethodTable*) = pContinuationType; + if (*ip == INTOP_HANDLE_CONTINUATION_GENERIC) + { + LOCAL_VAR(callArgsOffset + 2 * INTERP_STACK_SLOT_SIZE, int32_t) = pAsyncSuspendData->keepAliveOffset; + LOCAL_VAR(callArgsOffset + 3 * INTERP_STACK_SLOT_SIZE, uintptr_t) = LOCAL_VAR(ip[2], uintptr_t); + } + + pInterpreterFrame->SetContinuation(NULL); + targetMethod = pILTargetMethod; + ip += ipAdjust; + goto CALL_INTERP_METHOD; + } + + OBJECTREF chainedContinuation = pInterpreterFrame->GetContinuation(); + pInterpreterFrame->SetContinuation(NULL); + OBJECTREF* pDest = LOCAL_VAR_ADDR(ip[1], OBJECTREF); + if (*ip == INTOP_HANDLE_CONTINUATION_GENERIC) + { + uintptr_t context = LOCAL_VAR(ip[2], uintptr_t); + ip += ipAdjust; + *pDest = ObjectToOBJECTREF((Object*)helperFtnGeneric(OBJECTREFToObject(chainedContinuation), pContinuationType, pAsyncSuspendData->keepAliveOffset, (void*)context)); + } + else + { + ip += ipAdjust; + *pDest = ObjectToOBJECTREF((Object*)helperFtn(OBJECTREFToObject(chainedContinuation), pContinuationType)); + } + break; + } + + case INTOP_CAPTURE_CONTEXT_ON_SUSPEND: + { + CONTINUATIONREF continuation = LOCAL_VAR(ip[2], CONTINUATIONREF); + + if (continuation == NULL) + { + LOCAL_VAR(ip[1], CONTINUATIONREF) = continuation; + // No continuation to handle. + ip += 4; + break; + } + + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[3]]; + + THREADBASEREF threadBase = ((THREADBASEREF)GetThread()->GetExposedObject()); + continuation = LOCAL_VAR(ip[2], CONTINUATIONREF); + OBJECTREF executionContext = threadBase->GetExecutionContext(); + if (executionContext != NULL && ((EXECUTIONCONTEXTREF)executionContext)->IsFlowSuppressed()) + { + executionContext = NULL; + } + SetObjectReference((OBJECTREF *)((uint8_t*)(OBJECTREFToObject(continuation)) + pAsyncSuspendData->offsetIntoContinuationTypeForExecutionContext), executionContext); + continuation->SetFlags(pAsyncSuspendData->flags); + + if (pAsyncSuspendData->flags & CORINFO_CONTINUATION_HAS_CONTINUATION_CONTEXT) + { + MethodDesc *captureSyncContextMethod = pAsyncSuspendData->captureSyncContextMethod; + int32_t *flagsAddress = continuation->GetFlagsAddress(); + size_t continuationOffset = OFFSETOF__CORINFO_Continuation__data; + uint8_t *pContinuationData = (uint8_t*)OBJECTREFToObject(continuation) + continuationOffset; + if (pAsyncSuspendData->flags & CORINFO_CONTINUATION_HAS_EXCEPTION) + { + pContinuationData += sizeof(OBJECTREF); + } + + returnOffset = ip[1]; + callArgsOffset = pMethod->allocaSize; + + // Pass argument to the target method + LOCAL_VAR(callArgsOffset, void*) = pContinuationData; + LOCAL_VAR(callArgsOffset + INTERP_STACK_SLOT_SIZE, int32_t*) = flagsAddress; + targetMethod = captureSyncContextMethod; + ip += 4; + goto CALL_INTERP_METHOD; + } + else + { + ip += 4; + break; + } + } + + case INTOP_RESTORE_CONTEXTS_ON_SUSPEND: + { + CONTINUATIONREF newContinuation = LOCAL_VAR(ip[2], CONTINUATIONREF); + CONTINUATIONREF continuationArg = LOCAL_VAR(ip[3], CONTINUATIONREF); + + int32_t resumed = continuationArg != NULL; + if ((newContinuation == NULL) || resumed) + { + LOCAL_VAR(ip[1], CONTINUATIONREF) = newContinuation; + // No continuation to handle. + ip += 6; + break; + } + + OBJECTREF executionContext = LOCAL_VAR(ip[4], OBJECTREF); + OBJECTREF syncContext = LOCAL_VAR(ip[4] + INTERP_STACK_SLOT_SIZE, OBJECTREF); + + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[5]]; + MethodDesc *restoreContextsMethod = pAsyncSuspendData->restoreContextsMethod; + + returnOffset = ip[1]; + callArgsOffset = pMethod->allocaSize; + // Pass argument to the target method + LOCAL_VAR(callArgsOffset, int32_t) = resumed; + LOCAL_VAR(callArgsOffset + INTERP_STACK_SLOT_SIZE, OBJECTREF) = executionContext; + LOCAL_VAR(callArgsOffset + INTERP_STACK_SLOT_SIZE * 2, OBJECTREF) = syncContext; + targetMethod = restoreContextsMethod; + ip += 6; + goto CALL_INTERP_METHOD; + } + + case INTOP_GET_CONTINUATION: + { + LOCAL_VAR(ip[1], OBJECTREF) = pInterpreterFrame->GetContinuation(); + pInterpreterFrame->SetContinuation(NULL); + ip += 2; + break; + } + + case INTOP_SET_CONTINUATION: + { + pInterpreterFrame->SetContinuation(LOCAL_VAR(ip[1], CONTINUATIONREF)); + ip += 2; + break; + } + + case INTOP_SET_CONTINUATION_NULL: + { + pInterpreterFrame->SetContinuation(NULL); + ip += 1; + break; + } + + case INTOP_HANDLE_CONTINUATION_SUSPEND: + { + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[2]]; + CONTINUATIONREF continuation = LOCAL_VAR(ip[1], CONTINUATIONREF); + + if (continuation == NULL) + { + // No continuation to handle. + ip += 3 + 2; // (3 for this opcode + 2 for the continuation resume which follows) + break; + } + ip += 3; + + // copy locals that need to move to the continuation object + size_t continuationOffset = OFFSETOF__CORINFO_Continuation__data; + uint8_t *pContinuationDataStart = continuation->GetResultStorage(); + uint8_t *pContinuationData = pContinuationDataStart; + size_t bytesTotal = 0; + InterpIntervalMapEntry *pCopyEntry = pAsyncSuspendData->liveLocalsIntervals; + if (pCopyEntry->countBytes > 0) + { + GCHeapMemoryBarrier(); + while (pCopyEntry->countBytes != 0) + { + InlinedForwardGCSafeCopyHelper(pContinuationData, LOCAL_VAR_ADDR(pCopyEntry->startOffset, uint8_t), pCopyEntry->countBytes); + bytesTotal += pCopyEntry->countBytes; + pContinuationData += pCopyEntry->countBytes; + pCopyEntry++; + } + InlinedSetCardsAfterBulkCopyHelper((Object**)pContinuationDataStart, bytesTotal); + } + + int32_t returnValueSize = pAsyncSuspendData->asyncMethodReturnTypePrimitiveSize; + if (pAsyncSuspendData->asyncMethodReturnType != NULL) + { + if (pAsyncSuspendData->asyncMethodReturnType->IsValueType()) + returnValueSize = pAsyncSuspendData->asyncMethodReturnType->GetNumInstanceFieldBytes(); + else + returnValueSize = sizeof(OBJECTREF); + } + + if (returnValueSize > 0) + { + if (returnValueSize > INTERP_STACK_SLOT_SIZE) + { + memset(pFrame->pRetVal, 0, returnValueSize); + } + else + { + *(int64_t*)pFrame->pRetVal = 0; + } + } + + continuation->SetState((int32_t)((uint8_t*)ip - (uint8_t*)pFrame->startIp)); + _ASSERTE(pAsyncSuspendData->methodStartIP != 0); + continuation->SetResumeInfo(&pAsyncSuspendData->resumeInfo); + pInterpreterFrame->SetContinuation(continuation); + goto EXIT_FRAME; + } + + case INTOP_HANDLE_CONTINUATION_RESUME: + { + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[1]]; + CONTINUATIONREF continuation = (CONTINUATIONREF)ObjectToOBJECTREF(*(Object**)(stack + pAsyncSuspendData->continuationArgOffset)); + _ASSERTE(pInterpreterFrame->GetContinuation() == NULL); + + // The INTOP_CHECK_FOR_CONTINUATION opcode will have called to restore the execution context already + // Now copy the locals + + // copy locals that need to move from the continuation object + uint8_t *pContinuationData = continuation->GetResultStorage(); + InterpIntervalMapEntry *pCopyEntry = pAsyncSuspendData->liveLocalsIntervals; + while (pCopyEntry->countBytes != 0) + { + memcpy(LOCAL_VAR_ADDR(pCopyEntry->startOffset, uint8_t), pContinuationData, pCopyEntry->countBytes); + pContinuationData += pCopyEntry->countBytes; + pCopyEntry++; + } + + if (pAsyncSuspendData->flags & CORINFO_CONTINUATION_HAS_EXCEPTION) + { + // Throw exception if needed + OBJECTREF exception = *continuation->GetExceptionObjectStorage(); + + if (exception != NULL) + { + GetThread()->GetExceptionState()->SetRaisingForeignException(); + pInterpreterFrame->SetIsFaulting(true); + DispatchManagedException(exception); + UNREACHABLE(); + } + } + + ip += 2; + break; + } + + case INTOP_CHECK_FOR_CONTINUATION: + { + CONTINUATIONREF continuation = LOCAL_VAR(ip[1], CONTINUATIONREF); + _ASSERTE(pInterpreterFrame->GetContinuation() == NULL); + if (continuation != NULL) + { + // A continuation is present, begin the restoration process + int32_t state = continuation->GetState(); + ip = (int32_t*)((uint8_t*)pFrame->startIp + state); + + // Now we have an IP to where we should resume execution. This should be an INTOP_HANDLE_CONTINUATION_RESUME opcode + // And before it should be an INTOP_HANDLE_CONTINUATION_SUSPEND opcode + _ASSERTE(*ip == INTOP_HANDLE_CONTINUATION_RESUME); + _ASSERTE(*(ip-3) == INTOP_HANDLE_CONTINUATION_SUSPEND); + InterpAsyncSuspendData *pAsyncSuspendData = (InterpAsyncSuspendData*)pMethod->pDataItems[ip[1]]; + + returnOffset = pMethod->allocaSize; + callArgsOffset = pMethod->allocaSize; + + OBJECTREF execContext = ObjectToOBJECTREF(*(Object**)(((uint8_t*)OBJECTREFToObject(continuation)) + pAsyncSuspendData->offsetIntoContinuationTypeForExecutionContext)); + + // We need to call the RestoreExecutionContext helper method + LOCAL_VAR(callArgsOffset, OBJECTREF) = execContext; + + targetMethod = pAsyncSuspendData->restoreExecutionContextMethod; + goto CALL_INTERP_METHOD; + } + ip += 3; + break; + } default: EEPOLICY_HANDLE_FATAL_ERROR_WITH_MESSAGE(COR_E_EXECUTIONENGINE, W("Unimplemented or invalid interpreter opcode\n")); break; @@ -3673,6 +4094,9 @@ do \ EXIT_FRAME: + // Exit the current frame, MAKE CERTAIN not to trigger any GC between here and the return, since the interpreter + // async resumption logic depends on not triggering a GC here for correctness. + // Interpreter-TODO: Don't run PopInfo on the main return path, Add RET_LOCALLOC instead pThreadContext->frameDataAllocator.PopInfo(pFrame); if (pFrame->pParent && pFrame->pParent->ip) diff --git a/src/coreclr/vm/interpexec.h b/src/coreclr/vm/interpexec.h index fddebc41b86c72..f24ab63781bbb2 100644 --- a/src/coreclr/vm/interpexec.h +++ b/src/coreclr/vm/interpexec.h @@ -78,6 +78,7 @@ struct ExceptionClauseArgs }; void InterpExecMethod(InterpreterFrame *pInterpreterFrame, InterpMethodContextFrame *pFrame, InterpThreadContext *pThreadContext, ExceptionClauseArgs *pExceptionClauseArgs = NULL); +extern "C" void AsyncHelpers_ResumeInterpreterContinuation(QCall::ObjectHandleOnStack cont, uint8_t* resultStorage); extern "C" void LookupMethodByName(const char* fullQualifiedTypeName, const char* methodName, MethodDesc** ppMD); extern "C" void ExecuteInterpretedMethodFromUnmanaged(MethodDesc* pMD, int8_t* args, size_t argSize, int8_t* ret, PCODE callerIp); diff --git a/src/coreclr/vm/jitinterface.cpp b/src/coreclr/vm/jitinterface.cpp index 76bf7390d5848a..3e62ec13c5ba14 100644 --- a/src/coreclr/vm/jitinterface.cpp +++ b/src/coreclr/vm/jitinterface.cpp @@ -11332,6 +11332,12 @@ LPVOID CInterpreterJitInfo::GetCookieForInterpreterCalliSig(CORINFO_SIG_INFO* sz Module* mod = GetModule(szMetaSig->scope); MetaSig sig(szMetaSig->pSig, szMetaSig->cbSig, mod, &typeContext); + + if (szMetaSig->isAsyncCall()) + sig.SetIsAsyncCall(); + + _ASSERTE(szMetaSig->isAsyncCall() == sig.IsAsyncCall()); + result = GetCookieForCalliSig(sig); EE_TO_JIT_TRANSITION(); @@ -14606,6 +14612,27 @@ static Signature BuildResumptionStubCalliSignature(MetaSig& msig, MethodTable* m return AllocateSignature(alloc, sigBuilder, pamTracker); } +#ifdef FEATURE_INTERPRETER +CORINFO_METHOD_HANDLE CInterpreterJitInfo::getAsyncResumptionStub(void** entryPoint) +{ + CONTRACTL{ + THROWS; + GC_TRIGGERS; + MODE_PREEMPTIVE; + } CONTRACTL_END; + + MethodDesc *pMDResumeFunc = NULL; + JIT_TO_EE_TRANSITION(); + + pMDResumeFunc = CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__RESUME_INTERPRETER_CONTINUATION); + *entryPoint = (void*)pMDResumeFunc->GetMultiCallableAddrOfCode(); + + EE_TO_JIT_TRANSITION(); + + return (CORINFO_METHOD_HANDLE)pMDResumeFunc; +} +#endif // FEATURE_INTERPRETER + CORINFO_METHOD_HANDLE CEEJitInfo::getAsyncResumptionStub(void** entryPoint) { CONTRACTL{ @@ -14614,6 +14641,9 @@ CORINFO_METHOD_HANDLE CEEJitInfo::getAsyncResumptionStub(void** entryPoint) MODE_PREEMPTIVE; } CONTRACTL_END; + MethodDesc* result = NULL; + JIT_TO_EE_TRANSITION(); + MethodDesc* md = m_pMethodBeingCompiled; LoaderAllocator* loaderAlloc = md->GetLoaderAllocator(); @@ -14729,7 +14759,7 @@ CORINFO_METHOD_HANDLE CEEJitInfo::getAsyncResumptionStub(void** entryPoint) pCode->EmitLDLOC(newContinuationLoc); pCode->EmitRET(); - MethodDesc* result = + result = ILStubCache::CreateAndLinkNewILStubMethodDesc( md->GetLoaderAllocator(), md->GetLoaderModule()->GetILStubCache()->GetOrCreateStubMethodTable(md->GetLoaderModule()), @@ -14776,6 +14806,9 @@ CORINFO_METHOD_HANDLE CEEJitInfo::getAsyncResumptionStub(void** entryPoint) #endif *entryPoint = (void*)result->GetMultiCallableAddrOfCode(); + + EE_TO_JIT_TRANSITION(); + return CORINFO_METHOD_HANDLE(result); } diff --git a/src/coreclr/vm/jitinterface.h b/src/coreclr/vm/jitinterface.h index d71dc3d59304b5..24dcaace8988ed 100644 --- a/src/coreclr/vm/jitinterface.h +++ b/src/coreclr/vm/jitinterface.h @@ -19,6 +19,8 @@ #endif // !TARGET_UNIX #include "pgo.h" +class ILCodeStream; + enum StompWriteBarrierCompletionAction { SWB_PASS = 0x0, @@ -980,6 +982,8 @@ class CInterpreterJitInfo final : public CEECodeGenInfo void SetDebugInfo(PTR_BYTE pDebugInfo) override; LPVOID GetCookieForInterpreterCalliSig(CORINFO_SIG_INFO* szMetaSig) override; + + virtual CORINFO_METHOD_HANDLE getAsyncResumptionStub(void** entryPoint) override final; }; #endif // FEATURE_INTERPRETER diff --git a/src/coreclr/vm/object.h b/src/coreclr/vm/object.h index 66e7b779d99dc8..f210472f87e34e 100644 --- a/src/coreclr/vm/object.h +++ b/src/coreclr/vm/object.h @@ -1156,8 +1156,26 @@ class SynchronizationContextObject: public Object }; +class ExecutionContextObject: public Object +{ + friend class CoreLibBinder; +private: + // These field are also defined in the managed representation. (SecurityContext.cs)If you + // add or change these field you must also change the managed code so that + // it matches these. This is necessary so that the object is the proper + // size. + OBJECTREF m_localValues; + OBJECTREF m_localChangeNotifications; + CLR_BOOL m_isFlowSuppressed; +public: + bool IsFlowSuppressed() const + { + LIMITED_METHOD_CONTRACT; + return m_isFlowSuppressed; + } +}; typedef DPTR(class CultureInfoBaseObject) PTR_CultureInfoBaseObject; @@ -1170,6 +1188,7 @@ typedef REF ARRAYBASEREF; #else typedef SynchronizationContextObject* SYNCHRONIZATIONCONTEXTREF; typedef CultureInfoBaseObject* CULTUREINFOBASEREF; +typedef ExecutionContextObject* EXECUTIONCONTEXTREF; typedef PTR_ArrayBase ARRAYBASEREF; #endif @@ -1274,6 +1293,12 @@ class ThreadBaseObject : public Object return m_Name; } + OBJECTREF GetExecutionContext() + { + LIMITED_METHOD_CONTRACT; + return m_ExecutionContext; + } + OBJECTREF GetSynchronizationContext() { LIMITED_METHOD_CONTRACT; @@ -1423,6 +1448,10 @@ class WeakReferenceObject : public Object uintptr_t m_taggedHandle; }; + +typedef DPTR(class ContinuationObject) PTR_ContinuationObject; +class ContinuationObject; + #ifdef USE_CHECKED_OBJECTREFS typedef REF REFLECTMODULEBASEREF; @@ -1441,6 +1470,10 @@ typedef REF ASSEMBLYLOADCONTEXTREF; typedef REF ASSEMBLYNAMEREF; +typedef REF THREADBASEREF; + +typedef REF CONTINUATIONREF; + inline ARG_SLOT ObjToArgSlot(OBJECTREF objRef) { LIMITED_METHOD_CONTRACT; @@ -1483,6 +1516,8 @@ typedef PTR_ThreadBaseObject THREADBASEREF; typedef PTR_AssemblyBaseObject ASSEMBLYREF; typedef PTR_AssemblyLoadContextBaseObject ASSEMBLYLOADCONTEXTREF; typedef PTR_AssemblyNameBaseObject ASSEMBLYNAMEREF; +typedef PTR_ThreadBaseObject THREADBASEREF; +typedef PTR_ContinuationObject CONTINUATIONREF; #define ObjToArgSlot(objref) ((ARG_SLOT)(SIZE_T)(objref)) #define ArgSlotToObj(s) ((OBJECTREF)(SIZE_T)(s)) @@ -2140,6 +2175,98 @@ class GenericCacheStruct int32_t _maxCacheSize; }; +class ContinuationObject : public Object +{ + friend class CoreLibBinder; + + public: + CorInfoContinuationFlags GetFlags() const + { + LIMITED_METHOD_CONTRACT; + return (CorInfoContinuationFlags)Flags; + } + + void SetFlags(CorInfoContinuationFlags flags) + { + LIMITED_METHOD_CONTRACT; + Flags = (int32_t)flags; + } + + void SetResumeInfo(void* resumeInfo) + { + LIMITED_METHOD_CONTRACT; + ResumeInfo = resumeInfo; + } + + void* GetResumeInfo() const + { + LIMITED_METHOD_CONTRACT; + return ResumeInfo; + } + + void SetState(int32_t state) + { + LIMITED_METHOD_CONTRACT; + State = state; + } + + int32_t GetState() const + { + LIMITED_METHOD_CONTRACT; + return State; + } + + PTR_BYTE GetResultStorage() + { + LIMITED_METHOD_CONTRACT; + PTR_BYTE dataAddress = dac_cast((dac_cast(this) + OFFSETOF__CORINFO_Continuation__data)); + if (GetFlags() & CORINFO_CONTINUATION_HAS_OSR_ILOFFSET) + { + dataAddress += sizeof(void*); + } + if (GetFlags() & CORINFO_CONTINUATION_HAS_EXCEPTION) + { + dataAddress += sizeof(void*); + } + if (GetFlags() & CORINFO_CONTINUATION_HAS_CONTINUATION_CONTEXT) + { + dataAddress += sizeof(void*); + } + return dataAddress; + } + + PTR_OBJECTREF GetExceptionObjectStorage() + { + LIMITED_METHOD_CONTRACT; + _ASSERTE((GetFlags() & CORINFO_CONTINUATION_HAS_EXCEPTION)); + + PTR_BYTE dataAddress = dac_cast((dac_cast(this) + OFFSETOF__CORINFO_Continuation__data)); + if (GetFlags() & CORINFO_CONTINUATION_HAS_OSR_ILOFFSET) + { + dataAddress += sizeof(void*); + } + return dac_cast(dataAddress); + } + +#ifndef DACCESS_COMPILE + int32_t* GetFlagsAddress() + { + LIMITED_METHOD_CONTRACT; + return (int32_t*)&Flags; + } +#endif // DACCESS_COMPILE + +private: + // README: + // If you modify the order of these fields, make sure to update the definition in + // BCL for this object. + + CONTINUATIONREF Next; + void* ResumeInfo; + int32_t Flags; + int32_t State; +}; + // This class corresponds to Exception on the managed side. typedef DPTR(class ExceptionObject) PTR_ExceptionObject; #include "pshpack4.h" diff --git a/src/coreclr/vm/precode.cpp b/src/coreclr/vm/precode.cpp index aaf3d1ee485edf..031c0a06f57cfb 100644 --- a/src/coreclr/vm/precode.cpp +++ b/src/coreclr/vm/precode.cpp @@ -190,7 +190,7 @@ TADDR InterpreterPrecode::GetMethodDesc() LIMITED_METHOD_DAC_CONTRACT; InterpByteCodeStart* pInterpreterCode = dac_cast(GetData()->ByteCodeAddr); - return dac_cast(pInterpreterCode->Method->methodDesc); + return dac_cast(pInterpreterCode->Method->methodHnd); } #endif // FEATURE_INTERPRETER diff --git a/src/coreclr/vm/prestub.cpp b/src/coreclr/vm/prestub.cpp index c8cf921e1c2457..9702922d4b452a 100644 --- a/src/coreclr/vm/prestub.cpp +++ b/src/coreclr/vm/prestub.cpp @@ -1975,6 +1975,20 @@ extern "C" void* STDCALL ExecuteInterpretedMethod(TransitionBlock* pTransitionBl InterpExecMethod(&frames.interpreterFrame, &frames.interpMethodContextFrame, threadContext); + ArgumentRegisters *pArgumentRegisters = (ArgumentRegisters*)(((uint8_t*)pTransitionBlock) + TransitionBlock::GetOffsetOfArgumentRegisters()); + +#if defined(TARGET_AMD64) + pArgumentRegisters->RCX = (INT_PTR)*frames.interpreterFrame.GetContinuationPtr(); +#elif defined(TARGET_ARM64) + pArgumentRegisters->x[2] = (INT64)*frames.interpreterFrame.GetContinuationPtr(); +#elif defined(TARGET_RISCV64) + pArgumentRegisters->a[2] = (INT64)*frames.interpreterFrame.GetContinuationPtr(); +#elif defined(TARGET_WASM) + // We do not yet have an ABI for WebAssembly native code to handle here. +#else + #error Unsupported architecture +#endif + frames.interpreterFrame.Pop(); return frames.interpMethodContextFrame.pRetVal; diff --git a/src/coreclr/vm/qcallentrypoints.cpp b/src/coreclr/vm/qcallentrypoints.cpp index eeb41a5efa2b32..0adc704306eb3d 100644 --- a/src/coreclr/vm/qcallentrypoints.cpp +++ b/src/coreclr/vm/qcallentrypoints.cpp @@ -79,6 +79,10 @@ #include "entrypoints.h" #endif // TARGET_BROWSER +#ifdef FEATURE_INTERPRETER +#include "interpexec.h" +#endif // FEATURE_INTERPRETER + static const Entry s_QCall[] = { DllImportEntry(ArgIterator_Init) @@ -546,6 +550,9 @@ static const Entry s_QCall[] = DllImportEntry(SystemJS_ScheduleTimer) DllImportEntry(SystemJS_ScheduleBackgroundJob) #endif // TARGET_BROWSER +#ifdef FEATURE_INTERPRETER + DllImportEntry(AsyncHelpers_ResumeInterpreterContinuation) +#endif // FEATURE_INTERPRETER }; const void* QCallResolveDllImport(const char* name) diff --git a/src/coreclr/vm/riscv64/asmconstants.h b/src/coreclr/vm/riscv64/asmconstants.h index d49e9b32b046dc..121918c7502715 100644 --- a/src/coreclr/vm/riscv64/asmconstants.h +++ b/src/coreclr/vm/riscv64/asmconstants.h @@ -248,7 +248,7 @@ ASMCONSTANTS_C_ASSERT(OFFSETOF__Thread__m_pInterpThreadContext == offsetof(Threa #define OFFSETOF__InterpThreadContext__pStackPointer 0x10 ASMCONSTANTS_C_ASSERT(OFFSETOF__InterpThreadContext__pStackPointer == offsetof(InterpThreadContext, pStackPointer)) -#define OFFSETOF__CallStubHeader__Routines 0x10 +#define OFFSETOF__CallStubHeader__Routines 0x18 ASMCONSTANTS_C_ASSERT(OFFSETOF__CallStubHeader__Routines == offsetof(CallStubHeader, Routines)) #define SIZEOF__TransitionBlock 0xC0 diff --git a/src/coreclr/vm/riscv64/asmhelpers.S b/src/coreclr/vm/riscv64/asmhelpers.S index a708bf5db1186a..64b24957211ec8 100644 --- a/src/coreclr/vm/riscv64/asmhelpers.S +++ b/src/coreclr/vm/riscv64/asmhelpers.S @@ -1066,21 +1066,31 @@ Load_Ref A7, a7 // a1 - interpreter stack args location // a2 - interpreter stack return value location // a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetVoid, _TEXT, NoHandler - PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -16 + PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 + sd a4, 16(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 16(fp) + sd a2, 0(a4) EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR_INDEXED fp, ra, 16 + EPILOG_RESTORE_REG_PAIR_INDEXED fp, ra, 32 EPILOG_RETURN NESTED_END CallJittedMethodRetVoid, _TEXT +// a0 - routines array +// a1 - interpreter stack args location +// a2 - interpreter stack return value location +// a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetBuff, _TEXT, NoHandler - PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -16 + PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 + sd a4, 16(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 @@ -1088,20 +1098,30 @@ NESTED_ENTRY CallJittedMethodRetBuff, _TEXT, NoHandler ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 16(fp) + sd a2, 0(a4) EPILOG_STACK_RESTORE - EPILOG_RESTORE_REG_PAIR_INDEXED fp, ra, 16 + EPILOG_RESTORE_REG_PAIR_INDEXED fp, ra, 32 EPILOG_RETURN NESTED_END CallJittedMethodRetBuff, _TEXT +// a0 - routines array +// a1 - interpreter stack args location +// a2 - interpreter stack return value location +// a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetI8, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) sd a0, 0(a2) EPILOG_STACK_RESTORE @@ -1109,15 +1129,23 @@ NESTED_ENTRY CallJittedMethodRetI8, _TEXT, NoHandler EPILOG_RETURN NESTED_END CallJittedMethodRetI8, _TEXT +// a0 - routines array +// a1 - interpreter stack args location +// a2 - interpreter stack return value location +// a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetDouble, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) fsd fa0, 0(a2) EPILOG_STACK_RESTORE @@ -1129,15 +1157,19 @@ NESTED_END CallJittedMethodRetDouble, _TEXT // a1 - interpreter stack args location // a2 - interpreter stack return value location // a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2I8, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) sd a0, 0(a2) sd a1, 8(a2) @@ -1150,15 +1182,19 @@ NESTED_END CallJittedMethodRet2I8, _TEXT // a1 - interpreter stack args location // a2 - interpreter stack return value location // a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRet2Double, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) fsd fa0, 0(a2) fsd fa1, 8(a2) @@ -1171,15 +1207,19 @@ NESTED_END CallJittedMethodRet2Double, _TEXT // a1 - interpreter stack args location // a2 - interpreter stack return value location // a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetFloatInt, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) fsd fa0, 0(a2) sd a0, 8(a2) @@ -1192,15 +1232,19 @@ NESTED_END CallJittedMethodRetFloatInt, _TEXT // a1 - interpreter stack args location // a2 - interpreter stack return value location // a3 - stack arguments size (properly aligned) +// a4 - address of continuation return value NESTED_ENTRY CallJittedMethodRetIntFloat, _TEXT, NoHandler PROLOG_SAVE_REG_PAIR_INDEXED fp, ra, -32 sd a2, 16(fp) + sd a4, 24(fp) sub sp, sp, a3 mv t2, a0 mv t3, a1 ld t4, 0(t2) addi t2, t2, 8 jalr t4 + ld a4, 24(fp) + sd a2, 0(a4) ld a2, 16(fp) sd a0, 0(a2) fsd fa0, 8(a2) @@ -1225,7 +1269,7 @@ NESTED_ENTRY InterpreterStub, _TEXT, NoHandler bnez t4, LOCAL_LABEL(HaveInterpThreadContext) LOCAL_LABEL(NoManagedThreadOrCallStub): - addi a0, sp, __PWTB_TransitionBlock + 16 + addi a0, sp, __PWTB_TransitionBlock mv a1, t6 call C_FUNC(GetInterpThreadContextWithPossiblyMissingThreadOrCallStub) mv t4, a0 @@ -1245,6 +1289,8 @@ LOCAL_LABEL(HaveInterpThreadContext): ld t4, 0(t2) addi t2, t2, 8 jalr t4 + // Fill in the ContinuationContext register + ld a2, (__PWTB_ArgumentRegisters+16)(sp) EPILOG_WITH_TRANSITION_BLOCK_RETURN diff --git a/src/coreclr/vm/siginfo.hpp b/src/coreclr/vm/siginfo.hpp index fa063529d4be10..37d4be476d8f9e 100644 --- a/src/coreclr/vm/siginfo.hpp +++ b/src/coreclr/vm/siginfo.hpp @@ -746,7 +746,7 @@ class MetaSig //---------------------------------------------------------- // Is it an async call? //---------------------------------------------------------- - BOOL IsAsyncCall() + bool IsAsyncCall() { LIMITED_METHOD_CONTRACT; return m_CallConv & CORINFO_CALLCONV_ASYNCCALL; diff --git a/src/coreclr/vm/threads.cpp b/src/coreclr/vm/threads.cpp index fbec3458888816..748486d274268d 100644 --- a/src/coreclr/vm/threads.cpp +++ b/src/coreclr/vm/threads.cpp @@ -7318,7 +7318,7 @@ static InterpThreadContext* GetInterpThreadContextWithPossiblyMissingThreadOrCal Thread* currentThread, InterpByteCodeStart* pByteCodeStart) { - _ASSERTE(pByteCodeStart->Method->methodDesc->HasUnmanagedCallersOnlyAttribute()); + _ASSERTE(pByteCodeStart->Method->methodHnd->HasUnmanagedCallersOnlyAttribute()); InterpThreadContext *pThreadContext = NULL; @@ -7387,7 +7387,7 @@ extern "C" InterpThreadContext* STDCALL GetInterpThreadContextWithPossiblyMissin Thread::ObjectRefFlush(CURRENT_THREAD); #endif - PrestubMethodFrame frame(pTransitionBlock, pByteCodeStart->Method->methodDesc); + PrestubMethodFrame frame(pTransitionBlock, pByteCodeStart->Method->methodHnd); PrestubMethodFrame* pPFrame = &frame; pPFrame->Push(CURRENT_THREAD); @@ -7408,7 +7408,7 @@ extern "C" InterpThreadContext* STDCALL GetInterpThreadContextWithPossiblyMissin { OBJECTHANDLE ohThrowable = CURRENT_THREAD->LastThrownObjectHandle(); _ASSERTE(ohThrowable); - StackTraceInfo::AppendElement(ohThrowable, 0, (UINT_PTR)pTransitionBlock, pByteCodeStart->Method->methodDesc, NULL); + StackTraceInfo::AppendElement(ohThrowable, 0, (UINT_PTR)pTransitionBlock, pByteCodeStart->Method->methodHnd, NULL); EX_RETHROW; } EX_END_CATCH diff --git a/src/coreclr/vm/wasm/helpers.cpp b/src/coreclr/vm/wasm/helpers.cpp index 15c6c39d658c59..fc750c90eb0713 100644 --- a/src/coreclr/vm/wasm/helpers.cpp +++ b/src/coreclr/vm/wasm/helpers.cpp @@ -417,7 +417,7 @@ void _DacGlobals::Initialize() // Incorrectly typed temporary symbol to satisfy the linker. int g_pDebugger; -void InvokeCalliStub(PCODE ftn, void* cookie, int8_t *pArgs, int8_t *pRet) +void InvokeCalliStub(PCODE ftn, void* cookie, int8_t *pArgs, int8_t *pRet, Object** pContinuationRet) { _ASSERTE(ftn != (PCODE)NULL); _ASSERTE(cookie != NULL); @@ -434,7 +434,7 @@ void InvokeUnmanagedCalli(PCODE ftn, void *cookie, int8_t *pArgs, int8_t *pRet) ((void(*)(PCODE, int8_t*, int8_t*))cookie)(ftn, pArgs, pRet); } -void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target) +void InvokeDelegateInvokeMethod(MethodDesc *pMDDelegateInvoke, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet) { PORTABILITY_ASSERT("Attempted to execute non-interpreter code from interpreter on wasm, this is not yet implemented"); } @@ -710,14 +710,14 @@ void* GetUnmanagedCallersOnlyThunk(MethodDesc* pMD) return value->EntryPoint; } -void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target) +void InvokeManagedMethod(MethodDesc *pMD, int8_t *pArgs, int8_t *pRet, PCODE target, Object** pContinuationRet) { MetaSig sig(pMD); void* cookie = GetCookieForCalliSig(sig); _ASSERTE(cookie != NULL); - InvokeCalliStub(target == NULL ? pMD->GetMultiCallableAddrOfCode(CORINFO_ACCESS_ANY) : target, cookie, pArgs, pRet); + InvokeCalliStub(target == NULL ? pMD->GetMultiCallableAddrOfCode(CORINFO_ACCESS_ANY) : target, cookie, pArgs, pRet, pContinuationRet); } void InvokeUnmanagedMethod(MethodDesc *targetMethod, int8_t *pArgs, int8_t *pRet, PCODE callTarget) diff --git a/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.cs b/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.cs index c77caf9d912717..a4e7f4a0c072a9 100644 --- a/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.cs +++ b/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.cs @@ -40,7 +40,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - MyInt result = await Fib(new MyInt(25)); + MyInt result = await Fib(new MyInt(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25)); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result.i}"); diff --git a/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.csproj b/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.csproj index 197767e2c4e249..7548a677889c52 100644 --- a/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.csproj +++ b/src/tests/async/fibonacci-with-yields-struct-return/fibonacci-with-yields-struct-return.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.cs b/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.cs index 814b96eda0b974..34375bcbbf4fab 100644 --- a/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.cs +++ b/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.cs @@ -28,7 +28,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.csproj b/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.csproj index 197767e2c4e249..7548a677889c52 100644 --- a/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.csproj +++ b/src/tests/async/fibonacci-with-yields/fibonacci-with-yields.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.cs b/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.cs index 6b2a231db2326f..512473c2b2ed67 100644 --- a/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.cs +++ b/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.cs @@ -28,7 +28,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = AsyncHelpers.Await(Fib(30).ConfigureAwait(false)); + int result = AsyncHelpers.Await(Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 30).ConfigureAwait(false)); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.csproj b/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.csproj index 197767e2c4e249..7548a677889c52 100644 --- a/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.csproj +++ b/src/tests/async/fibonacci-without-yields-config-await/fibonacci-without-yields-config-await.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.cs b/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.cs index 11dd3db5168e47..2cfc3b332145a7 100644 --- a/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.cs +++ b/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.cs @@ -28,7 +28,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.csproj b/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.csproj index 197767e2c4e249..7548a677889c52 100644 --- a/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.csproj +++ b/src/tests/async/fibonacci-without-yields/fibonacci-without-yields.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.cs b/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.cs index 3af178b89a15d1..54f0e23ebf6135 100644 --- a/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.cs +++ b/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.cs @@ -8,6 +8,30 @@ public class InstUnBoxThunks { + class Utility + { + public static void ValidateArgs(object a0, object a1, object a2, object a3, object a4, object a5, object a6, object a7, object a8) + { + if ((string)a0 != "a0") + throw new Exception("a0 not 'a0'"); + if ((string)a1 != "a1") + throw new Exception("a1 not 'a1'"); + if ((string)a2 != "a2") + throw new Exception("a2 not 'a2'"); + if ((string)a3 != "a3") + throw new Exception("a3 not 'a3'"); + if ((string)a4 != "a4") + throw new Exception("a4 not 'a4'"); + if ((string)a5 != "a5") + throw new Exception("a5 not 'a5'"); + if ((string)a6 != "a6") + throw new Exception("a6 not 'a6'"); + if ((string)a7 != "a7") + throw new Exception("a7 not 'a7'"); + if ((string)a8 != "a8") + throw new Exception("a8 not 'a8'"); + } + } interface I0 { Task M0(); @@ -24,6 +48,7 @@ public async Task M0() public async Task M1(object a0, object a1, object a2, object a3, object a4, object a5, object a6, object a7, object a8) { + Utility.ValidateArgs(a0, a1, a2, a3, a4, a5, a6, a7, a8); await Task.Yield(); return "hello"; } @@ -40,7 +65,7 @@ static async Task CallStruct0M0() static async Task CallStruct0M1() { o01 = new Struct0(); - return await o01.M1(default, default, default, default, default, default, default, default, default); + return await o01.M1("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8"); } struct Struct1 : I0 @@ -53,6 +78,7 @@ public async Task M0() public async Task M1(object a0, object a1, object a2, object a3, object a4, object a5, object a6, object a7, object a8) { + Utility.ValidateArgs(a0, a1, a2, a3, a4, a5, a6, a7, a8); await Task.Yield(); return typeof(T).ToString(); } @@ -69,7 +95,7 @@ static async Task CallStruct1M0() static async Task CallStruct1M1() { o11 = new Struct1(); - return await o11.M1(default, default, default, default, default, default, default, default, default); + return await o11.M1("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8"); } class Box where U : I0 @@ -84,7 +110,7 @@ static async Task CallStruct1M0Field(Box arg) where T : I0 static async Task CallStruct1M1Field(Box arg) where T : I0 { - return await arg.f.M1(default, default, default, default, default, default, default, default, default); + return await arg.f.M1("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8"); } static Box> b1 = new(); @@ -158,6 +184,7 @@ public async Task M0() public async Task M1(object a0, object a1, object a2, object a3, object a4, object a5, object a6, object a7, object a8) { + Utility.ValidateArgs(a0, a1, a2, a3, a4, a5, a6, a7, a8); await Task.Yield(); return typeof(T).ToString(); } @@ -173,7 +200,7 @@ static async Task CallClass2M0() static async Task CallClass2M1() { o2 = new Class2(); - return await o2.M1(default, default, default, default, default, default, default, default, default); + return await o2.M1("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8"); } [Fact] @@ -198,6 +225,7 @@ async Task M0() async Task M1(object a0, object a1, object a2, object a3, object a4, object a5, object a6, object a7, object a8) { + Utility.ValidateArgs(a0, a1, a2, a3, a4, a5, a6, a7, a8); await Task.Yield(); return typeof(T).ToString(); } @@ -215,7 +243,7 @@ static async Task CallClass3M0() static async Task CallClass3M1() { o3 = new Class3(); - return await o3.M1(default, default, default, default, default, default, default, default, default); + return await o3.M1("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8"); } [Fact] diff --git a/src/tests/async/pgo/pgo.cs b/src/tests/async/pgo/pgo.cs index 0290a4c67f8a58..78488ac054c68e 100644 --- a/src/tests/async/pgo/pgo.cs +++ b/src/tests/async/pgo/pgo.cs @@ -19,12 +19,13 @@ public static void EntryPoint() internal static async Task AsyncEntryPoint() { - int[] arr = Enumerable.Range(0, 100_000).ToArray(); + int[] arr = Enumerable.Range(0, TestLibrary.Utilities.IsCoreClrInterpreter ? 100 : 100_000).ToArray(); int sum = 0; + int jCount = TestLibrary.Utilities.IsCoreClrInterpreter ? 10 : 100; for (int i = 0; i < 4; i++) { - for (int j = 0; j < 100; j++) + for (int j = 0; j < jCount; j++) sum += await AggregateDelegateAsync(arr, new AggregateSum(), 0); await Task.Delay(100); diff --git a/src/tests/async/pgo/pgo.csproj b/src/tests/async/pgo/pgo.csproj index 3fc50cde4b3443..ff94a80928aef7 100644 --- a/src/tests/async/pgo/pgo.csproj +++ b/src/tests/async/pgo/pgo.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/returns/returns.cs b/src/tests/async/returns/returns.cs index 3df7a8a65899bd..3cc1262694b39b 100644 --- a/src/tests/async/returns/returns.cs +++ b/src/tests/async/returns/returns.cs @@ -17,7 +17,8 @@ public static void TestEntryPoint() [MethodImpl(MethodImplOptions.NoInlining)] private static async Task Returns(C c) { - for (int i = 0; i < 20000; i++) + int count = TestLibrary.Utilities.IsCoreClrInterpreter ? 200 : 20000; + for (int i = 0; i < count; i++) { S val = await ReturnsStruct(); diff --git a/src/tests/async/returns/returns.csproj b/src/tests/async/returns/returns.csproj index 3fc50cde4b3443..ff94a80928aef7 100644 --- a/src/tests/async/returns/returns.csproj +++ b/src/tests/async/returns/returns.csproj @@ -1,5 +1,6 @@ + diff --git a/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.cs b/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.cs index c6eeb9725d70f7..d2e9c1d4954a4c 100644 --- a/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.cs +++ b/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.cs @@ -28,7 +28,7 @@ public static void Entry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = Fib(25); + int result = Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.csproj b/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.csproj index 6650af6b67607c..a2b68cb4f03c2f 100644 --- a/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.csproj +++ b/src/tests/async/syncfibonacci-without-yields/syncfibonacci-without-yields.csproj @@ -5,5 +5,6 @@ + diff --git a/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.cs b/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.cs index 2465903d710a81..fb19bd5c54e1b7 100644 --- a/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.cs +++ b/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.cs @@ -29,7 +29,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.csproj b/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.csproj index 6650af6b67607c..a2b68cb4f03c2f 100644 --- a/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.csproj +++ b/src/tests/async/taskbased-asyncfibonacci-with-yields/taskbased-asyncfibonacci-with-yields.csproj @@ -5,5 +5,6 @@ + diff --git a/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.cs b/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.cs index 2f8ddb5f06658e..f0b586ba0c784b 100644 --- a/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.cs +++ b/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.cs @@ -29,7 +29,7 @@ public static async Task AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.csproj b/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.csproj index 6650af6b67607c..a2b68cb4f03c2f 100644 --- a/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.csproj +++ b/src/tests/async/taskbased-asyncfibonacci-without-yields/taskbased-asyncfibonacci-without-yields.csproj @@ -5,5 +5,6 @@ + diff --git a/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.cs b/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.cs index f34cc93b1d6bb2..339ae88f25fd14 100644 --- a/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.cs +++ b/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.cs @@ -29,7 +29,7 @@ public static async ValueTask AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.csproj b/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.csproj index 6650af6b67607c..a2b68cb4f03c2f 100644 --- a/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.csproj +++ b/src/tests/async/valuetaskbased-asyncfibonacci-with-yields/valuetaskbased-asyncfibonacci-with-yields.csproj @@ -5,5 +5,6 @@ + diff --git a/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.cs b/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.cs index 64d57c46f8ee56..519a09cfda6c33 100644 --- a/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.cs +++ b/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.cs @@ -29,7 +29,7 @@ public static async ValueTask AsyncEntry() for (int i = 0; i < iterations; i++) { var sw = Stopwatch.StartNew(); - int result = await Fib(25); + int result = await Fib(TestLibrary.Utilities.IsCoreClrInterpreter ? 5 : 25); sw.Stop(); Console.WriteLine($"{sw.ElapsedMilliseconds} ms result={result}"); diff --git a/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.csproj b/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.csproj index 6650af6b67607c..a2b68cb4f03c2f 100644 --- a/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.csproj +++ b/src/tests/async/valuetaskbased-asyncfibonacci-without-yields/valuetaskbased-asyncfibonacci-without-yields.csproj @@ -5,5 +5,6 @@ +