Initial thoughts on Native Vararg support on Unix platforms for RyuJit

[jashook]

PR dotnet/coreclr#18373 will re-enable __arglist (native vararg support) for coreclr; however, currently only Windows platforms support the ABI. This "issue" is more of a brain dump to help any future work to allow native varargs on other currently unsupported/untested platforms. It is platform and arch specific, if there are gaps in my familiarity with other platform's/arch's ABIs please let me know I will update the issue.

Unix Amd64

x64 Unix should only require using RAX to pass the number of floating point arguments passed in registers for setting up the caller function. Change the prolog to spill all the General purpose registers and FP registers. ArgIterator would need to get the next arg based on type from the stack offset.

Unix Arm64

Currently there are a fair amount of example of where we mangle the vararg type for all arm arches. See https://github.com/dotnet/coreclr/blob/master/src/jit/compiler.hpp#L2607. This and other instances of forcing the argument types away from floating point types such as https://github.com/dotnet/coreclr/blob/master/src/jit/morph.cpp#L3322, need to also be restricted to windows. In addition, the prolog (here also assumes all arguments are passing in the general purpose registers.

Unix arm32

Everything for Unix arm64 applies. The only difference I can see is that non-fixed args (args in the ..., varargs) that are TYP_FLOAT will need to be retyped and used as if they were TYP_DOUBLE. This will require annotating the arguments with new additional information isFixedArg.

Windows arm32

This is the same ABI as arm64 varargs. Investigation needs to be done to determine if back-filling is supported.

category:reach
theme:varargs
skill-level:expert
cost:medium
impact:small

[jashook]

/cc @dotnet/jit-contrib

[BruceForstall]

I'm worried that we've created a situation where all targets/platforms are not at functional parity: they don't all support this the same. And, it's not clear we have commitment right now to add such support across all platforms.

[kekyo]

Is there any progress on this story? I'm interested in whether it is possible to achieve a multi-platform implementation of calling an entry point with vararg using PInvoke, and having the callback also accept vararg.