P/Invokes should use C types not C++.

[AaronRobinsonMSFT]

A recent C++ toolchain update uncovered an invalid P/Invoke signature using C++ types as opposed to C types. This caused the test to fail due to invalid assumptions. The test was updated to be valid.

Fixes #94931

[AaronRobinsonMSFT]

/cc @jkotas @elinor-fung

[jkotas]

We do not seem to have the difference between managed and unmanaged x86/x64 calling conventions for small return types noted in https://github.com/dotnet/runtime/blob/main/docs/design/coreclr/botr/clr-abi.md . It would be nice to fix that.

For unmanaged calling conventions, the state of unused bits in the value returned in RAX or XMM0 is undefined. (https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#return-values)

For managed calling conventions, the primitive values smaller than 32-bits must be zero-extended (or sign-extended?) to 32-bits. @dotnet/jit-contrib What is the exact rule that the JIT implements?

[AndyAyersMS]

For managed methods, the callee widens small return values. For native methods, the caller widens small return values.

I think we always zero extend. Looks like we sign extend in some cases.

runtime/src/coreclr/jit/importercalls.cpp

Lines 1428 to 1445 in 8a1b985

	if (!bIntrinsicImported)
	{
	//-------------------------------------------------------------------------
	//
	/* If the call is of a small type and the callee is managed, the callee will normalize the result
	before returning.
	However, we need to normalize small type values returned by unmanaged
	functions (pinvoke). The pinvoke stub does the normalization, but we need to do it here
	if we use the shorter inlined pinvoke stub. */
	if (checkForSmallType && varTypeIsIntegral(callRetTyp) && genTypeSize(callRetTyp) < genTypeSize(TYP_INT))
	{
	call = gtNewCastNode(genActualType(callRetTyp), call, false, callRetTyp);
	}
	}
	impPushOnStack(call, tiRetVal);
	}

[AaronRobinsonMSFT]

@AndyAyersMS Is there a follow up issue to file here?

[AndyAyersMS]

You mean for codegen? I don't see one, unless we're not actually doing what is outlined above.

[AaronRobinsonMSFT]

This is step one of the fix. The current update works and passes on macOS. Pushing to CI to make sure Windows and Linux remain in the same state.

[AaronRobinsonMSFT]

@jkotas Please take another look at this. I'd like to get the CI unblocked. Feel free to merge if you are okay with the changes.

[jkotas]

Thank you