Fix hanging in JIT for ARM64 release compilation

[AntonLapounov]

dotnet/runtime#50832 added support for Vector<nint> and Vector<nuint>, which caused release builds of JIT to hang while compiling Vector<nuint> methods. Debug builds of JIT would fail the following assertion in Compiler::raUpdateRegStateForArg:

runtimelab/src/coreclr/jit/regalloc.cpp

Line 180 in fda312f

assert(!regState->rsIsFloat);

The issue is caused by disagreement between JIT and ILCompiler on whether the aforementioned types are homogeneous vector aggregates (HVA). The fix is to extend the HVA check in VectorOfTFieldLayoutAlgorithm.ComputeValueTypeShapeCharacteristics to include IntPtr and UIntPtr element types.

The introduced IsSupportedElementType method may be temporary. When Runtime repo adds support for nint and nuint element types for Vector64<T>, Vector128<T>, and Vector256<T> types, they would have to extend similarly the TypeDesc.IsPrimitiveNumeric check here:

runtimelab/src/coreclr/tools/Common/Compiler/VectorFieldLayoutAlgorithm.cs

Lines 95 to 96 in fda312f

	if (type.Context.Target.Architecture == TargetArchitecture.ARM64 &&
	type.Instantiation[0].IsPrimitiveNumeric)

If they update/rename the TypeDesc.IsPrimitiveNumeric property, we will use the updated property in this class as well.

[jkotas]

IsPrimitiveNumeric returning false for nint/nuint looks like a oversight. @tannergooding Should we rather fix IsPrimitiveNumeric to return true for nint/nuint?

[tannergooding]

Yes, this is just an oversight and I missed that it needed to also be updated.

[tannergooding]

Noting that Vector64/128/256<T> don't currently support nint/nuint.

They were excluded since we have no APIs that support them today and adding that support requires API review.

[jkotas]

Yes, this is just an oversight and I missed that it needed to also be updated.

Would you mind sending PR to dotnet/runtime with the fix?

[AntonLapounov]

Note that IsPrimitiveNumeric is also used here:

runtimelab/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs

Lines 2272 to 2280 in fda312f

	private CorInfoType getTypeForPrimitiveNumericClass(CORINFO_CLASS_STRUCT_* cls)
	{
	var type = HandleToObject(cls);
	if (type.IsPrimitiveNumeric)
	return asCorInfoType(type);
	return CorInfoType.CORINFO_TYPE_UNDEF;
	}

I doubt we should extend IsPrimitiveNumeric, because it would create disagreement on HVA between JIT and Crossgen2 in VectorFieldLayoutAlgorithm.

[AntonLapounov]

Also note that the property in question is expected to mimic this one that does not accept IntPtr/UIntPtr types:

runtimelab/src/coreclr/vm/jitinterface.cpp

Line 4338 in fda312f

CorInfoType CEEInfo::getTypeForPrimitiveNumericClass(

[tannergooding]

I doubt we should extend IsPrimitiveNumeric, because it would create disagreement on HVA between JIT and Crossgen2 in VectorFieldLayoutAlgorithm.

This sounds like is its own problem. On x64, two structs that only differ based on their usage of nint vs long should be treated identically by HVA, as they are identical at the ABI level.

If the only limiter here is that Vector128<nint> is disallowed because we don't have any public APIs taking it, then we should probably just relax that restriction as well and have it work as expected.

We will eventually have to do the same for Half, which is likewise specially treated for HFA/HVA, when we add the respective intrinsic APIs.

[tannergooding]

The relevant checks in https://github.com/dotnet/runtime/blob/main/src/coreclr/jit/simd.cpp#L607 and in the importer itself should be correctly blocking nint/nuint from matching in signatures even if getTypeForPrimitiveNumericClass returns a type.

So this should really only come down to how Vector128<nint> is treated when passed to or returned from a function.

[AntonLapounov]

If the only limiter here is that Vector128<nint> is disallowed because we don't have any public APIs taking it, then we should probably just relax that restriction as well and have it work as expected.

That should be fine; I guess that would require adding code to recognize those types in Compiler::getBaseJitTypeAndSizeOfSIMDType.

The relevant checks in https://github.com/dotnet/runtime/blob/main/src/coreclr/jit/simd.cpp#L607 and in the importer itself should be correctly blocking nint/nuint from matching in signatures even if getTypeForPrimitiveNumericClass returns a type.

Right now they will not recognize Vector128<nint> the same way as Vector128<long> for ARM64. And the corresponding Crossgen2 check reflects that. Those two checks must be kept in sync.

[AntonLapounov]

Closing in favor of dotnet/runtime#52016.