Check return types in isCompatibleMethodGDV

[EgorBo]

I've hacked a stress mode for GDV delegate inlining and found an issue that we don't validate return types.

Might fix #117867

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

Might fix #117867

That seems unlikely, this check should never fail with dynamic PGO, only with potentially outdated static PGO.

[EgorBo]

Might fix #117867

That seems unlikely, this check should never fail with dynamic PGO, only with potentially outdated static PGO.

Yeah, but I think even for Dynamic PGO we should be resilient there (I know there is a debug assert there, but who knows).

[EgorBo]

Still, the issue might be the Static PGO? The one that we ship BCL with,.

[jakobbotsch]

I've hacked a stress mode for GDV delegate inlining and found an issue that we don't validate return types.

How did this issue surface? The point of the check is just to ensure that the JIT does not blow up. If the signatures don't match then the guard will never pass at runtime, so I don't think it would ever result in bad codegen.

[EgorBo]

I've hacked a stress mode for GDV delegate inlining and found an issue that we don't validate return types.

How did this issue surface? The point of the check is just to ensure that the JIT does not blow up. If the signatures don't match then the guard will never pass at runtime, so I don't think it would ever result in bad codegen.

Makes sense, but in my case it just threw a bunch of JIT asserts here and there

[EgorBo]

Does it look good? @jakobbotsch cc @dotnet/jit-contrib

[Copilot]

Pull Request Overview

This PR adds return type validation to the isCompatibleMethodGDV function in the JIT compiler to prevent incorrect delegate inlining optimizations. The change addresses a bug discovered through stress testing where Guarded Devirtualization (GDV) for delegate calls was not properly validating that the candidate method's return type matches the call site's expected return type.

Key changes:

• Added return type compatibility checks using both implicit coercion validation and exact type matching
• Added specialized handling for struct return types with layout and passing convention validation
• Enhanced logging for debugging incompatible method scenarios

[EgorBo]

Reverted to getMethodSig because call->gtRetClsHnd is not set at that point (it's set in impFixupCallStructReturn which is called later). Since delegate devirtualization is a rare event, I guess it's not a big deal to call it here once again.