Add new SafeHandleMarshaller type to provide out-of-generator marshalling support.

[jkoritzinsky]

Fixes #74035

We can't remove the built-in marshalling support from the generator yet, but once the out-of-band packages we ship don't support .NET 6. we can remove the built-in support that emits the marshalling code in the stub. I believe the .NET 9 packages won't support .NET 6, so once we snap for .NET 9 and update how we ship the packages, we can clean this up.

This PR also adds a requested feature to the SafeHandle marshaller: If the call to native code fails, we'll call Dispose() on the pre-allocated handle to avoid leaking it to the finalizer queue.

[dotnet-issue-labeler]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[ghost]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #74035

We can't remove the built-in marshalling support from the generator yet, but once the out-of-band packages we ship don't support .NET 6. we can remove the built-in support that emits the marshalling code in the stub. I believe the .NET 9 packages won't support .NET 6, so once we snap for .NET 9 and update how we ship the packages, we can clean this up.

This PR also adds a requested feature to the SafeHandle marshaller: If the call to native code fails, we'll call Dispose() on the pre-allocated handle to avoid leaking it to the finalizer queue.

Author:	jkoritzinsky
Assignees:	-
Labels:	area-System.Runtime.InteropServices, source-generator
Milestone:	8.0.0

[AaronRobinsonMSFT]

Do we need any additional testing here or is this simply making our existing semantics public? Please do a quick audit just to make sure we have good coverage here.

[AaronRobinsonMSFT]

It would be good to see if we have a test for this marshaller and arrays. Hopefully we are close to it just being apart of the system and that should be naturally covered, but that is a scenario that is troublesome and very bad when we get it wrong.

[jkoritzinsky]

Explicitly did not support marshalling arrays of SafeHandles beforehand and now we won't by construction of our custom type marshalling system. I'll add some tests in CompileFails to validate.

[jkoritzinsky]

Looks like adding this marshaller might have caused a significant size regression in NativeAOT, though I'm not sure why the regression would be as large as it is. I'll try to investigate that today.

[jkoritzinsky]

Found the problem, and it's not good. NativeAOT doesn't accelerate the Activator.CreateInstance(Type, bool) method, so having the marshaller use that ends up being extremely expensive as it pulls in the whole reflection stack. I'll try to figure out something here.

[jkoritzinsky]

Marking as blocked as we need to resolve how to avoid rooting the reflection stack before we merge this in.

I'm proposing a breaking change to avoid the NativeAOT size regression and that should be reviewed.

[ghost]

Added needs-breaking-change-doc-created label because this PR has the breaking-change label.

When you commit this breaking change:

1. Create and link to this PR and the issue a matching issue in the dotnet/docs repo using the breaking change documentation template, then remove this needs-breaking-change-doc-created label.
2. Ask a committer to mail the .NET Breaking Change Notification DL.

Tagging @dotnet/compat for awareness of the breaking change.

[jkoritzinsky]

Based on a conversation with @jkotas in #85541, we have decided to take a breaking change to only allow SafeHandle-derived types with public constructors in ref, out, and return scenarios. That way our source-generated marshalling rules are following our guidance and the SafeHandleMarshaller type does not pull in the reflection stack.

[f2bo]

How would an array be handled using this marshaller? For example,

[LibraryImport("some_lib")]
public static partial void WaitForMultipleSignals([MarshalUsing(typeof(ArrayMarshaller<,>), CountElementName = nameof(numSignals))] SafeWaitHandle[] signals, int numSignals);

Wouldn't it also require support for ElementIn and ElementOut marshal modes, or is there some other mechanism to handle this scenario?

[AaronRobinsonMSFT]

How would an array be handled using this marshaller?

We don't support arrays of SafeHandles.

Wouldn't it also require support for ElementIn and ElementOut marshal modes

Yes. We are not supporting this at present.

[AaronRobinsonMSFT]

I think we should also update our compat doc about SafeHandle scenarios. This is breaking change from DllImport, which we've typically documented in the design doc.