Proposal: Address gap in ComImport support of WinRT interop

[Scottj1s]

Summary
There are a number of COM interfaces that are not modeled in Windows metadata (the format defined in ECMA-335), and yet are required to make full use of many WinRT APIs. Historically, these interfaces have been supported by the ComImport attribute, which the C# compiler and runtime use to generate projected interfaces on demand. With .Net5.0, while ComImport support remains, it can no longer be used to interop with WinRT interfaces, and thus needs a replacement.

Background
Most non-metadata interfaces exist to provide interop functionality. Currently, the SDK “um” folder has 28 interop style headers, the majority for explicit HWND hosting, to support Desktop and multi-window UWP apps - the so-called interop pattern. There are also interop interfaces that don't follow this pattern, and of course there are many user-defined interop interfaces. So while it's possible to hand roll each interop interface definition, the work is non-trivial and doesn't scale to the need.

With C++/WinRT, the common practice is to excerpt an interop interface definition from its header and include it near the call site. This ignores any type mappings that would otherwise be provided by the projection (e.g., HSTRING <--> std::wstring). The same could be done with C#/WinRT, only mapping return HRESULT to exceptions. This would be much simpler than providing a complete projection of an interface, and could be implemented with a source generator. But many ComImport definitions also include MarshalAs attributes to provide natural type mappings on params and returns. Here, the caller would be programming to the ABI, making it much less useable.

Alternately, a source generator could duplicate much of the type-mapping (projection) logic of cswinrt.exe, but that's a poor design decision. It's not clear whether a source generator could exec cswinrt.exe to provide the projection, given that a fully built assembly would be required to provide interface definitions and ignore everything else.

Proposal
Given the constraints, the simplest and best solution is to model the interop interface in WinRT IDL so that a winmd can be produced from it, and consumed by cswinrt.exe. In many cases, this would require modest edits to COM IDL to make it WinRT IDL compatible (adding namespaces, removing HRESULT returns, etc). To support IUnknown-based interfaces, which are prohibited by ECMA 335, an InterfaceIsIUnknown attribute (defined in any namespace) could be attached to the interface definition and used by cswinrt modified to provide the proper projection. The attribute name is borrowed from the ComInterfaceType enum used with ComImport.

This system is definitely constrained. It does not provide exhaustive coverage of all ComImport capabilities (marshaling, proxying, BSTRs, SafeArrays, etc). Instead, it provides the subset of functionality typically required of interop interfaces - access to HWNDs, buffers, etc.

Based on this scheme, these interop scenarios can be supported:

1. IInspectable-based interfaces defined in WinRT IDL: currently supported
2. IInspectable-based interfaces defined in COM IDL: migrate the definition to WinRT IDL
3. IUnknown-based interfaces defined in COM IDL: migrate the definition to WinRT and mark it with an InterfaceIsIUnknown attribute
4. Any interface not defined in IDL (e.g., hand-rolled): model the interface in WinRT IDL, marking it with an InterfaceIsIUnknown attribute as needed

Sample Interop IDL

/*  Interop.idl
 *
 *	Experiments in generating interop interface projections for C#/WinRT
 */

/* a namespace is required for all WinRT interfaces */
namespace Interop
{

/* For any IInspectable-based interop currently defined in  COM IDL, modify it to produce metadata */
[uuid(D3EE12AD-3865-4362-9746-B75A682DF0E6)]
interface IAccountsSettingsPaneInterop
{
    UInt64 GetForWindow(UInt64 appWindow, GUID riid);
    UInt64 ShowManageAccountsForWindowAsync(UInt64 appWindow, GUID riid);
    UInt64 ShowAddAccountForWindowAsync(UInt64 appWindow, GUID riid);
}

/* Define attribute that cswinrt can look for to provide IUnknown-based interface projection */
[attributeusage(target_interface)]
[attributename("InterfaceIsIUnknown")]
attribute InterfaceIsIUnknownAttribute
{
}

/* For IUnknown-based interops (including those cswinrt currently hand rolls), attribute accordingly */ 
[uuid(905a0fef-bc53-11df-8c49-001e4fc686da)]
[InterfaceIsIUnknown]
interface IBufferByteAccess
{
    UInt64 Buffer();    
}

}

[stevenbrix]

This system is definitely constrained. It does not provide exhaustive coverage of all ComImport capabilities (marshaling, proxying, BSTRs, SafeArrays, etc).

I'm still a bit confused on how this works in practice for existing code that is 100% COM (scenario 3 that you described). Will I be able to use BSTR and SafeArray in WinRT IDL if I mark the type with the InterfaceIsIUnknown attribute, or do I have to change those to the WinRT equivalent?

[Scottj1s]

This system is definitely constrained. It does not provide exhaustive coverage of all ComImport capabilities (marshaling, proxying, BSTRs, SafeArrays, etc).

I'm still a bit confused on how this works in practice for existing code that is 100% COM (scenario 3 that you described). Will I be able to use BSTR and SafeArray in WinRT IDL if I mark the type with the InterfaceIsIUnknown attribute, or do I have to change those to the WinRT equivalent?

No, I mentioned above that BSTRs, SafeArrays, and so on wouldn't be supported (no marshaling support). This solution is scoped to providing interop support, which is typically just ferrying IntPtrs around. Anything beyond supporting WinRT ventures into general ComImport support which is out of scope.

To get an idea of target scope, take a look at C:\Program Files (x86)\Windows Kits\10\Include*\um*interop.idl. These define many of the interfaces we're talking about, and a lot of these already derive from IInspectable. They don't use any COM only types.

[Scottj1s]

see also:
dotnet/runtime#35318

/cc: @AaronRobinsonMSFT

[AaronRobinsonMSFT]

There are a number of COM interfaces that are not modeled in ECMA 335 (Windows metadata),

I realize this is the format being used for Windows metadata, but I am unclear if makes sense to phrase it this way. ECMA 335 formalizes a compliant CLR runtime system complete with metadata, verification, instruction set, and semantics. The metadata is a subset of that, I might invert the statement "modeled in Windows metadata (format defined in ECMA-335)".

Historically, these interfaces have been supported by the ComImport attribute, which the C# compiler and runtime use to generate projected interfaces on demand. With .Net5.0, this support has been removed and needs a replacement.

This support still exists. The issue here is "crossing the streams". The built-in system works within itself, but can't work with other interop system built with fancy new APIs like ComWrappers.

[AaronRobinsonMSFT]

This is a great idea. Can't wait to see it.

[stevenbrix]

To get an idea of target scope, take a look at C:\Program Files (x86)\Windows Kits\10\Include*\um*interop.idl. These define many of the interfaces we're talking about, and a lot of these already derive from IInspectable. They don't use any COM only types.

Ok, so let me repeat what I think I'm hearing and you can let me know if I'm correct. What you are proposing is the following:

1. For interfaces/types that Microsoft ships that interop with WinRT but aren't exposed in WinRT, provide a mechanism for them to be expressed in metadata
2. For 3rd party COM code that has nothing to do with WinRT, those can still use the COMImport attribute and whatever other COM facilities that currently exist in the .NET runtime.

Is that correct?

[weltkante]

How far will this go? WinRT/WinUI has SwapChain interop, I assume you don't want to pull in DXGI/D3D and leave this for a 3rd party projection? Just bringing this up so you can keep the interop transition in mind and make sure it works in the initial release, its a primary usecase (game-) devs will want to use WinUI for, SwapChain integration right into UI is something lacking in other UI frameworks.

[jtbrower]

Forgive any Misunderstanding on my Part

Before I share details below, I am adding this paragraph in retrospect. My background in C# began in 2007 when I was tossed into it coming from operating systems development in Unix'like OSes. Please forgive any misunderstandings that I may have with respect to COM. I know enough to make it work, but cannot fully appreciate the amazing details that @Scottj1s listed in the OP without writing a component myself. Since I avoid legacy as much as possible I have avoided a deep dive into COM.

The COM component that I speak of below is ActiveX based and it is used in the Credit Card Processing industry.

What my Company Does

We/I design and develop Point-of-Sale applications that are licensed to third party companies who use these applications to create their own vision of what a Point-of-Sale application should be. In many ways, our business model is similar to Telerik's where the source code comes with the product.

I have been actively re-writing the entire application from the ground up since December of 2016. Since I want my own customers to be able to take advantage of the best that .Net has to offer I have been keeping steady pace with all pre-release versions of .Net Core, now .Net 5 pre-release 5. This application is sitting at 149 libraries as of today and I am a one-person company.

Credit Card Processing Industry and Exposure to COM

Although I can easily do away with the direct ComImport use have in our enterprise application, I am more concerned about a third party COM component that I have used for a decade. This component seems to have no hope of being migrated away from COM (I have spoken to them about this in the past) but it is still an extremely important component that credit card processing applications use today. From my perspective, access to this component involves two simple ComReference nodes in my csproj. I have done everything in my power to completely isolate my application from the use of this assembly, but I still have an active positive relationship with the company that produced it as well as my own customers that use the source code I license to them. This component is an intrusive little booger because it prefers to be used in-process and mucks with the WPF dispatcher loop (there are workarounds, but that's outside the scope of this thread). I would love to move away from using that library, but to date the only alternative to COM that this company offers involves additional hardware cost that would be imposed upon each and every merchant that my own customers sell POS applications to. The point-of-sale ecosystem is extremely competitive and that competitiveness leads to thin profit margins. Any additional hardware requirements that lead to increased costs are quickly frowned upon.

The industry has plenty of alternatives to this company, but this company happens to be one of the top providers of a component that that has features that most other companies do not. This feature then reduces that list down to a select few options and being able to support those select few is a top requirement of mine as it is other companies. The main feature is being able to run credit card (and gift card) transactions with the majority of all north American credit card processors.

Although I will not allow that component to prevent me from moving my entire application into the future of .Net, its worth me mentioning how this could affect each and every company out there that uses this third parties processing components. They are an important provider in the Credit Card Processing ecosystem with hundreds of reputable POS Software Applications using their middleware to process a substantial amount of credit card transactions in north America. The importance of this corporation's contribution in the world of point-of-sale cannot be overstated.

What their COM Component Provides

They provide the glue necessary to connect hundreds of types of credit card processing hardware devices to all of the important credit card processors in North America and they do this in a manner that helps assure that most POS applications remain "out-of-scope" regarding PCI-DSS's PA-DSS requirements. The alternative to using companies like this are to integrate directly with each and every credit card processor. Although there are some advantages to this, it is a costly undertaking to support all of those direct integrations (and certifications) and IMO most Software Engineering teams in this industry are already understaffed. Direct integrations are also less appealing because most credit card processors want you to sign exclusivity agreements that you will not process with any other processor. (They may verbally tell you that you can process with others, but the contracts state otherwise.)

Credit Card transaction profits are arguably the primary source of income for just about everyone in this industry. For that reason, it wouldn't be unheard of for a software company to live in legacy code solely because something in this industry forced it to do so. Essentially it's not a situation where companies can just say "well, we just won't use them then".

Innovation in Credit Card Processing

Innovation in this industry moves slowly. It is based upon decades of legacy standards that make it difficult for innovation to take place. The thought of re-writing such a massive amount of certified code that these middleware companies support seems to be a difficult problem to solve and potentially part of the bottleneck that keeps their own technology stuck in the past.

COM Wrapper interface Definitions

After reading the proposal to remove ComImport support, I took a look at my companies use of that third party component by opening up the COM wrappers that were generated by adding two ComReference nodes in the csproj. At first glance, every function defined in those wrappers has a return type attributed with [return: MarshalAs(UnmanagedType.BStr)] and any of those functions that accept a string as a parameter marks the parameter with the [MarshalAs(UnmanagedType.BStr)] attribute. As I understand it, and as was stated in the proposal of the OP, BSTR will not be supported.

Summary and Disclaimer

For anyone inside of the credit card processing industry that reads this, I feel the need to state the following.

1. I am familiar with alternatives offered by companies that compete with the anonymous company I am talking about. I have heard many sales pitches, but this company is still a dominant force that cannot be overlooked.
2. I am familiar with the alternatives that this anonymous company offers, but again they involve additional hardware costs that trickle down to a merchant who has such a huge amount of alternatives.

As a final note, there may already be other reasons why this component won't work with .Net 5.0's final release. I am a one-person company developing the 5th version of this product that I offer. I am not actually focused on the Credit Card processing layer right now but try to stay up-to-date with how Microsoft's proposals may affect people in the niche world of Point-of-Sale. I certainly do not want to hinder innovation and as I mentioned I avoid COM whenever possible. I planned to go back and re-address this issue once I returned to those modules. I was just happy that I had the component up and running back around NetCore 2 or so.

I love the fast pace that Microsoft is moving at, keep up the good work!

[AaronRobinsonMSFT]

@jtbrower Please contact me offline via email - you can find it in my github profile. I want to chat with you about your scenario. Supporting companies like yours is a priority for .NET and we want you to be successful.

The discussion in this issue is for a very narrow slice of COM. Support for the COM scenarios described in your issue should not be impacted. COM in .NET is going to stay for a very long time (read forever).

[Scottj1s]

@stevenbrix - as @AaronRobinsonMSFT has pointed out, ComImport use independent of WinRT interop remains supported. I've clarified the title and opening paragraph.

@weltkante - dxinterop is a good question. I'd assume most developers would use SharpDx for that, but your general point is taken. Ideally, definitions like IDXGISurface, IDXSISwapChain would not need to be replicated in WinRT IDL. That would leave the challenge of wiring up a raw COM interface pointer, however obtained, and using the C#/WinRT marshaling wrappers to pass it to the interop interfaces like ISwapChainPanelNative. Some research needed there.

@jtbrower - thanks for your thorough writeup and defense of ComImport. I hope it amounts to clumsy wording on my part, which I've tried to clarify. Any use of ComImport independent of WinRT interop would still be supported by the runtime. The interaction of ComImport and C#/WinRT projected interfaces would amount to "crossing the streams" of projection techniques, and would no longer be supported.

[jtbrower]

@Scottj1s thank you for the clarification.

[pjmlp]

@Scottj1s SharpDX project is dead, and the same is bound to happen to every DX interop project with a bus factor of 1, including the current ones that are trying to fit into SharpDX shoes, so it would be nice if C#/WinRT would reduce the use cases where .NET developers are forced to use C++, or manually write P/Invoke marshaling code for COM like libraries not exposed to .NET by the Windows team.

[jkoritzinsky]

As part of this, we'll need to address the fact that some of the native interfaces, such as IGraphicsEffectD2D1Interop::GetProperty use illegal types like IPropertyValue in their signature.

[stevenbrix]

@jkoritzinsky can you elaborate more on why we need to do that?

[jkoritzinsky]

The IPropertyValue type cannot be legally expressed in a WinRT signature according to the docs, and WinRT.Runtime currently doesn't expose IPropertyValue as a result. Additionally, IPropertyValue needs to be treated specially anyway because we should surface it as object since it represents a boxed value.

Types such as IGraphicsEffectD2D1Interop violate the documented rules of the WinRT type system by including signatures that contain IPropertyValue.

[Scottj1s]

@jkoritzinsky - should interfaces like that, which intentionally cross the COM/WinRT streams, just be special-cased (hand-rolled)?

[jkoritzinsky]

Yes, hand rolling the few interfaces that use WinRT types in their signatures would be reasonable.

[Scottj1s]

After a design discussion on this topic, the idea was floated to use the existing ComImport support, provided there was a way to marshal a raw interface pointer to a ComImport interface. The Marshal class provides that and it works very nicely. The associated PR, #333, is POC which would obsolete all the discussion here.