Add proposed plan for the COM source generator

[jkoritzinsky]

The interop team has decided to invest some time into creating a COM source generator. This doc describes some of the goals of the project, various checkpoints where we can stop our work, and opens a discussion for how we want users to identify which types to generate source for.

[AaronRobinsonMSFT]

Migration might not be so bad depending on the users in question. I'd like to start with the premise the defaults, unadorned arguments, be the most common which results in COM interface definitions being void of MarshalAs.

[tannergooding]

Just noting that while COM supports IDL and TLB, it is not the case that all COM starts out from IDL or TLB. There are a few cases of COM even in the Windows SDK (D2D1, for example) which only exist as C++ header files.

There are also cases, like most of DirectX, where its not the type of IDL/TLB that you find in "traditional COM". Instead, it uses what is sometimes referred to as "nano COM" which is a limited subset and which doesn't use CoTaskMemAlloc, CLSID (only using IID), or other functionality. Additionally, the IDL files don't have all the required contrsucts to work with the existing tlbgen.exe or related tooling that exists on Full Framework.

I think we should ensure that any tooling here can properly account for and handle these types of scenarios.

• https://github.com/microsoft/clangsharp currently achieves this non-source generator based support by simply parsing and traversing the C/C++ AST, and while it can't handle everything that real C++ supports; it does support effectively all the COM constructs you'd encounter in nano-COM and several of the constructs from full COM.

[jkoritzinsky]

Due to the issues with making IDL/TLB the source of truth (as you mention), the proposed plan is to leave TLB/IDL consumption and production for later work and develop a solution that uses a C# source of truth first. That way, projects like ClangSharp could produce C# that the source generator can consume even when the source of truth isn't IDL or a TLB.

[AaronRobinsonMSFT]

I would prefer to avoid providing any additional tooling beyond what TlbImp or TlbExp could provide. Parsing C/C++ headers looks to be well covered by the clangsharp tool and I'd argue that tool should continue to do that. In this case the proposed source generator would be able to consume the output of clangsharp and treat that as the source of truth. What would be the benefit of supplanting this tool or am I missing some nuance here?

[tannergooding]

What would be the benefit of supplanting this tool or am I missing some nuance here?

That was somewhat my point. This section discusses IDL and TLB files and conversions from them to something that the source generator can handle.

However, they are not always available or usable and there exists other tooling that could already generate the required "stuff". ClangSharp for example already generates the full VTBLs and helper methods required to support COM today: https://source.terrafx.dev/#TerraFX.Interop.Windows/um/d3d12/ID3D12Device8.cs,56ac150c61296fc3

It just doesn't have the "stuff" to connect these raw COM views into the COM Wrappers support.

[jkoritzinsky]

@AaronRobinsonMSFT thats exactly what my proposal is.

We as the runtime team would provide the “C# source of truth” generator.

Either our team or someone else would provide the TlbImp/Exp replacement, which could generate C# that our generator reads in.

ClangSharp (or another community project) would provide the “C++ to C#” tool that would generate code that our generator reads in to do the work.

The only work we need to do is the “C# source of truth” generator.

[AaronRobinsonMSFT]

they are not always available or usable and there exists other tooling that could already generate the required "stuff"

I don't think I am getting the point here. The current TlbImp tools generate things that may not work with this if we use new attributes and to some degree we would like to have a single assembly that would contain both definition and goo - should be possible to not have that but that would ideally be P0 given the MCG replacement goal. Consuming a TLB as the source of truth is the canonical way for COM interop so that is the primary focus. DirectX scenarios are P2 at this point but once something has converted those definitions into C# we could consume that and clangsharp seems to be able to handle that so I am back to being confused I think. Is the point here that tools exist that already do part of the job?

The TlbExp tooling however does need a full replacement so that would be another reason TLB is called out.

[tannergooding]

Is there an example of what an a more complex inheritance hierarchy might look like?

Will users continue needing to "duplicate" base interface members with the new slot keyword like with the built-in COM support?

How do you indicate this is IUnknown vs no base vs IDispatch or something else?

[jkoritzinsky]

I will add some examples for these scenarios in my next iteration.

[AaronRobinsonMSFT]

I'm not sure what no base means in this case because without it what is being generated? These tooling is specific to IUnknown based interfaces so that is where it would stop. The IDispatch base is interesting and I'd imagine we can reuse some existing attributes – ComInterfaceType has baggage but could be reused in this case, ignoring InterfaceIsIInspectable of course.

[tannergooding]

Likewise, what's the expected model for common patterns where managed objects (e.g. interfaces) don't translate cleanly to the native signature?

For example, there are often APIs that take the form of HRESULT M(..., void** ppOutput);

If ppOutput is null, the method returns S_FALSE if the passed in parameters would have allowed the method to succeed (it acts effectively as a validation method without allocating or doing other costly internal operations).

This means the signature is effectively out T output where Unsafe.NullRef<T>() is valid; but that has various difficulties among other things and is "traditionally" handled by out IntPtr output or IntPtr* output where you lose all real safety and support, so you aren't significantly "better off" than if you were simply using the underlying vtbls in the first place (outside of basic lifetime tracking).

[tannergooding]

I'm not sure what no base means in this case because without it what is being generated? These tooling is specific to IUnknown based interfaces so that is where it would stop. The IDispatch base is interesting and I'd imagine we can reuse some existing attributes – ComInterfaceType has baggage but could be reused in this case, ignoring InterfaceIsIInspectable of course.

@AaronRobinsonMSFT, there exist COM interfaces in a few scenarios that don't inherit from IUnknown. One example is ID3DInclude, which is simply declared using DECLARE_INTERFACE(iface) and so it follows the COM ABI, but is not itself IUnknown. This interface is expected to be implemented by the consumer and passed into a consuming function provided by the library.

The tooling should likely not break down in the face of such an interface declaration; regardless of whether the user implements it alongside IUnknown on the derived type

[AaronRobinsonMSFT]

there exist COM interfaces in a few scenarios that don't inherit from IUnknown.

That isn't a COM interface then – by definition. The entire ComWrappers API is about providing IUnknown ABI support and removing that is, in my opinion, a non-starter as it breaks down lifetime and identity semantics fast. In fact exposing the ID3DInclude type seems like a "dead-end" because you can't go anywhere from it.

I guess one could attempt to use virtual inheritance to inherit from both IUnknown and ID3DInclude to "unify" them but I don't know if the order of that is actually defined so how would we enable this in a stable way? What am I missing?

[tannergooding]

That isn't a COM interface then – by definition.

Sorry, that was a misstatement by me. I meant "there exist interfaces in a few COM related scenarios".

In particular I'm just trying to call out some scenarios that I think are important to consider, at least from one aspect. I don't think they have to be handled the same way or even at all, as long as they are thought about and a decision is made about them.

I just struggle with finding the right words sometimes 😄

[AaronRobinsonMSFT]

@tannergooding I think this conversation revolves around two principles – (1) let's make sure we support as many scenarios as possible compared to the build-in and (2) how can we ensure our source gen provides maximum benefit for the .NET community. I agree with these principles and I think the tenor of the initial questions demonstrates a similar thinking. Instead of focusing on concrete examples like ID3DInclude, I'll try to adhere to the above principles.

The scenario above, I am sure there are others, demonstrates that we going to fail to please everyone. Therefore in an effort to bring (1) closer to reality we will need to be permissive in some manner. Principle (2) dictates saying "You can build your own source generator from these N assemblies" or "your special cases will require you to perform unnatural build system hacks" aren't acceptable. So, I propose the following:

• We provide a way for users to define their own manually written/defined wrappers (RCW/CCW) for an interface. These would then be discoverable through some attribute or other gesture by the source generator. This would allow the majority of people to benefit from the simple well-defined COM that is likely the common support scenario (for example, WinForms) and at the same time permit special cases to be addressed while not degrading the overall experience if a small minority of types are an issue.

@jkoritzinsky I'd imagine we could have users attribute types for an RCW (a la, IDIC) and/or add attributes to static methods marked with UnmanagedCallersOnly as to where in a CCW generated vtable they wanted to place the method. There are lots of options here but in the end we can focus on typical COM support circa 2002 but permit anyone to say "I want the tool to generate 99 % of my interfaces, but these 2 I will define myself using all the fancy Marshal helpers written in .NET 7 and the tool will fold them into the process naturally."

[wjk]

I fully support Option 3 in the markdown document. As I see it, the developer should be able to simply replace ComImportAttribute with GeneratedComImportAttribute (or whatever) in their code and recompile. Same goes for ComVisibleAttribute. Boom, done. But if it only were that easy to implement. 😅 Here are my thoughts:

Multiple attributes would be required to actually trigger the code generation.

Not necessarily. Remember that you can use Roslyn to detect if a type in the user code does not have an attribute. Hence, we can create GenerateIntoNativeComWrapperAttribute and GenerateFromNativeComWrapperAttribute, which trigger generation of RCW and CCW code in the attributed partial class, respectively. (Side note: I like “into native code” and “from native code” because it’s much more obvious which is which, instead of having to remember that RCW does managed ➪ native and CCW does native ➪ managed, as we must do currently.) Therefore, the user can define their own type (internal or public) that subclasses ComWrappers and add those attributes to it to have the appropriate code generated. If there is no type in the project that uses one of those Generate…ComWrapperAttributes, a suitable ComWrappers implementation will be synthesized automatically by the source generator.

What if two different ComWrappers-derived types have a GenerateComWrappersForAttribute that point to the same interface? Which one do we use for default marshalling?

The ComWrappers instance and marshaling code is generated as a single unit. As such, could we not hardcode the instance/marshaling site relationship in the generated RCW or CCW code by using global:: syntax? This should unambiguously identify a specific implementation of ComWrappers.

COM has a platform-agnostic source of truth with IDL files and TLBs (type libraries).

Not (entirely) true! People often forget about the the win32metadata project, which contains a representation of just about every COM type in the Windows SDK, including those that ship without IDL files. If this source generator is to be useful, it must be able to interoperate with CsWin32 as far as consuming Windows.Win32.winmd is concerned. As such, why not ship this generator in the CsWin32 bits? A developer can use either the existing generator, or this COM interop generator, but if they use both, the output of the former will be compatible with the latter.

[jkoritzinsky]

I fully support Option 3 in the markdown document. As I see it, the developer should be able to simply replace ComImportAttribute with GeneratedComImportAttribute (or whatever) in their code and recompile. Same goes for ComVisibleAttribute. Boom, done. But if it only were that easy to implement. 😅 Here are my thoughts:

Multiple attributes would be required to actually trigger the code generation.

Not necessarily. Remember that you can use Roslyn to detect if a type in the user code does not have an attribute. Hence, we can create GenerateIntoNativeComWrapperAttribute and GenerateFromNativeComWrapperAttribute, which trigger generation of RCW and CCW code in the attributed partial class, respectively. (Side note: I like “into native code” and “from native code” because it’s much more obvious which is which, instead of having to remember that RCW does managed ➪ native and CCW does native ➪ managed, as we must do currently.) Therefore, the user can define their own type (internal or public) that subclasses ComWrappers and add those attributes to it to have the appropriate code generated. If there is no type in the project that uses one of those Generate…ComWrapperAttributes, a suitable ComWrappers implementation will be synthesized automatically by the source generator.

I've incorporated some of this feedback into Option 5 for the code-gen.

What if two different ComWrappers-derived types have a GenerateComWrappersForAttribute that point to the same interface? Which one do we use for default marshalling?

The ComWrappers instance and marshaling code is generated as a single unit. As such, could we not hardcode the instance/marshaling site relationship in the generated RCW or CCW code by using global:: syntax? This should unambiguously identify a specific implementation of ComWrappers.

I've provided a specific example of where the "two ComWrappers" problem would arise.

COM has a platform-agnostic source of truth with IDL files and TLBs (type libraries).

Not (entirely) true! People often forget about the the win32metadata project, which contains a representation of just about every COM type in the Windows SDK, including those that ship without IDL files. If this source generator is to be useful, it must be able to interoperate with CsWin32 as far as consuming Windows.Win32.winmd is concerned. As such, why not ship this generator in the CsWin32 bits? A developer can use either the existing generator, or this COM interop generator, but if they use both, the output of the former will be compatible with the latter.

I've addressed the reasons why we wouldn't use the Win32Metadata format in an update part of the doc. The main issues are that the primary use case for the generator is for non-Win32 COM APIs, which the Win32Metadata project has no plans for supporting. Additionally, CsWin32 has no plans of supporting consuming multiple Win32Metadata files, so even hacking together your own Win32Metadata file wouldn't enable a good experience.

[tannergooding]

For what its worth, Win32Metadata starts out via ClangSharp processing the C/C++ headers.

The process is basically:
C/C++ Headers -> ClangSharpPInvokeGenerator -> C# Source Files -> Win32MetadataSpecificPostProcessing -> WinMD Files

The WinMD Files then get consumed by other projects, such as CsWin32, RsWin32, CppWin32, etc to produce the "consumable output".

Extending the pipeline to expose all the required metadata and to consume that metadata such that CsWin32 can successfully use some of the shared generator infrastructure logic would probably involve:

• ClangSharp (myself)
• Win32Metadata (Steve Otteson)
• CsWin32 (Andrew Arnott)
• Interop Team

[wjk]

the primary use case for the generator is for non-Win32 COM APIs

That's perfectly all right. I just wanted to ensure that Win32Metadata will be supported, as it's the only managed definition for many desirable Windows native APIs.

[wjk]

Another important point I just thought of: Since you cannot directly feed one source generator's output into another, we will need to have the ability to place the generation-triggering attributes in a user-code file that does not contain the definition of the type that code should be generated for. Even though private COM APIs are the primary target of this project, please do not forget that this it will also be used with Windows APIs in a great many cases.

[tannergooding]

These are apparently the only two codebases in all of open-source that wrap DirectX in a comprehensive enough fashion (without using C++/CLI) to serve as a general-purpose interop library

@wjk. This gets slightly off topic, but I maintain https://github.com/terrafx/terrafx.interop.windows, which is powered by https://github.com/microsoft/clangsharp (which I also maintain). ClangSharp itself also powers win32metadata as the first step in the overall pipeline

The bindings are 1-to-1 raw/unsafe interop bindings and about as close to #include <Windows.h> as you can get in .NET. This includes DX10, DX11, DX12, D2D1, DWrite, WIC, XAudio2, and more and some basic samples showing DX12 usage: https://github.com/terrafx/terrafx.interop.windows/blob/main/samples/DirectX/D3D12/HelloMultiSampling12.cs

You end up doing a bit more manual management and unsafe code yourself; but at the benefit of less overhead, things that are correct by default, no missing functionality, additional type safety, and no having to unnecessarily deal with IntPtr and Marshal APIs

[wjk]

@tannergooding Please accept my apologies for forgetting about your library. In hindsight that entire paragraph was rather obviously biased towards only my use-cases anyway, so I deleted it. We must remember that the use-cases that we foresee do not necessarily contain every use-case someone will try to use this source generator for. (One of my major criticisms of the Linux movement is the refusal for maintainers to consider points of view, use cases, or design patterns that they did not think of or do not agree with personally.) But let's get back on topic here, shall we? 😄

[RussKie]

/cc: @JeremyKuhne @weltkante FYI

[AaronRobinsonMSFT]

The ExportMetadata property

It might be better to be explicit about what the "metadata" format. Based on some offline conversations it would seem IDL is what is going to be the source generator's artifact. From that, a TLB could be generated if needed.

Also, I just realized that expecting this to be a TLB is going to make the cross-platform scenario weird since the attribute would mean nothing without us implementing CreateTypeLib() for non-Windows... blech.