[runtime] Infrastructure to build runtime components using NativeAOT

[lambdageek]

Motivation: While the .NET runtime itself is perfectly capable of running .NET code, we also provide other components that currently have to be implemented in native code. For example the DAC - its job is to introspect a live CoreCLR process or a memory dump and provide the information to diagnostic tools. The DAC is not a .NET runtime and it exposes a native interface to other tooling. But as we evolve the DAC it might make sense to implement parts of it in C#.

Native runtime component libraries using NativeAOT

This PR allows managed libraries to be compiled using NativeAOT and to be used in the runtime using normal CMake idioms.

Adding a new managed library

Add a new subdirectory to src/native/managed for your library with a src and inc subdirectories:

$ mkdir -p libMyNewLibrary/src libMyNewLibrary/inc
$ dotnet new classlib -n libMyNewLibrary -o libMyNewLibrary/src

In src/native/managed/compile-native.proj, add
src/native/managed/libMyNewLibrary/src/libMyNewLibrary.csproj to the NativeLibsProjectsToBuild
item group.

In src/native/managed/libMyNewLibrary/src/libMyNewLibrary.csproj:

1. Near the top, add <Import Project="..\..\native-library.props" />
2. Near the bottom, add <Import Project="..\..\native-library.targets" />
3. Define an item @(InstallRuntimeComponentDest) that has directory names relative to artifacts/bin/<runtimeFlavor>/<os.arch.config>/ where the shared library should be installed. It's a good idea to have at least .:

     <ItemGroup>
         <InstallRuntimeComponentDest Include="." />
         <InstallRuntimeComponentDest Include="sharedFramework" Condition="'$(RuntimeFlavor)' == 'coreclr'"/>
     </ItemGroup>

Limitations:

• The project should be called libXXXX - currently the infrastructure expects a lib prefix on all platforms.
• Currently only shared library output is supported. In principle static linking is possible, but the
  infrastructure is not finished yet. Additionally, mixing Debug/Release configurations with static
  linking will not be supported on Windows.

---

Ideally we'd like to be able to produce shared or static libraries. Currently only shared libraries really work robustly (in that only the exported native symbols of the C# library are exposed and the NativeAOT implementation details are completely hidden).

There was an attempt at doing static linking too, but it needs a bit more work on Linux/Mac (using -r to create relocatable objects in conjunction with -Wl,--exclude-libs or -Wl,-hidden-l to hide the symbols of the NativeAOT runtime). Doing naïve static linking results in duplicate GC and eventpipe symbols (since CoreCLR and NativeAOT both define them). Static linking on Windows probably isn't worthwhile due to #98356 and because I'm not sure if hiding symbols will be possible.

There was some work to have cmake integration: runtime could do find_nativeaot_library(libWhatever [REQUIRED]) and get two cmake targets: libWhatever::libs and libWhatever::headers that could be used for linking with other targets. The current usecase doesn't need this level of integration however, so the cmake stuff was moved to this branch: https://github.com/lambdageek/runtime/tree/naot-runtime-lib-with-cmake

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Motivation: While the .NET runtime itself is perfectly capable of running .NET code, we also provide other components that currently have to be implemented in native code. For example the DAC - its job is to introspect a live CoreCLR process or a memory dump and provide the information to diagnostic tools. The DAC is not a .NET runtime and it exposes a native interface to other tooling. But as we evolve the DAC it might make sense to implement parts of it in C#.

This PR: we provide two things: an example, and the CMake and MSBuild infrastructure for defining src/native components that are actually written in C# that will be compiled using NativeAOT into native code which can in turn be consumed by src/coreclr or src/mono (although currently only the coreclr cmake integration is done).

Ideally we'd like to be able to produce shared or static libraries. Currently only shared libraries really work robustly (in that only the exported native symbols of the C# library are exposed and the NativeAOT implementation details are completely hidden).

There's an attempt at doing static linking too, but it needs a bit more work on Linux/Mac (using -r to create relocatable objects in conjunction with -Wl,--exclude-libs or -Wl,-hidden-l to hide the symbols of the NativeAOT runtime). Doing naïve static linking results in duplicate GC and eventpipe symbols (since CoreCLR and NativeAOT both define them). Static linking on Windows probably isn't worthwhile due to #98356 and because I'm not sure if hiding symbols will be possible.

The example is just a library that writes a message to the console when mscordac is loaded. We shouldn't merge that bit.

Possible improvements before this is ready to merge:

• we need to produce a .dylib.dwarf, .so.dbg, or .pdb symbol file for the imported library like the rest of the build
• consider making a FindPackage dependency provider and generate a cmake config-package (cmake: Using Dependencies Guide for each library instead of doing simple include files.
• Annoyingly, cmake doesn't consider the modification time of IMPORTED libraries when deciding whether to rebuild a target. So if you change a managed library and then do build.sh clr it won't necessarily pick up the changes. Ithink it might be possible to fix it up by adding some custom commands that make it seem like the imported libraries are actually generated by the cmake build

Author:	lambdageek
Assignees:	lambdageek
Labels:	area-Infrastructure-libraries
Milestone:	-

[lambdageek]

I'm going to remove the example now. I've seen it build with the latest changes and there's nothing interesting in the example itself anymore. I think I'm ready for another round of reviews

[lambdageek]

I think I want to remove the libFoo::headers imported target. for the cdac work we end up using dlopen/LoadLibrary to get at the nativeaot'd lib, so there's not a build-time dependency between the shared library and the code that consumes it. In fact it simplifies the build dependencies a bit if locating the include files doesn't depend on AOTing the library - and also allows the crossdacs to consume the cdac reader (which is, by design, debuggee platform agnostic)

[jkoritzinsky]

I think I want to remove the libFoo::headers imported target. for the cdac work we end up using dlopen/LoadLibrary to get at the nativeaot'd lib, so there's not a build-time dependency between the shared library and the code that consumes it. In fact it simplifies the build dependencies a bit if locating the include files doesn't depend on AOTing the library - and also allows the crossdacs to consume the cdac reader (which is, by design, debuggee platform agnostic)

I think I agree here. If we exclusively reference the exports lazily through dlopen/LoadLibrary, then we don't need to have the header target.

If we were to link (dynamically or statically) against the lib, then I think we should expose the header as part of the same target that has the lib so linking against one CMake target would pull everything in needed to use that target.

In fact, if we are going to reference at runtime with dlopen/LoadLibrary, we don't need the full CMake integration at all. We only need one (imported?) target that adds the include directory for the header that has the function pointer types for dynamic interaction.

[lambdageek]

In fact, if we are going to reference at runtime with dlopen/LoadLibrary, we don't need the full CMake integration at all.

yea, that's true. I'll move the cmake stuff to a separate branch/PR and keep it on the back burner until we have a need for it.

If we were to link (dynamically or statically) against the lib, then I think we should expose the header as part of the same target that has the lib so linking against one CMake target would pull everything in needed to use that target.

I'm not sure I totally agree. Particularly for static linking it's sometimes helpful to only have a single target that depends on the libs, while multiple targets depend on the headers. In any case, that can be hashed out later - when we have a need for the cmake stuff.

[lambdageek]

Removed cmake integration. Verified that the cdac stuff still builds.

[jkoritzinsky]

A few comments to reduce the MSBuild logic and reduce the number of steps to set up a new project.

[jkoritzinsky]

LGTM!