Support not using Win32 APIs that aren't allowed for Microsoft Store UWP apps & games on NativeAOT

[ahmed605]

Problem

Using UWP is a requirement for creating & publishing apps for Xbox without being in the XDK/ID@Xbox program, and realistically it's the only way community members can write games and apps for Xbox since they won't be approved to join the ID@Xbox program.

Currently apps compiled using NativeAOT are statically importing these Win32 functions:

GetProcessGroupAffinity
IsProcessInJob
QueryInformationJobObject
SetXStateFeaturesMask 
VirtualAllocExNuma

While these functions work perfectly under UWP, they aren't whitelisted by Microsoft Store for UWP apps, a list of whitelisted Win32 APIs for apps that target Windows SDK 22621.755 can be found here. (explanation why this list should be used instead of docs)

Suggested solution

There's two possible ways this can be fixed:

• Adding an MSBuild property that prevents NAOT from using these APIs

  • Example:

    <UseUapCompatibleImports>true</UseUapCompatibleImports>

    Another possible name can be UseOneCoreCompatibleImports.

• Adding an MSBuild property to make NAOT use dynamic loading through LoadLibrary and GetProcAddress for specified set of imports instead of statically linking them

  • Example:

    <DynamicallyLoadedImports>
        Kernel32.dll!GetProcessGroupAffinity; 
        Kernel32.dll!IsProcessInJob; 
        Kernel32.dll!QueryInformationJobObject;
        ...
    </DynamicallyLoadedImports>

    And since ApiSets are a thing I suggest making the library part (ex. Kernel32.dll!) optional

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Problem

Using UWP is a requirement for creating & publishing apps for Xbox without being in the XDK/ID@Xbox program, and realistically it's the only way community members can write games and apps for Xbox since they won't be approved to join the ID@Xbox program.

Currently apps compiled using NAOT are statically importing these Win32 functions:

GetProcessGroupAffinity
IsProcessInJob
QueryInformationJobObject
SetXStateFeaturesMask 
VirtualAllocExNuma

While these functions work perfectly under UWP, they aren't whitelisted by Microsoft Store for UWP apps, a list of whitelisted Win32 APIs for apps that target Windows SDK 22621.755 can be found here.

Suggested solution

There's two possible ways this can be fixed:

• Adding an MSBuild property that prevents NAOT from using these APIs

  • Example:

    <UseUapCompatibleImports>true</UseUapCompatibleImports>

    Another possible name can be UseOneCoreCompatibleImports.

• Adding an MSBuild property to make NAOT use dynamic loading through LoadLibrary and GetProcAddress for specified set of imports instead of statically linking them

  • Example:

    <DynamicallyLoadedImports>
        Kernel32.dll!GetProcessGroupAffinity; 
        Kernel32.dll!IsProcessInJob; 
        Kernel32.dll!QueryInformationJobObject;
        ...
    </DynamicallyLoadedImports>

    And since ApiSets are a thing I suggest making the library part (ex. Kernel32.dll!) optional

Author:	ahmed605
Assignees:	-
Labels:	area-Meta
Milestone:	-

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Problem

Using UWP is a requirement for creating & publishing apps for Xbox without being in the XDK/ID@Xbox program, and realistically it's the only way community members can write games and apps for Xbox since they won't be approved to join the ID@Xbox program.

Currently apps compiled using NativeAOT are statically importing these Win32 functions:

GetProcessGroupAffinity
IsProcessInJob
QueryInformationJobObject
SetXStateFeaturesMask 
VirtualAllocExNuma

While these functions work perfectly under UWP, they aren't whitelisted by Microsoft Store for UWP apps, a list of whitelisted Win32 APIs for apps that target Windows SDK 22621.755 can be found here.

Suggested solution

There's two possible ways this can be fixed:

• Adding an MSBuild property that prevents NAOT from using these APIs

  • Example:

    <UseUapCompatibleImports>true</UseUapCompatibleImports>

    Another possible name can be UseOneCoreCompatibleImports.

• Adding an MSBuild property to make NAOT use dynamic loading through LoadLibrary and GetProcAddress for specified set of imports instead of statically linking them

  • Example:

    <DynamicallyLoadedImports>
        Kernel32.dll!GetProcessGroupAffinity; 
        Kernel32.dll!IsProcessInJob; 
        Kernel32.dll!QueryInformationJobObject;
        ...
    </DynamicallyLoadedImports>

    And since ApiSets are a thing I suggest making the library part (ex. Kernel32.dll!) optional

Author:	ahmed605
Assignees:	-
Labels:	untriaged, area-NativeAOT-coreclr
Milestone:	-

[tannergooding]

Notably https://learn.microsoft.com/en-us/uwp/win32-and-com/win32-apis is the official list of supported APIs and is likely a better resource than the checked in xml file referenced.

[Sergio0694]

cc. @tommcdon

Of course, another option is also to just have these APIs being added to the set of allowed APIs for UWP apps (which would also make sense, given as there's nothing inherently dangerous about these APIs, not to mention the fact that as mentioned, one could literally just use LoadLibrary + GetProcAddress to use them, and that would pass the Store certification just fine). This is just an issue with the WACK tool that Partner Center runs, and not really a limitation of UWP per se.

That said, assuming there was interest in making this scenario viable (which I would certainly love to see, because like, apps and games using NativeAOT on Xbox, what's not to love), it would be nice to at least add some smoke tests or something to easily validate that new updates to NativeAOT don't accidentally bring in any new imports that might break WACK.

[ahmed605]

Notably https://learn.microsoft.com/en-us/uwp/win32-and-com/win32-apis is the official list of supported APIs and is likely a better resource than the checked in xml file referenced.

This list is very old and outdated, Microsoft Partner Center itself uses that xml file which comes with Windows App Certification Kit.

note that there's a bug in WACK where it replaces the xml file that is bundled with Windows SDK with a really old version, and that's why that repo the file I linked from exists, to preserve the original xml file for each Windows SDK version (which is what Partner Center actually use, not the outdated one).

[jkotas]

Currently apps compiled using NativeAOT are statically importing these Win32 functions:

How did you compile this list? I expect that many more APIs are going to show up once you start using libraries like crypto or networking.

another option is also to just have these APIs being added to the set of allowed APIs on UWP

Yes. The workarounds for disallowed APIs caused a constant pain when we were targeting UWP subset. I do not think we want to go there again.

[ahmed605]

How did you compile this list? I expect that many more APIs are going to show up once you start using libraries like crypto or networking.

By creating a basic small game in UWP, compiled it using NAOT, then ran it against WACK.
but yeah, networking might show more but afaik crypto APIs are fully allowed under UWP

[MichalStrehovsky]

I think we'd basically need to restore the uap RID for this. This is not just what the AOT compiler makes, it's also about how the pieces of the the libraries or even the native runtime (GC) are built. E.g. the affinity API is called by the GC, not from managed code:

runtime/src/coreclr/gc/windows/gcenv.windows.cpp

Line 226 in 4838097

if (GetProcessGroupAffinity(GetCurrentProcess(), &groupCount, NULL) || GetLastError() != ERROR_INSUFFICIENT_BUFFER)

There was a lot of UAP code in the frameworks that was nuked from orbit in dotnet/corefx#41759, we'd probably need that back.

[Sergio0694]

I have a couple questions on that:

• Considering that in this scenario, these UWP apps would use the same exact .NET as everyone else (and it would no longer be that weird combination of the .NET Core v5 CoreCLR for UWP + .NET Native in release, with custom ref assemblies and whatever), and also that this time around there's no need for any built-in WinRT support (since these apps would just be using normal CsWinRT, potentially with WUX/MUX prototype microsoft/CsWinRT#1421 in case they also wanted to use XAML), would the work to bring back some uap RID be significantly less than it originally was?
  • That is, wouldn't such a uap RID only include few targeted changes where needed to ensure things work fine when running in the UWP container, but otherwise being much simpler than what it used to be in that linked PR?
• Is this something that would actually be considered?
• I can see this also being useful for packages bundling native dependencies. This would allow them to include UAP-specific binaries (when applicable, eg. for the PIX libraries), and then publishing would automatically pick those up as you'd expect. What would this mean for libraries with no UAP-specific libs though? Would uap just fallback to using normal win binaries, if there's no uap folder in the NuGet package? This would avoid work from library authors in case they were bundling the same native library that would already work fine under both scenarios.

[jkotas]

That is, wouldn't such a uap RID only include few targeted changes

Answering this question would require analysis of the delta between allowed UAP APIs and what's actually used by libraries in the repo now. My guess is that it would not be a few target changes. Also, UAP is its own unique OS from infrastructure point of view. Adding a new unique OS to the official build and test matrix is very non-trivial cost.

[ahmed605]

Also, UAP is its own unique OS from infrastructure point of view

I don't think .NET should add special handling for anything UAP other than these disallowed functions, at least for now, I have tested many parts of the runtime under UWP and none of what I have tested seem to have any problems, both under CoreCLR and NAOT

So I think the uap RID should just map to the win RID except in places we don't want it to map, such as places that use disallowed APIs

Is there a list of all Win32 APIs that .NET runtime (or its libraries) would use?

[jkotas]

Is there a list of all Win32 APIs that .NET runtime (or its libraries) would use?

This list would need to be generated by analyzing the repo source code or by analyzing binaries produced by the repo build.

[MichalPetryka]

I think it's worth noting that such support will likely be needed once Unity decides to migrate from IL2CPP to NativeAOT for their XBOX support so I'd assume the discussion is really whether such support needs to be added now or in a few years when they request it.

[MichalStrehovsky]

I think it's worth noting that such support will likely be needed once Unity decides to migrate from IL2CPP to NativeAOT for their XBOX support so I'd assume the discussion is really whether such support needs to be added now or in a few years when they request it.

Unity doesn't request platform support. They're not going to request Playstation builds, for example. They have their own fork with separate build and testing. There can be overlaps in supported platforms but all of it is orthogonal. E.g. The fact that Loongson contributes loongarch support still doesn't mean we make official builds or test it.

[driver1998]

If one can workaround the WACK checks with GetProcAddress, it might be possible to fabricate custom static libs to override these imports with GetProcAddress shims.

See Chuyu-Team/YY-Thunks#66 where people are trying to make NativeAOT work in Windows XP, it does not seems to need a CoreCLR recompile according to the author.

[Sergio0694]

Closing this one, as no changes are actually needed. That whitelist of Win32 APIs is only applied to submissions using .NET Native, but native apps are not subject to it. And publishing a UWP app with NativeAOT is exactly identical to publishing a UWP app with C++/WinRT (or any other language). The security is enforced via the sandbox and all the other restrictions that UWP has anyway. You can declare a P/Invoke for whatever you want, and if that API is not safe, it'll just not work at runtime.

TLDR: the Store will not block UWP apps using NativeAOT just because they are importing arbitrary Win32 APIs.