RFC: dotnet [Arm64 | x64] coexistence #16896
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
richlander
changed the title
|
This issue is a kind of RFC. I'm currently leaning to "Hide architectural differences" option 3. It offers much of the same experience as "Native dotnet" while providing a little more wiggle room than some of the other options. |
+1
Why not park the frameworks, sdks and everything else under the x64 directory? ie make this |
|
That makes total sense. My main point was that the I'm not going to update the document now. Let's wait for a first round of review and then produce a new doc with the final plan. |
|
I really like Hide Architectural Differences option 3. Arm64 native with x64 mixed in, keeps everything together with a less confusing folder structure and would likely solve the issue of the hostfxr.dll being loaded for the incorrect architecture due to the incorrect x64 installation path - one of the big pains with arm64 and x64 coexistence. I would absolutely LOVE to never see this again: |
|
Another nice suggestion would be to allow an option to include both arm64/x64 hostfxr.dlls in with framework-dependent deployments. Procmon lead me to discover that upon the load of a .NET executable, it first checks the directory the application exists for a hostfxr.dll. By copying an x64 hostfxr.dll to the same directory of an x64 targeted framework-dependent app and running the executable, it properly loaded (In reference to microsoft/PowerToys#10658) This is more of a short-term workaround than an actual solution, though. |
|
As you suggest, this option is more of a short term workaround. We will get this working so that this suggestion isn't needed. Also, it wouldn't work for cross platform apps on one hand (they would need a lot of hostfxr binaries) and it wouldn't match with apps that were R2R compiled. |
|
Can someone catch me up on why this is needed? Why would I want x64 dotnet on my arm64 computer, once a native version is available? I'm sure this completely outs me as a noob, but why would installing to the same path be a bad idea? So that the two architectures replace each other? |
x64 on Arm64 emulation exists currently for the Windows Insiders Dev Channel builds, so we're able to run both Arm64 and x64 applications side by side. The current structure of the .NET installation throws a wrench in the mix for being able to execute .NET Core/.NET 5+ applications that are either of those architectures - since they both install to C:\Program Files\dotnet by default. Since the same directory is being used, you're likely to have multiple SDK versions and packs installed with no way of telling whether they're x64 or Arm64. When dotnet loads the native hostfxr.dll (.NET Host Resolver) of one of the runtimes installed and it doesn't match the architecture of the calling executable (either Basically, everything being loaded by either the dotnet executable or the executable/dll produced after building your project, and the .NET runtime dlls need to be built in the same architecture. It's the same as if you had an x64 executable on an x64 build of Windows trying to load an ARM64 dll. |
This can be in the list of going forward options for the sake of completeness. Something like: Ignore the problemPremise: The non-native execution is point-in-time problem and it is not worth complicating the product for it. We will recomend to use dotnet-install script and Option 1: Maintain status quo Option 2: Block installation of non-native .NET Runtime |
I was thinking of doing this as another workaround, since the current SDK/Runtime installers don't properly respect the Instructions running the dotnet-install scripts to install the x64 runtimes to a common base path, and for setting that path in the InstallLocation REG_SZ value under EDIT: Oh. It actually worked.
$dotnetInstallPath = "HKLM:\Software\WOW6432Node\dotnet\Setup\InstalledVersions\x64"
if(!(Test-Path $dotnetInstallPath))
{
New-Item -Path $dotnetInstallPath -Force;
}
New-ItemProperty -Path $dotnetInstallPath -Name "InstallLocation" -Value "c:\dotnetx64" -PropertyType String -Force;
By this point, you should be able to run an x64-compiled .NET executable without trouble. |
I like this approach, as it's explicit and makes the most sense to me. I would also potentially explore the idea of renaming the x64 folder to amd64 for additional clarity. I firmly reject any idea that repurposes the "x64" moniker as a bucket for "64-bit" software. "x64", in the Windows community, has strong ties to amd64, and any other definition would introduce a lot of confusion. I'm not a fan of universal binaries either, as it introduces a sort of non-determinism. "Did I flip the right switches to make this universal thing go to amd64 and not arm64 for my cross-compile project?" |
|
Option 1 definitely has its merits, as you call out @riverar. The reason I like this option less than option 3 is that it is less future oriented. It says "x64 and Arm64 will be at parity forever." In Apple land, we know that's not true. We guess that Apple will stop supporting x64 emulation relatively soon, almost certainly before 2025. Windows is a different story. We've decided that we're going to make the same choice on Windows and macOS, at the very least for our sanity. Option 3 pulls ahead as the winner because the "arm64" folder quickly becomes unnecessary, at least on macOS. I understand the |
Not sure what you mean by "will be at parity forever". Is the hypothetical here that developers will see these folders and somehow get the impression that the x64 runtime will work on their machines after a hypothetical update to macOS removes Rosetta 2 support? Or something else? Struggling to understand the problem you're trying to solve there and what makes option 3 "future oriented". Apologies. |
+1 and +100 for @jkotas suggestion to keep the same folder structure for both architectures - so everything x64 goes under the x64 directory.
As mentioned we expect x64 emulation to disappear at least on macOS relatively soon. Once that happens we would like to get back to the simple world of having one dotnet installation on the machine - and that one should live in the default location - (Bit of a sidetrack but I wanted to clear this up)
In answer to @snickler comment about ability to manually install and register via registry.
|
Isn't the most consistent option then to explicitly create the architecture folders per Option 1? The "platform native" folder then stops being this thing that flaps around, requiring you to think "hm, what is platform native here?". I don't see any benefits to having this "symlink" for platform native. Just confusion. (Good luck with xplat documentation too!) |
I agree. Since most of the issues are around the apphost executables, that could be solved in this manner. For the |
|
@riverar -- the architecture folders approach is also consistent, but again, it's oriented on here and now. We don't want architectural folders in the future as Arm64 becomes the prevalent option. We have a strong philosophy that the native arch will have the good name. We expect that most other dev platforms and apps will do the same thing if they support x64 emulation. In short, this boils down to "what do we want the platform to look like in 5 and 10 year". We want it to look the same as today, simply a "dotnet" folder in the place where you install dev platforms on a given OS. After .NET Core 3.1 goes EOL, I expect we'll see x64 downloads from Arm64 machines drop significantly, below 10%. If that's true, do you hold the same view (which option to pick)? Or do you believe that the ratio will be different than the 9:1 one I've suggested? |
|
im trying to do what they are saying but its not detailed enough for me i ranh the script and it didnt ask where to install? it just auto installed somewhere now im trying to figure that part out |
|
I can't see how anything other than a universal binary is the right approach for macOS. The OS support for UB is on a completely different level than in Windows and any attempt to create two sets of binaries and switch via PATH is bound to create more problems than it solves. Also, while it is unlikely Windows will ever reach even a majority of machines running ARM, it is for all practical purposes a certainty with macOS, since all new machines sold will be ARM soon. Another point - native code constitutes only about 5% of the entire SDK package (26M out of 454M for .NET 6 preview 3, ARM64). While the remaining files differ for some reason between architecture builds, I hope this is due to some build process difference. As a quick PoC, I tried stitching the arm64 and x86_64 SDKs using |
That's not accurate. Nearly all managed .dlls in the SDK are AOT compiled and have architecture specific code in them as well. |
In that case, I have to apologize. I must have lived under a rock for a long time as I assumed the same technique is still applied as in FX 2.0 days when the .dlls got compiled only after installation. @jkotas Can you please provide some link to how this works? I'm really curious and a quick google turned out nothing relevant, just tons of R2R, CoreRT and similar stuff. |
|
AOT compilation is done during app build/publish in .NET Core. https://devblogs.microsoft.com/dotnet/conversation-about-crossgen2/ has some good context for this. |
|
FYI: This doc is very incomplete. It has just enough detail in it to get feedback on high-level choices. I am writing another doc that goes into much more detail on the chosen option. I'll link to it from here. Zooming out, I'd like to see a system where using x64 emulation was natural and didn't rely on the |
|
(Sorry, I haven't read the conversation here, jumped down from reading the initial issue) Have you folks looked at how Debian suggests where architecture-specific packages (eg, dotnet) should be located? https://www.debian.org/doc/debian-policy/ch-opersys.html:
That seems to suggest another option in addition to the ones suggested in the first post:
Perhaps we can re-use an existing policy instead of inventing our own? Btw, the FHS and Debian policy guidelines both suggest that dotnet belongs under |
|
Thanks @omajid. That makes sense in terms of a Linux option. It doesn't seem directly applicable for macOS and even less so for Windows. I think the big difference is that Linux has a strong separation between install location and the user UX in many cases, with a heavy use of symlinks. Fair? |
|
@richlander @omajid Thinking about this some more, I actually think we didn't design anything which would tie us to a specific location. The |
|
"Hide architectural differences" option 3 doesn't seem like a good solution to me. It requires the installer to detect that it is running under emulation and install to a different location in that scenario. This is not something that you can fix retroactively. For example, what if Windows decides to add Arm64 emulation to x64 OS in 5-10 years? You'll run into this same problem again, all of the Arm64 installers previously released will start installing into the native location. |
Highly unlikely scenario, but ignoring that. This same model would apply. We'd create an What's the other option that you would endorse that would enable the scenario you've outlined that allows installers to work retroactively? |
That's my point. You can't retroactively update the Arm64 installers. Maybe it could work with your option if you can make all installers, today, detect when they are running under emulation and install to the architecture specific folder underneath the |
|
We're not going to change the x64 model on x64. That's not warranted. We've been installing the |
|
It is not unusual that we have to update the shipped .NET versions in servicing to account for OS changes. We have done that number of times. One example from many: dotnet/corefx#34443 . Updating the installers for this (theoretical) case would be one of those changes. |
|
This issue has been superseded by #17464. Design conversation is hosted at dotnet/designs#217 |
Microsoft and Apple are both producing new OSes for Arm64 that support x64 emulation. For .NET, we need to determine how to support .NET Arm64 and x64 builds on those OSes in a way that offers a reasonable user experience and that has reasonable cost.
Problem to solve
As stated, we need to offer x64 and Arm64 builds of .NET within one 64-bit namespace (without much help from the OS on how to manage it). That leads to three issues:
One could reasonable ask why we need to support coexistence once native Arm64 build are available.
Context
The Windows and macOS plans and implementations are similar but have key differences, per our understanding.
The following are various high-level solutions that we could employ, with pros and cons.
Status quo
On Windows x64, .NET is installed to two locations (depending on architecture):
Today, customers need one of those two directories in the path in order to get the "dotnet" experience they want, like if they want a 32-bit or 64-bit "dotnet build".
Note: There will not be a C:\Program Files (x64) directory on Windows Arm64. We are expected to install 64-bit products (Intel and/or Arm based) in C:\Program Files.
On macOS, .NET is installed to one location:
Going forward
There will be one dotnet in the path, just like today. Much like the 32- and 64-bit support we offer on Windows, we'll offer both Arm64 and x64 builds of .NET and customers can install both or either, and can control which is in the
PATH. That will determine if they get an Arm64 or x64 dotnet build. We are not planning on building a .NET version manager that enables switching which architecture you get. We intend to rely on thePATH.The question is what structure we offer for the two 64-bit products, how intuitive that is (now and later) and how expensive that is for us to offer.
Native dotnet
Premise: there is a "dotnet" directory on every OS, and it is the native architecture.
We'd end up with the following, on Windows and macOS, respectively:
This is the usual "who gets the good name" problem.
Pros:
Cons:
Archify dotnet
Premise: fully embrace multi-arch support, with arch-specific directories.
We'd end up with the following:
On Arm64, we'd have these arch-specific directories. This is the "no one gets the good name option; everybody loses" option.
Pros:
Cons
Hide architecture differences
Premise: These differences don't need to be so apparent. We already have version folders under dotnet. We can add arch folders (or something similar). This option has a lot of sub-options, too.
Option 1 -- Insert a new folder, with discrete .NET hives underneath:
Option 2 -- Intermix architectures in one structure (here, just shown with Windows, for simplicity):
Put "x64" and "arm64" somewhere in the folder hierarchy. Where it is, is an implementation decision. It is the same (in spirit) as C:\Windows\Microsoft.NET\Framework and C:\Windows\Microsoft.NET\Framework64.
Option 3 -- Arm64 is native architecture with x64 intermixed in:
Pros:
Cons
Hide architectural differences with universal binaries
Both Windows and macOS offer a form of universal binaries, which we could use as a multiplexer between the two 64bit products. There are two problems with that.
We're rejecting this option for now. We could decide to adopt this solution on macOS and not Windows, should feedback push us in that direction.
Compatibility Wins
Premise: People have already installed .NET x64 on Apple Silicon machines. We need to respect that. This is the opposite of "Native dotnet"
We'd end up with the following:
Pros:
Cons:
Hard design questions
PATHby default (Native Dotnet solution above)?PATHis already the case since typically only the x64 dotnet is onPATH, but it's possible to use it to build, run and test x86 apps.Other considerations
DOTNET_ROOT(x64), which would be analogous to the existingDOTNET(x86)ENV.The text was updated successfully, but these errors were encountered: