dotnet tools use wrong architecture of libhostfxr.dylib on Mac M1

[mattjohnsonpint]

When installing a tool with dotnet tool install --global on a Mac M1 that has both arm64 and x64 versions of the .NET 6 SDK installed, the tool installs but fails to run because it tries to use the x64 version of libhostfxr.dylib instead of the arm64 version. The tool gives the following error:

Failed to load /usr/local/share/dotnet/x64/host/fxr/6.0.8/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/x64/host/fxr/6.0.8/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/x64/host/fxr/6.0.8/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/x64/host/fxr/6.0.8/libhostfxr.dylib failed

Installing with --arch arm64 does not help. The problem perisists.

The only workaround I've found is to install the tool with --arch x64 so that the x64 version of the tool uses the x64 version of libhostfxr.dylib. This is not ideal though, as it runs the tool under Rosetta emulation - so it is slower.

I've encountered this twice with two different tools, so I think it's a problem with the tooling infrastructure rather than these specific libraries, but see the following for examples:

• dotnet-lambda failed to load libhostfxr.dylib on Mac M1 aws/aws-lambda-dotnet#1273
• XHarness fails to run on Mac M1 unless installed with x64 architecture xharness#945

Might be related to #21187 or #26417

Thanks.

[mattjohnsonpint]

Output of dotnet --info (arm64 versions):

.NET SDK (reflecting any global.json):
 Version:   6.0.400
 Commit:    7771abd614

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.5
 OS Platform: Darwin
 RID:         osx.12-arm64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.400/

global.json file:
  Not found

Host:
  Version:      6.0.8
  Architecture: arm64
  Commit:       55fb7ef977

.NET SDKs installed:
  6.0.400 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Output of /usr/local/share/dotnet/x64/dotnet --info (x64 versions):

.NET SDK (reflecting any global.json):
 Version:   6.0.400
 Commit:    7771abd614

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.5
 OS Platform: Darwin
 RID:         osx.12-x64
 Base Path:   /usr/local/share/dotnet/x64/sdk/6.0.400/

global.json file:
  Not found

Host:
  Version:      6.0.8
  Architecture: x64
  Commit:       55fb7ef977

.NET SDKs installed:
  2.1.818 [/usr/local/share/dotnet/x64/sdk]
  3.1.421 [/usr/local/share/dotnet/x64/sdk]
  3.1.422 [/usr/local/share/dotnet/x64/sdk]
  5.0.408 [/usr/local/share/dotnet/x64/sdk]
  6.0.400 [/usr/local/share/dotnet/x64/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.30 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.30 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.27 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.28 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.8 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.30 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.27 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.28 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.8 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]

[baronfel]

Yeah, this is a gap with tool install - I believe you can get around it for now with the --arch option on dotnet tool install. If you specify --arch arm64 the correct apphost and runtime should be used. I think this has been logged in a couple other places as well, I'll try to locate and link them.

[mattjohnsonpint]

arch --arm64 installs the arm version of the tool, but it then still tries to use the x64 version of the libhostfxr.dylib at run time.

Only installing arch --x64 works.

I believe the issue that it's scanning the /usr/local/share/dotnet directory for the first version of the dylib, and finds the one in the x64 directory first and then stops scanning.

[mattjohnsonpint]

Ah, I figured it out. It is indeed a runtime issue (feel free to migrate this to dotnet/runtime).

The error I get is from here:
https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/native/corehost/apphost/standalone/hostfxr_resolver.cpp#L35-L52

Tracing in to fxr_resolver::try_get_path, I end up here:
https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/native/corehost/fxr_resolver.cpp#L55

I see a comment:

// For framework-dependent apps, use DOTNET_ROOT_<ARCH>

So I set in my ~/.zshenv:

export DOTNET_ROOT_ARM64=/usr/local/share/dotnet
export DOTNET_ROOT_X64=/usr/local/share/dotnet/x64

Then I uninstall and reinstall the tool (with either --arch arm64 or without specifying) and now it works correctly.

[mattjohnsonpint]

I'd say it's still a bug in that by default after installing the SDKs, those environment variables are not set. Probably the runtime should continue to try to resolve one matching the architecture of the tool rather than giving up after finding the mismatch.

[vitek-karas]

What is in your /etc/dotnet/ ?

The env. variables are typically only for private (.zip) installs. Globally installed runtime should register itself in /etc/dotnet/.
To diagnose this better you can also try to collect a trace: export COREHOST_TRACE=1 , export COREHOST_TRACEFILE=host.txt and repro the failure. The host.txt will have information about why it was looking for the wrong file. (Please be aware that the file contains local paths and environment before sharing it).

[mattjohnsonpint]

@vitek-karas - In /etc/dotnet/ there are two files, install_location and install_location_x64. Both contain the same path /usr/local/share/dotnet/x64. I updated install_location to point at /usr/local/share/dotnet and undid my environment variable changes, and everything is working fine. So now the question is, how did they get set that way to begin with?

[vitek-karas]

The right question is: why is there no install_location_arm64? The two you have are correct, but there should be an arm64 one if you have arm64 installed.
Did you install the arm64 SDK/runtime via the official installers? (those should write the file there).

[mattjohnsonpint]

Ok. I just deleted everything to start over:

sudo rm -rf /usr/local/share/dotnet
sudo rm -rf /etc/dotnet
sudo rm -rf ~/.dotnet

Then I reinstalled the latest x64 and arm64 versions of .NET 6 SDK and the x64 of .NET Core 3.1. I get all three folders, install_location, install_location_x64, and install_location_arm64. I tried every combination of sequence of installing them, and it always comes out the same and works fine.

So, I'm not sure how I ended up not having an install_location_arm64 file originally, but that was the problem. Perhaps a bug with an older SDK installer that has since been resolved? Not sure. I've only used the official installers.

Anyway, since it seems to work correctly with the current versions, I'll close the issue. Thanks for the assistance!