Incompatible architecture errors on mac runners (Failed to load libhostfxr.dylib, libtrace.dylib, libhostpolicy.dylib)

[Haard30]

Description

Recently, our pipelines started failing during dotnet restore & dotnet test step on osx-x64 and osx-arm64.
Here's the full error message:

Failed to load /Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib, error:
dlopen(/Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib, 0x0001): tried: '/Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')),
'/System/Volumes/Preboot/Cryptexes/OS/Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib' (no such file),
'/Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
The library libhostfxr.dylib was found, but loading it from /Users/runner/.dotnet/host/fxr/8.0.4/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

We see similar issues with Initialize CodeQL actions as well:

dyld[4503]: terminating because inserted dylib
'/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' could not be loaded:
tried: '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but
missing compatible architecture (have 'x86_64,arm64', need 'arm64e')),
'/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools
osx64/libtrace.dylib' (no such file)

We see the same error for both osx-x64 and osx-arm64.

Consider 2 same runs on 6th March 2024 and 29th April 2024 (same code but one succeeding and other failing)

1. https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8177668677 (succeeded)
2. https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8884826546/job/24394808000 (failed)
   YAML file: https://github.com/AzureAD/microsoft-authentication-cli/blob/main/.github/workflows/release.yml

So far, we have tried following but with not much success:

1. Bump actions/setup-dotnet to v4. (This took us to a further step where dotnet test started failing with similar error. https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8883829023/job/24391532134)
2. Bump actions/checkout to v4.
3. Update github/codeql-action to v3.

More information:

• We are using dotnet-version: 6.0.x.

Please let me know if I can provide any additional information.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8884826546/job/24394808000

Is it regression?

https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8177668677

Expected behavior

It should work like it did for previous runs: https://github.com/AzureAD/microsoft-authentication-cli/actions/runs/8177668677/job/22362969465

Actual behavior

Failing with failed to load .dylib errors.

Repro steps

1. Run the Release pipeline.

[sergei-pyshnoi]

Hello @Haard30 . Looks like you are trying to build x64 arch using arm64 runner. Can you specify runs-on: macos-12 instead runs-on: macos-latest on your workflow and try again?

[Haard30]

Hello @Haard30 . Looks like you are trying to build x64 arch using arm64 runner. Can you specify runs-on: macos-12 instead runs-on: macos-latest on your workflow and try again?

Thanks @sergei-pyshnoi . We tried macos-12 and macos-13 and it works perfectly fine. This seems like a regression in macos-latest. Is the recommendation to use macos-13 for now (in such cases)?

[erik-bershel]

Hey @Haard30!

I'm really sorry, but it's not an image issue, but CodeQL and .NET compatibility with the arm64e issue itself. Please, report the same story here: https://github.com/github/codeql-action and https://github.com/actions/setup-dotnet.
As a temporary workaround we may recommend to use non-arm versions of macOS.