Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preparing for Apple Silicon Macs #318

Closed
4 tasks done
TheSpydog opened this issue Jul 2, 2020 · 19 comments
Closed
4 tasks done

Preparing for Apple Silicon Macs #318

TheSpydog opened this issue Jul 2, 2020 · 19 comments

Comments

@TheSpydog
Copy link
Member

TheSpydog commented Jul 2, 2020

Since the Apple Silicon Macs are on their way, and especially now that devkits are starting to ship, we need to figure out what all must be done to prepare for their arrival. Thankfully Apple is working on porting Mono to Apple Silicon, so that much is taken care of, but there are still several other items we'll need to figure out:

  • SDL2 support. I expect the new architecture (and/or Big Sur) will break parts of SDL, so it will need to be thoroughly tested. Additionally, since we only ship tagged releases of SDL2, we're limited by their release schedule. Hopefully they'll release 2.0.14 (or whatever) with Apple Silicon support sooner rather than later.
  • Native library compilation and distribution. Aside from any potential SDL2 problems, the fnalibs are already known to work well on ARM, but we'll need to figure out a new cross-compilation toolchain and a means of distributing them separately from the current osx binaries. (Maybe we could ship them in a new osx-arm folder?)
  • MonoKickstart universal binary support. I have absolutely no idea how universal binaries are supposed to work, or if they're backwards compatible with older mac versions, or how Kick can manage them.
  • The current DllMap uses target=osx arch=armv8 lines to remap iOS libs to "__Internal". This is going to cause conflicts with the new macs, so we'll need to remove those lines. Maybe we could distribute a separate app.config file for iOS?

There are probably other issues I'm not thinking of. Feel free to mention anything I missed. Thankfully we still have some time before consumer machines start shipping, so we should be able to complete all of this before then!

@flibitijibibo
Copy link
Member

flibitijibibo commented Jul 3, 2020

Compilation and distribution won't be an issue, it's basically just us going back to setting CC="cc -arch x86_64 -arch arm64" and shipping the universal binaries, like the old days. For my local setup I'll most likely continue building x86_64 builds with osxcross and then lipoing that with arm64 binaries built on a Mac (until osxcross catches up).

I'm up for making iOS/tvOS dllconfig files separate, since that's a whole other build process anyway.

We'll have to look at how Big Sur support is doing on the SDL Bugzilla. It'll suck, but that's how it is every year.

@flibitijibibo
Copy link
Member

SDL is done (Needed one asm line for breakpoints and a couple fixes in the configure script), so the only change left is the dllmap. Once the stable compiler is released I'll do a rebuild of everything and we can wrap this issue up.

@flibitijibibo
Copy link
Member

dllmap has been updated in master and the docs have been updated alongside it. Just need the stable compiler to come out...

@flibitijibibo
Copy link
Member

The macOS ARM patches for Mono have been merged:

mono/mono#20166

@flibitijibibo
Copy link
Member

osxcross has added initial support for ARM, so I've made early ARM64-only binaries for testing:

http://fna.flibitijibibo.com/archive/fnalibs_osxarm.tar.bz2

All we have to do is lipo this with the existing fnalibs and that'll wrap up fnalibs. Not sure whether or not I want to update SDKs for x86_64, but having two separate builds for macOS is pretty annoying so we'll see.

Pulling in the Mono source to try an osxcross build now.

@flibitijibibo
Copy link
Member

Mono build fails in a few places, assuming this has more to do with the build host than the ARM64 support. Important details:

Configure line:

CC=arm64-apple-darwin20-clang CXX=arm64-apple-darwin20-clang++ ./configure --host=arm64-apple-darwin20 --disable-boehm

Git diff with all the hacked lines:

diff --git a/mono/metadata/sgen-stw.c b/mono/metadata/sgen-stw.c
index e59c0c7..7f88a21 100644
--- a/mono/metadata/sgen-stw.c
+++ b/mono/metadata/sgen-stw.c
@@ -73,7 +73,7 @@ update_current_thread_stack (void *start)
 	g_assert (info->client_info.stack_start >= info->client_info.info.stack_start_limit && info->client_info.stack_start < info->client_info.info.stack_end);
 
 #if !defined(MONO_CROSS_COMPILE) && MONO_ARCH_HAS_MONO_CONTEXT
-	MONO_CONTEXT_GET_CURRENT (info->client_info.ctx);
+	//MONO_CONTEXT_GET_CURRENT (info->client_info.ctx);
 #elif defined (HOST_WASM)
 	//nothing
 #else
diff --git a/mono/mini/exceptions-arm.c b/mono/mini/exceptions-arm.c
index 4433324..3cab76c 100644
--- a/mono/mini/exceptions-arm.c
+++ b/mono/mini/exceptions-arm.c
@@ -610,7 +610,7 @@ mono_arch_handle_exception (void *ctx, gpointer obj)
 	/* Pass the ctx parameter in TLS */
 	mono_sigctx_to_monoctx (sigctx, &jit_tls->ex_ctx);
 	/* The others in registers */
-	UCONTEXT_REG_R0 (sigctx) = (gsize)obj;
+	//UCONTEXT_REG_R0 (sigctx) = (gsize)obj;
 
 	/* Allocate a stack frame */
 	sp -= 16;
diff --git a/mono/mini/interp/interp.c b/mono/mini/interp/interp.c
index 445da6d..1e85b37 100644
--- a/mono/mini/interp/interp.c
+++ b/mono/mini/interp/interp.c
@@ -1557,7 +1557,7 @@ ves_pinvoke_method (
 	args = margs;
 #endif
 
-	INTERP_PUSH_LMF_WITH_CTX (&frame, ext, exit_pinvoke);
+	//INTERP_PUSH_LMF_WITH_CTX (&frame, ext, exit_pinvoke);
 	entry_func ((gpointer) addr, args);
 	if (save_last_error)
 		mono_marshal_set_last_error ();
@@ -2190,7 +2190,7 @@ static MONO_NO_OPTIMIZATION MONO_NEVER_INLINE stackval *
 do_icall_wrapper (InterpFrame *frame, MonoMethodSignature *sig, int op, stackval *sp, gpointer ptr, gboolean save_last_error)
 {
 	MonoLMFExt ext;
-	INTERP_PUSH_LMF_WITH_CTX (frame, ext, exit_icall);
+	//INTERP_PUSH_LMF_WITH_CTX (frame, ext, exit_icall);
 
 	sp = do_icall (sig, op, sp, ptr, save_last_error);
 
diff --git a/mono/mini/mini-exceptions.c b/mono/mini/mini-exceptions.c
index 9f88c52..fa0bae4 100644
--- a/mono/mini/mini-exceptions.c
+++ b/mono/mini/mini-exceptions.c
@@ -3712,7 +3712,7 @@ mono_thread_state_init (MonoThreadUnwindState *ctx)
 #if defined(MONO_CROSS_COMPILE)
 	ctx->valid = FALSE; //A cross compiler doesn't need to suspend.
 #elif MONO_ARCH_HAS_MONO_CONTEXT
-	MONO_CONTEXT_GET_CURRENT (ctx->ctx);
+	//MONO_CONTEXT_GET_CURRENT (ctx->ctx);
 #else
 	g_error ("Use a null sigctx requires a working mono-context");
 #endif
diff --git a/mono/utils/mono-codeman.c b/mono/utils/mono-codeman.c
index 24c90a3..94a167c 100644
--- a/mono/utils/mono-codeman.c
+++ b/mono/utils/mono-codeman.c
@@ -662,7 +662,7 @@ void
 mono_codeman_enable_write (void)
 {
 #ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
-	if (__builtin_available (macOS 11, *)) {
+	if (1) { /* __builtin_available (macOS 11, *)) { */
 		int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
 		level ++;
 		mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
@@ -681,7 +681,7 @@ void
 mono_codeman_disable_write (void)
 {
 #ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
-	if (__builtin_available (macOS 11, *)) {
+	if (1) { /* __builtin_available (macOS 11, *)) { */
 		int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
 		g_assert (level);
 		level --;
diff --git a/mono/utils/mono-context.c b/mono/utils/mono-context.c
index 62b1feb..8f22b17 100644
--- a/mono/utils/mono-context.c
+++ b/mono/utils/mono-context.c
@@ -451,9 +451,9 @@ mono_sigctx_to_monoctx (void *sigctx, MonoContext *mctx)
 	mctx->pc = UCONTEXT_REG_PC (my_uc);
 	mctx->regs [ARMREG_SP] = UCONTEXT_REG_SP (my_uc);
 	mctx->cpsr = UCONTEXT_REG_CPSR (my_uc);
-	memcpy (&mctx->regs, &UCONTEXT_REG_R0 (my_uc), sizeof (host_mgreg_t) * 16);
+	//memcpy (&mctx->regs, &UCONTEXT_REG_R0 (my_uc), sizeof (host_mgreg_t) * 16);
 #ifdef UCONTEXT_REG_VFPREGS
-	memcpy (&mctx->fregs, UCONTEXT_REG_VFPREGS (my_uc), sizeof (double) * 16);
+	//memcpy (&mctx->fregs, UCONTEXT_REG_VFPREGS (my_uc), sizeof (double) * 16);
 #endif
 #endif
 }
@@ -479,9 +479,9 @@ mono_monoctx_to_sigctx (MonoContext *mctx, void *ctx)
 	UCONTEXT_REG_SP (my_uc) = mctx->regs [ARMREG_SP];
 	UCONTEXT_REG_CPSR (my_uc) = mctx->cpsr;
 	/* The upper registers are not guaranteed to be valid */
-	memcpy (&UCONTEXT_REG_R0 (my_uc), &mctx->regs, sizeof (host_mgreg_t) * 12);
+	//memcpy (&UCONTEXT_REG_R0 (my_uc), &mctx->regs, sizeof (host_mgreg_t) * 12);
 #ifdef UCONTEXT_REG_VFPREGS
-	memcpy (UCONTEXT_REG_VFPREGS (my_uc), &mctx->fregs, sizeof (double) * 16);
+	//memcpy (UCONTEXT_REG_VFPREGS (my_uc), &mctx->fregs, sizeof (double) * 16);
 #endif
 #endif
 }

I believe the only people other than us that currently care about this are the Godot team, per godotengine/godot-build-scripts#10. @akien-mga, some notes above for whenever you look at this!

@akien-mga
Copy link

Thanks for the ping! CC @neikeq if you want to start poking at this for Godot.

@flibitijibibo
Copy link
Member

flibitijibibo commented Aug 20, 2020

Found the issue, turns out Mono's configure scripts have a bug in them even for the official build environment where you also need to specify target in addition to host. There also seems to be a mismatch where Apple's toolchain uses aarch64 while osxcross uses arm64. This gets the master branch to build after fixing config.sub, similar to SDL:

CC=arm64-apple-darwin20-clang CXX=arm64-apple-darwin20-clang++ ./configure --host=arm64-apple-darwin20 --target=arm64-apple-darwin20
diff --git a/configure.ac b/configure.ac
index 72abca2..861148b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -452,6 +452,10 @@ case "$host" in
                                RID="osx-x64"
                                COREARCH="x64"
                                ;;
+                       arm64*-darwin*)
+                               # osxcross/arm64
+                               support_boehm=no
+                               ;;
                        arm*-darwin*)
                                platform_ios=yes
                                has_dtrace=no
@@ -4634,6 +4638,13 @@ case "$host" in
                CPPFLAGS="$CPPFLAGS -D__ARM_EABI__"
                ;;
 
+       arm64*-darwin*)
+               TARGET=ARM64;
+               arch_target=arm64;
+               boehm_supported=false
+               ACCESS_UNALIGNED="no"
+               ;;
+
        arm*-darwin*)
                TARGET=ARM;
                arch_target=arm;
diff --git a/mono/utils/mono-codeman.c b/mono/utils/mono-codeman.c
index 24c90a3..3561850 100644
--- a/mono/utils/mono-codeman.c
+++ b/mono/utils/mono-codeman.c
@@ -662,13 +662,17 @@ void
 mono_codeman_enable_write (void)
 {
 #ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
+#if !TARGET_ARM64
        if (__builtin_available (macOS 11, *)) {
+#endif
                int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
                level ++;
                mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
                pthread_jit_write_protect_np (0);
+#if !TARGET_ARM64
        }
 #endif
+#endif
 }
 
 /*
@@ -681,13 +685,17 @@ void
 mono_codeman_disable_write (void)
 {
 #ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
+#if !TARGET_ARM64
        if (__builtin_available (macOS 11, *)) {
+#endif
                int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
                g_assert (level);
                level --;
                mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
                if (level == 0)
                        pthread_jit_write_protect_np (1);
+#if !TARGET_ARM64
        }
 #endif
+#endif
 }

MonoKickstart diff, hoping this name is fixed for x86_64 as well...

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 505a9b3..ed8f4dc 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,7 +16,8 @@ if (APPLE)
     SET(BIN_RPATH "@executable_path/osx")
     SET(KICKLIBS
         iconv
-        "-Wl,-force_load,${CMAKE_SOURCE_DIR}/../mono/mono/native/.libs/libmono-native-compat.a"
+        z
+        "-Wl,-force_load,${CMAKE_SOURCE_DIR}/../mono/mono/native/.libs/libmono-native.a"
     )
 elseif (CMAKE_SIZEOF_VOID_P MATCHES "8" AND NOT(FORCE32))
     SET(BIN_LIBROOT "lib64")

@flibitijibibo
Copy link
Member

flibitijibibo commented Aug 20, 2020

Latest osxcross adds the aarch64 triple, so cross compiling now works with SDL and Mono with no modifications to the automake files:

CC=aarch64-apple-darwin20-clang CXX=aarch64-apple-darwin20-clang++ ./autogen.sh --host=aarch64-apple-darwin20 --target=aarch64-apple-darwin20

mono-codeman.c and CMakeLists.txt patches still apply.

@flibitijibibo
Copy link
Member

flibitijibibo commented Dec 22, 2020

Fresh Apple Silicon builds of fnalibs are here: http://fna.flibitijibibo.com/archive/fnalibs_arm64.tar.bz2

No MonoKickstart build because the Extinguish stage for .NET kicked in and we don't have standalone Mono releases anymore - the last one did not backport Apple Silicon support so for now this is screwed until we can get everyone together to take over maintainership of Mono.

@flibitijibibo
Copy link
Member

I now have Universal Binaries working on flibitBuild... only had to compile Clang/LLVM from scratch. The builds are not in fnalibs.tar.bz2 yet, waiting on this report to be resolved:

KhronosGroup/MoltenVK#1182

Once I have that I'll update fnalibs to be universal. Still no word on Mono, Steamworks, or basically anybody except for us.

@akien-mga
Copy link

akien-mga commented May 12, 2021

I've been poking at Mono 6.12.0.144 (which includes a backport of ARM64 support for the 2020-02 branch) and ran into the mono-codeman.c issue described in #318 (comment)

Namely that __builtin_available doesn't seem to work with osxcross. @flibitijibibo Did you look more into it, and should we report it to https://github.com/tpoechtrager/osxcross?

This simple example fails building:

int main() {
  if (__builtin_available(macOS 11, *)) {
    return 1;
  }
  return 0;
}

Edit: My bad, it does compile fine if I don't forget to specify the min macOS version (x86_64-apple-darwin20.2-clang -mmacosx-version-min=11.0 main.c). So I guess the issue is that it can only build if using 11.0 as min version? (MACOS_VERSION_MIN in Mono makefile)

@flibitijibibo
Copy link
Member

I believe this requires building Apple's Clang/LLVM source code, but I don't totally remember - for arm64 I just made a temporary def to make it always use the path that Big Sur would use (didn't even bother with x86_64 at the time since I could just lipo it to my old build made from Mojave). Whatever the fix is, it's at least worth adding to the osxcross README.

@akien-mga
Copy link

I opened tpoechtrager/osxcross#278 to keep track of that __builtin_available issue.

For now I'm also patching Mono to work it around, but that means not have this code running when actually running x86_64 binaries on Big Sur.

diff --git a/mono/utils/mono-codeman.c b/mono/utils/mono-codeman.c
index 234aac4b..608fad42 100644
--- a/mono/utils/mono-codeman.c
+++ b/mono/utils/mono-codeman.c
@@ -666,8 +666,10 @@ mono_code_manager_size (MonoCodeManager *cman, int *used_size)
 void
 mono_codeman_enable_write (void)
 {
-#if defined(HAVE_PTHREAD_JIT_WRITE_PROTECT_NP) && defined(TARGET_OSX)
-	if (__builtin_available (macOS 11, *)) {
+#if defined(HAVE_PTHREAD_JIT_WRITE_PROTECT_NP) && defined(TARGET_OSX) && defined(TARGET_ARM64)
+        // __builtin_available only works on osxcross if min version matches the version we check,
+        // so it's pointless. Hardcode ARM64 == Big Sur or later.
+	if (1) { // (__builtin_available (macOS 11, *)) {
 		int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
 		level ++;
 		mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
@@ -685,8 +687,10 @@ mono_codeman_enable_write (void)
 void
 mono_codeman_disable_write (void)
 {
-#if defined(HAVE_PTHREAD_JIT_WRITE_PROTECT_NP) && defined(TARGET_OSX)
-	if (__builtin_available (macOS 11, *)) {
+#if defined(HAVE_PTHREAD_JIT_WRITE_PROTECT_NP) && defined(TARGET_OSX) && defined(TARGET_ARM64)
+        // __builtin_available only works on osxcross if min version matches the version we check,
+        // so it's pointless. Hardcode ARM64 == Big Sur or later.
+        if (1) { // (__builtin_available (macOS 11, *)) {
 		int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
 		g_assert (level);
 		level --;

@akien-mga
Copy link

I opened tpoechtrager/osxcross#278 to keep track of that __builtin_available issue.

Turns out that one needs compiler-rt for this, which is an optional build step for osxcross. Installing it solves the issue, so mono-codeman.c no longer needs to be patched.

@flibitijibibo
Copy link
Member

flibitijibibo commented May 14, 2021

Can confirm that Mono and the fnalibs compile with the necessary components built and installed.

At this point FNA has done everything it can to make Apple Silicon support possible, and a Mono release has Apple Silicon support officially. This is enough to produce a DRM-free build of an FNA game!

However, we're still blocked by basically the entire rest of the industry:

  • Steamworks
  • Galaxy
  • Epic Online Services
  • FMOD
  • Wwise
  • RAD/Epic Game Tools
  • Many, many others

These are all essential parts of the back catalog, and while many of these middlewares provide arm64 support for their newest version, the lack of Steamworks/Galaxy support in any capacity along with no backports for older versions make providing arm64 by default extremely difficult, since the Universal Binary means potentially running arm64 with incomplete dependencies, resulting in the game not being loadable. Having developers selectively run lipo in just about every scenario is out of the picture since it's as annoying as it is fragile (I had to fight build mixups a lot when Mono SGen only worked on i386). Simply put, it's easier and safer to selectively add arm64 for the one narrow case than it is to remove it for virtually every build made today.

So, here is FNA's official plan for Apple Silicon as of now:

  • Developers who really want Apple Silicon for their DRM-free releases can now compile their own arm64 binaries using the Big Sur SDK, and lipo them to the x86_64 binaries we already ship.
  • fnalibs/MonoKickstart providing arm64 by default will hinge entirely on Valve shipping Steam for Apple Silicon, with GOG Galaxy, Epic Games Store, and the Itch app as secondary pressure. Should this actually happen, we can just tell developers to lipo remove -arch arm64 kick.bin.osx -output kick for stores that don't support it yet.

Do I expect stores to ship for Apple Silicon? Truthfully, I don't think so - if I were running a store my decision would be based on the outcome of Apple vs. Epic Games; if Apple wins (and, cynically, this is what I expect will happen), I would use it as an excuse to end support for the Mac since you could combine the complete loss of a massive back catalog with the inevitable move to an App-Store-only Mac, which Apple would now be 100% allowed to do (and unlike Windows 10 S it would probably actually sell).

I'm keeping a close eye on what stores are doing since I still have a crapload of Mac games to support, but I'm not hopeful.

@kode54
Copy link

kode54 commented Oct 18, 2022

Is this no longer a supported configuration? I assume that's why you removed the dependencies packages.

@crjenkins
Copy link

Truthfully, another half-assed lib that claims to support Mac desktop. Please remove it from your list of supported platforms.

@MBeijer
Copy link

MBeijer commented Jan 14, 2023

@crjenkins did you even read any of the comments in this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants