Support Apple Silicon (ARM64)

[bdkjones]

I realize it was just announced, but there are six months until ARM Macs are in the wild. To be ready for them, apps that integrate libmultimarkdown need to be able to build a Universal version of the library. To rebuild, they'll first need a Universal version of the MultiMarkdown library.

The CMake dependency probably complicates this transition a lot because I imagine it won't be possible to build a Universal version of libmultimarkdown until CMake supports ARM64.

However, because this is a blocker for any apps that include MultiMarkdown, I wanted to open the issue so progress could be tracked and any related problems could be discussed. Thanks for your help!

[fletcher]

You can tweak the Xcode project to build for whatever architectures you want that Xcode supports. It might take a while before CMake offers that as an automatic choice, or it might be possible to manually override it today. But either way, you can adjust the Xcode project to do whatever you like.

[bdkjones]

It sounds like a new version of CMake may not be required, just a tweak to the makefile:

https://mobile.twitter.com/kubamracek/status/1275545458706755584

[nickasd]

I'm trying to build MultiMarkdown with the make xcode comand (I have Xcode 12.0.1), and I get the following error:

cd build-xcode; \
	cmake -G Xcode ..
-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -- broken
CMake Error at /usr/local/Cellar/cmake/3.15.1/share/cmake/Modules/CMakeTestCCompiler.cmake:60 (message):
  The C compiler

    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: MultiMarkdown-6/build-xcode/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/xcodebuild -project CMAKE_TRY_COMPILE.xcodeproj build -target cmTC_a1b9a -configuration Debug -hideShellScriptEnvironment && Command line invocation:
        /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -project CMAKE_TRY_COMPILE.xcodeproj build -target cmTC_a1b9a -configuration Debug -hideShellScriptEnvironment
    
    User defaults from command line:
        HideShellScriptEnvironment = YES
    
    Prepare build
    note: The Legacy Build System will be removed in a future release. You can configure the selected build system and this deprecation message in File > Project Settings.
    === BUILD TARGET cmTC_a1b9a OF PROJECT CMAKE_TRY_COMPILE WITH CONFIGURATION Debug ===
    
    Check dependencies
    No architectures to compile for (ONLY_ACTIVE_ARCH=YES, active arch=arm64, VALID_ARCHS=arm64 arm64e i386 x86_64, EXCLUDED_ARCHS=(
        arm64,
        arm64e
    )).
    
    ** BUILD FAILED **
    
    
    The following build commands failed:
    	Check dependencies
    (1 failure)
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:24 (project)


-- Configuring incomplete, errors occurred!
See also "MultiMarkdown-6/build-xcode/CMakeFiles/CMakeOutput.log".
See also "MultiMarkdown-6/build-xcode/CMakeFiles/CMakeError.log".
make: *** [xcode] Error 1

Is this related to what you were saying above? If yes, what do I have to do to make the build work? (I'm not an expert with CMake and environment variables.)

[fletcher]

I'll need to look at the CMakeLists.txt file -- I think it is related to X86_64 not being the "only" architecture when building for macOS any more.

[fletcher]

I think you can fix this when building for x86_64 by adding SET (CMAKE_OSX_ARCHITECTURES x86_64) before the project (${My_Project_Title}) line near the top of the file. This is likely a bandaid and a better fix is needed, and this is obviously not what you want to do when building for arm64.

[nickasd]

Thanks a lot, that line fixed the build. Unfortunately I'm unable to support you in providing a general fix (it sounds like we need a different solution for arm64), since I know very little about CMake etc, but it would be great if someone could figure out how to fix this in the near future.

[fletcher]

I am running macOS 10.15.7 on a Late 2014 Mac mini (surprisingly this machine does everything I need it to!) and using Xcode 12.0.1.

Whether I use MultiMarkdown or a new Xcode project, I don't seem to be able to compile for arm64 (Mac), only x86_64. I can build for arm64 for iOS (last I checked anyways). My Architectures Build Setting shows "Standard architectures - $(ARCHS_STANDARD)" not "Standard architectures (64-bit Intel and ARM) - $(ARCHS_STANDARD)" as seen here

Is there something obvious I am missing?

I'm trying to come up with the proper Cmake configuration to allow universal binaries in anticipate of Silicon computers becoming widespread.

Thanks!

[fletcher]

I just pushed a couple of updates. 6.6.0 uses a revised Cmake configuration that works better for me on Xcode 12 with x86_64.

I think it should work better when transitioning towards ARM, but have not tested it yet (I don't have an ARM machine).

Input appreciated!

[bdkjones]

Thanks for jumping on this! I have a total of 24 third-party libraries to recompile for Apple Silicon and it's been a nightmare of trying to figure out most of them. It's nice to see this project staying ahead of the curve!

[bdkjones]

Okay, I've built the framework for x86_64 and arm64. Here's my experience:

1. Downloaded the "latest release" tag's source as a zip archive.
2. Updated Cmake to 3.18
3. Ran make
4. Ran make xcode

This produced an error:

CMake Error at CMakeLists.txt:166 (project):
No CMAKE_C_COMPILER could be found. 

It sounded to me like the script couldn't find Cmake something-or-other, so I unzipped a fresh copy of the source code and then:

3. Opened the CMake GUI
4. Clicked "Configure"
5. Clicked "Generate"
6. Selected Xcode as the generator and left all options as-is.

CMake now built the Xcode project. I opened that in Xcode and noticed that several settings were messed up by CMake:

• Architectures
• Base SDK
• Build Active Architectures Only

So, I then:

7. Opened the "libMultiMarkdown" scheme in the scheme editor and changed the build configuration from debug to release.
8. Selected the libMultiMarkdown project and adjusted the architectures to this:

9. Selected the "libmultimarkdown" target and changed these same settings.
10. Built the project (on my x86_64 MacBook Pro)

Inspecting the compiled framework with lipo -archs <path-to-framework> returns the expected result:

arm64 x86_64

[bdkjones]

Hmm. I may have spoken too soon. When I attempt to USE the compiled framework in my app, I now get several errors related to undefined functions for

token_pool_init()
token_pool_drain()
token_pool_free()

I think that's related to 8afd127#diff-47babe9dd5e41858b6944d0e59f0f8656922ae67efa9f1251f4ef09d6e10ff08

In the headers I'm getting, this change is not included; I still see:

#define kUseObjectPool_disabled 1

I can modify the header, but when I do that the Linker gives me this:

I think if I modify the header BEFORE I compile the framework, that should solve this?

[fletcher]

@bdkjones -- a couple of issues

1. If you're using latest release then you are missing some of the fixes. For this you should be using the develop branch.
2. You don't need to run make, then make xcode. make prepares a CMake build using gcc/clang from the command line. make xcode prepares an xcodeproj file that can be used in Xcode or xcodebuild. The two are independent. It's not harmful to run make first, just unnecessary.
3. I can't speak to the initial error you describe since I can't test it without an ARM machine. That sounds like Xcode command line utilities were not properly configured?? Or it's related to not using the develop branch, as above.
4. Yes -- you have to change #defines before compiling. They are compile time directives, not run time directives. And yes, if you don't have a strong understanding of memory management, should definitely have the object pool disabled when using libMultiMarkdown and not multimarkdown. It makes multithreaded programs much more difficult.

[bdkjones]

Perfect. I'll re-build from the latest develop commit and confirm the Universal binary is working.

As for the token pools, I think I've done that correctly? My app runs the multimarkdown compiler in a separate process over XPC. It's set up like this:

[fletcher]

That looks right, but you have to be certain that libMultiMarkdown can only be used on a single thread at a time. The object pool is a single shared instance, so freeing it in one thread would wreck another thread that was using it. In a multithreaded application, that might be difficult, and might take more overhead and trouble than you gain in performance.

Also, it doesn't speed compiling time, it speeds processing time when converting text to HTML.

[bdkjones]

Yea. As far as I know, the XPC service is single-threaded. It doesn't matter how many threads from the host application call into the service, the service has only one thread. I've tested it by pummeling the thing with hundreds of MD files to compile and have never had a crash.

By "compiling" in the comment I meant the processing of MD to HTML; not the compiling of the app in Xcode.

[bdkjones]

I can also confirm that the Universal binary is working (at least on x86_64). After cloning the repo (instead of downloading the zipped source code from the latest release tag) running make xcode in Terminal worked perfectly. I opened the Xcode project and adjusted the settings as described in the post above. That resulted in a correct, Universal binary.

[fletcher]

What were the arch settings prior to changing them? It should have defaulted to Standard architectures and "No" to build active only. The goal is for you not to have to manually adjust anything for standard uses of the library.

[bdkjones]

These are the settings from make xcode:

(Although I have the "all build" target selected, the settings are identical for all other targets in the list.)

I'm running macOS 11.0.1 and Xcode 12.2 — the final, GM releases of both.

[fletcher]

Can you make a fresh copy of the .xcodeproj, open it, change one setting and then undo that change (in order to ensure Xcode rewrites the file), and then zip/send it to me? I'm trying to see what is different.

On my build libMultiMarkdown has Build Active Arch Only set to No and Arch set to $(ARCHS_STANDARD).

Thanks!

(It might be a while before I get a Silicon machine, so I can test this myself yet.)

[bdkjones]

My app uses 29 libraries/frameworks, so I've been using CMake a LOT. It's doing the same thing on all other libraries I'm building as well—the same settings show up in the generated Xcode project.

[fletcher]

What version of CMake are you using?

[nickasd]

Since updating to macOS Big Sur, I get this output for the command make xcode:

git log master..develop --format="*    %s" | sort | uniq > CHANGELOG-UNRELEASED
mkdir build-xcode 2>/dev/null
make: [build-xcode] Error 1 (ignored)
cd build-xcode; rm -rf *
cd build-xcode; \
	cmake -G Xcode ..
-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
CMake Error at CMakeLists.txt:167 (project):
  No CMAKE_C_COMPILER could be found.



CMake Error at CMakeLists.txt:167 (project):
  No CMAKE_CXX_COMPILER could be found.



-- Configuring incomplete, errors occurred!
See also "MultiMarkdown-6/build-xcode/CMakeFiles/CMakeOutput.log".
make: *** [xcode] Error 1

I ran xcode-select --install as suggested on Stackoverflow, but still the same output. Anyone knows what's the problem?

[nickasd]

After uninstalling Homebrew and hence cmake, reinstalling both, and running sudo xcode-select --reset, it worked again.

[DivineDominion]

With the CMakeList changes introduced recently, I building on Intel Macs for both architectures works. I believe this also solves building on an ARM Mac. Can anyone with an M1 Mac try the latest develop tip?

[fletcher]

Closing -- I have been using libMultiMarkdown via CMake for nvUltra for quite a while and building for x86 and ARM universal binaries without trouble.