feat(ios): embed ggml-metal source into framework binary#349
Merged
Conversation
Fixes mybigday#348. Following the precompile-rollback rationale in mybigday#252, switch to upstream llama.cpp's GGML_METAL_EMBED_LIBRARY pattern: emit ggml-metal-embed.s during bootstrap, link it into the rnllama framework, and let ggml-metal-device.m's embedded code path supply the Metal source to newLibraryWithSource: directly. Removes the .metal resource shipping and the LM_GGML_METAL_PATH_RESOURCES fallback chain. ggml-metal-device.m itself is unchanged — its #if GGML_METAL_EMBED_LIBRARY branch already supports this path; we only define the macro, generate the asm wrapper, and link it. Cross-platform notes from porting: - scripts/bootstrap.sh: emit ggml-metal-embed.s right after the LM_ prefix sed pass. The od invocation intentionally omits -w16 because macOS BSD od rejects it ("illegal option -- w"); GNU od and BSD od both default to 16 bytes per line. - The Mach-O section name is __ggml_metallib (15 chars), not the LM_-prefixed variant, because Mach-O caps section specifiers at 16 characters. Only the exported symbols carry the LM_ prefix. - ios/CMakeLists.txt: project() needs LANGUAGES ASM so the Xcode generator emits a build rule for .s sources; without it, .s files are silently dropped from the build phase and the link step fails with undefined _lm_ggml_metallib_{start,end}. Tested on iPhone 13 mini (A15) with iOS 26.5 — initLlama succeeds and the native log reports "lm_ggml_metal_library_init: using embedded metal library" / "loaded in 9.690 sec" instead of the "default.metallib not found ... XPC_ERROR_CONNECTION_INTERRUPTED" chain from the mybigday#348 repro. nm confirms _lm_ggml_metallib_start / _lm_ggml_metallib_end are present as data section symbols (S) in all four xcframework slices (ios-arm64, ios-simulator arm64+x86_64, tvos-arm64, tvos-simulator arm64+x86_64).
kiyohken2000
added a commit
to kiyohken2000/feedown
that referenced
this pull request
May 22, 2026
Captures the three corrections discovered while porting on macOS: - 01-bootstrap.diff: drop `od -w16` (macOS BSD od rejects it); use Mach-O section name `__ggml_metallib` (15 chars) instead of the 18-char LM_-prefixed variant — only the exported symbols carry LM_. - 02-cmake.diff: add `ASM` to `project(... LANGUAGES ...)` and `set_source_files_properties(... LANGUAGE ASM)` so the Xcode generator actually emits a build rule for the .s wrapper. These drafts are now in sync with mybigday/llama.rn#349.
de45212 to
2254a86
Compare
jhen0409
approved these changes
May 25, 2026
Member
jhen0409
left a comment
There was a problem hiding this comment.
Thanks! I also removed ggml-metal-embed.s and ignore it because it is not worth to track and it's also unreadable for diff.
- The latest CI failure looks not related to this PR.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #348 by switching from the runtime
.metalfile load (introduced in #252) to the embedded-source pattern that upstream llama.cpp uses behindGGML_METAL_EMBED_LIBRARY. The merged Metal source is baked into the framework binary as a__DATAsymbol range and consumed directly byggml-metal-device.m's embedded code path.This addresses the maintenance concern from #252 (no per-target
.metallibto ship or precompile) while removing the runtime NSBundle resource lookup that #348 reported as failing on iPhone A15 withXPC_ERROR_CONNECTION_INTERRUPTED.cpp/ggml-metal/ggml-metal-device.mitself is unchanged — its#if GGML_METAL_EMBED_LIBRARYbranch already supports this path; we only define the macro, generate the asm wrapper, and link it.Changes
scripts/bootstrap.sh— after theLM_prefix sed pass, emitcpp/ggml-metal/ggml-metal-embed.swith the merged metal source as.bytedirectives between_lm_ggml_metallib_startand_lm_ggml_metallib_endsymbols.ios/CMakeLists.txt— addASMtoproject(... LANGUAGES ...)so the Xcode generator registers a build rule for.ssources, defineLM_GGML_METAL_EMBED_LIBRARY=1, glob*.sfromggml-metal/into the rnllama target, andset_source_files_properties(... LANGUAGE ASM).llama-rn.podspec— extendRNLLAMA_BUILD_FROM_SOURCE=1source glob to include*.s, dropcpp/ggml-metal/ggml-metal.metalfroms.resources, define the macro alongsideLM_GGML_USE_METALwhenever Metal is enabled.scripts/build-ios.sh— stop copyingggml-metal.metalinto the framework root incopy_framework_support_files.AGENTS.md— describe the new bootstrap step.cpp/ggml-metal/ggml-metal-embed.s— checked-in bootstrap output (mirrors howcpp/ggml-metal/ggml-metal.metalis already checked in).Cross-platform porting notes (worth a quick look)
od -An -vtx1 -w16works on Linux/Git-Bash but macOS BSDodrejects-w16withillegal option. The script omits-w16; both GNU and BSDoddefault to 16 bytes per line.__ggml_metallib(15 chars), not__lm_ggml_metallib(18 chars). Mach-O caps section specifiers at 16 characters, so this had to drop theLM_prefix. Only the exported symbols (_lm_ggml_metallib_start/_end) carry the LM_ prefix.LANGUAGES ASMonproject(), the Xcode generator silently drops.sfiles from the build phase. The simulator x86_64 link step is the first to fail withUndefined symbols: _lm_ggml_metallib_start/_end. WithLANGUAGES ASM+set_source_files_properties(... LANGUAGE ASM), all four xcframework slices compile and link cleanly.Testing
gemma-3-4B-it-Q4_K_M.initLlamasucceeds and the native log shows:default.metallib not found … XPC_ERROR_CONNECTION_INTERRUPTEDchain reported in the issue. (The 4B model itself then hits A15 Metal working-set limits —current allocated size is greater than the recommended max working set size— but that's an independent memory-pressure issue, not the [Bug] iOS: "Failed to load model" — default.metallib not bundled, runtime Metal source compilation fails with XPC_ERROR_CONNECTION_INTERRUPTED #348 XPC failure.)nmverification —_lm_ggml_metallib_startand_lm_ggml_metallib_endare present as data section symbols (S) in all four xcframework slices:ios-arm64,ios-arm64_x86_64-simulator,tvos-arm64,tvos-arm64_x86_64-simulator. NoU(undefined) entries.npm run typecheck— clean.Open questions
LLAMA_METAL_EMBED=0to fall back to the file-based path). Happy to add a flag if you'd prefer keeping the file-based path available as a fallback — let me know.android/build.gradle.🤖 Generated with Claude Code