Skip to content

feat(ios): embed ggml-metal source into framework binary#349

Merged
jhen0409 merged 2 commits into
mybigday:mainfrom
kiyohken2000:fix/metal-embed-library
May 25, 2026
Merged

feat(ios): embed ggml-metal source into framework binary#349
jhen0409 merged 2 commits into
mybigday:mainfrom
kiyohken2000:fix/metal-embed-library

Conversation

@kiyohken2000
Copy link
Copy Markdown
Contributor

Summary

Fixes #348 by switching from the runtime .metal file load (introduced in #252) to the embedded-source pattern that upstream llama.cpp uses behind GGML_METAL_EMBED_LIBRARY. The merged Metal source is baked into the framework binary as a __DATA symbol range and consumed directly by ggml-metal-device.m's embedded code path.

This addresses the maintenance concern from #252 (no per-target .metallib to ship or precompile) while removing the runtime NSBundle resource lookup that #348 reported as failing on iPhone A15 with XPC_ERROR_CONNECTION_INTERRUPTED.

cpp/ggml-metal/ggml-metal-device.m itself is unchanged — its #if GGML_METAL_EMBED_LIBRARY branch already supports this path; we only define the macro, generate the asm wrapper, and link it.

Changes

  • scripts/bootstrap.sh — after the LM_ prefix sed pass, emit cpp/ggml-metal/ggml-metal-embed.s with the merged metal source as .byte directives between _lm_ggml_metallib_start and _lm_ggml_metallib_end symbols.
  • ios/CMakeLists.txt — add ASM to project(... LANGUAGES ...) so the Xcode generator registers a build rule for .s sources, define LM_GGML_METAL_EMBED_LIBRARY=1, glob *.s from ggml-metal/ into the rnllama target, and set_source_files_properties(... LANGUAGE ASM).
  • llama-rn.podspec — extend RNLLAMA_BUILD_FROM_SOURCE=1 source glob to include *.s, drop cpp/ggml-metal/ggml-metal.metal from s.resources, define the macro alongside LM_GGML_USE_METAL whenever Metal is enabled.
  • scripts/build-ios.sh — stop copying ggml-metal.metal into the framework root in copy_framework_support_files.
  • AGENTS.md — describe the new bootstrap step.
  • cpp/ggml-metal/ggml-metal-embed.s — checked-in bootstrap output (mirrors how cpp/ggml-metal/ggml-metal.metal is already checked in).

Cross-platform porting notes (worth a quick look)

  • od -An -vtx1 -w16 works on Linux/Git-Bash but macOS BSD od rejects -w16 with illegal option. The script omits -w16; both GNU and BSD od default to 16 bytes per line.
  • The Mach-O section name is __ggml_metallib (15 chars), not __lm_ggml_metallib (18 chars). Mach-O caps section specifiers at 16 characters, so this had to drop the LM_ prefix. Only the exported symbols (_lm_ggml_metallib_start / _end) carry the LM_ prefix.
  • Without LANGUAGES ASM on project(), the Xcode generator silently drops .s files from the build phase. The simulator x86_64 link step is the first to fail with Undefined symbols: _lm_ggml_metallib_start/_end. With LANGUAGES ASM + set_source_files_properties(... LANGUAGE ASM), all four xcframework slices compile and link cleanly.

Testing

Open questions

  • Flag-gate? I kept the embed path unconditional rather than gating it behind an env var (e.g. LLAMA_METAL_EMBED=0 to fall back to the file-based path). Happy to add a flag if you'd prefer keeping the file-based path available as a fallback — let me know.
  • Android scope — these changes are iOS-only. Android doesn't use Metal so it's a non-issue; not touching android/build.gradle.

🤖 Generated with Claude Code

Fixes mybigday#348.

Following the precompile-rollback rationale in mybigday#252, switch to upstream
llama.cpp's GGML_METAL_EMBED_LIBRARY pattern: emit ggml-metal-embed.s
during bootstrap, link it into the rnllama framework, and let
ggml-metal-device.m's embedded code path supply the Metal source to
newLibraryWithSource: directly. Removes the .metal resource shipping
and the LM_GGML_METAL_PATH_RESOURCES fallback chain.

ggml-metal-device.m itself is unchanged — its #if GGML_METAL_EMBED_LIBRARY
branch already supports this path; we only define the macro, generate the
asm wrapper, and link it.

Cross-platform notes from porting:

- scripts/bootstrap.sh: emit ggml-metal-embed.s right after the LM_
  prefix sed pass. The od invocation intentionally omits -w16 because
  macOS BSD od rejects it ("illegal option -- w"); GNU od and BSD od
  both default to 16 bytes per line.
- The Mach-O section name is __ggml_metallib (15 chars), not the
  LM_-prefixed variant, because Mach-O caps section specifiers at
  16 characters. Only the exported symbols carry the LM_ prefix.
- ios/CMakeLists.txt: project() needs LANGUAGES ASM so the Xcode
  generator emits a build rule for .s sources; without it, .s files
  are silently dropped from the build phase and the link step fails
  with undefined _lm_ggml_metallib_{start,end}.

Tested on iPhone 13 mini (A15) with iOS 26.5 — initLlama succeeds and
the native log reports "lm_ggml_metal_library_init: using embedded
metal library" / "loaded in 9.690 sec" instead of the
"default.metallib not found ... XPC_ERROR_CONNECTION_INTERRUPTED"
chain from the mybigday#348 repro. nm confirms _lm_ggml_metallib_start /
_lm_ggml_metallib_end are present as data section symbols (S) in all
four xcframework slices (ios-arm64, ios-simulator arm64+x86_64,
tvos-arm64, tvos-simulator arm64+x86_64).
kiyohken2000 added a commit to kiyohken2000/feedown that referenced this pull request May 22, 2026
Captures the three corrections discovered while porting on macOS:

- 01-bootstrap.diff: drop `od -w16` (macOS BSD od rejects it); use
  Mach-O section name `__ggml_metallib` (15 chars) instead of the
  18-char LM_-prefixed variant — only the exported symbols carry LM_.
- 02-cmake.diff: add `ASM` to `project(... LANGUAGES ...)` and
  `set_source_files_properties(... LANGUAGE ASM)` so the Xcode
  generator actually emits a build rule for the .s wrapper.

These drafts are now in sync with mybigday/llama.rn#349.
@jhen0409 jhen0409 force-pushed the fix/metal-embed-library branch from de45212 to 2254a86 Compare May 25, 2026 04:57
Copy link
Copy Markdown
Member

@jhen0409 jhen0409 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also removed ggml-metal-embed.s and ignore it because it is not worth to track and it's also unreadable for diff.

  • The latest CI failure looks not related to this PR.

@jhen0409 jhen0409 merged commit 2897793 into mybigday:main May 25, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] iOS: "Failed to load model" — default.metallib not bundled, runtime Metal source compilation fails with XPC_ERROR_CONNECTION_INTERRUPTED

2 participants