-
-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/add mlx #73
Feature/add mlx #73
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LeonNissen Thank you for the nice additions and working on this PR; very important and great to move to MLX for this package!
I have added some higher-level comments that I identified and added to the git diff.
Please ensure that all UI tests are passing + REUSE and SwiftLint elements are fixed. I also observe a lot of Swift 6 warning in the PR. I know that @paulhdk is also having an open PR where he is addressing most of them; please ensure that you sync with him and @philippzagar to ensure that these elements are in sync.
Apart from this I am happy to see this merged and addressed in this PR; we might want to tag a 2.0 beta version after this is merged and we had some time to bake it a bit together with the PRs from @paulhdk 🚀
Sources/SpeziLLMLocal/Helpers/ModelConfiguration+PromptFormat.swift
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the updates; let me know once you need an additional round of reviews and have identified a way to pass all UI tests and remaining checks 🚀
add note to README
fix memory selection
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @LeonNissen; amazing additions and thank you for incorporating all the feedback!
Replace Llama.cpp with MLX
♻️ Current situation & Problem
Llama.cpp has been a reliable, multiplatform option. However, Apple’s recent release of MLX provides significant performance improvements, especially optimized for Apple Silicon devices. This PR proposes migrating to MLX to leverage these optimizations.
⚙️ Release Notes
Note: MLX-Swift is currently incompatible with simulators, so testing and deployment require a physical Apple Silicon device. Additionally, when using MLX with large language models (LLMs), the Increase Memory Limit Entitlement must be added to the application’s entitlements for optimal functionality.
This update removes certain APIs from the existing configuration. Please ensure your implementation aligns with the new MLX API.
🚀 Benchmark
Benchmarks indicate substantial performance gains across all tested devices. Refer to the figure below for detailed results:
📝 Code of Conduct & Contributing Guidelines
By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines: