Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CoreML support #59

Merged
merged 1 commit into from
Jun 3, 2023
Merged

Conversation

drasticactions
Copy link
Contributor

This PR adds new scripts and a nuget nuspec for Whisper CoreML support. No additional APIs are required for this to work, just the runtime files and the mlmodelc files for your whisper models. You can read about that support here: https://github.com/ggerganov/whisper.cpp#core-ml-support

The biggest difference from a user perspective is that if you wish to use CoreML support, you need to build the mlmodelc files yourself, and you have to load your whisper model from a file rather than a buffer.

The mlmodelc files are, at the moment, not shared by the whisper.cpp project (they did at one point, but the repo hosting them is either removed or hidden). There are some hosted elsewhere on hugging face, but IMO, unless we built and hosted them ourselves, I feel bad linking to random builds by others.

The file usage is a limitation of whisper.cpp, which searches for the mlmodelc file based on the model path. Using the buffer would mean no file path, so it would always fail to load. Until that changes, there's not much we can do here.

@sandrohanea
Copy link
Owner

Hello again @drasticactions,
This looks great!

Thanks for the continuous contributions and support across many platforms.
Will test it asap and will also probably add some examples in https://github.com/sandrohanea/whisper.net/tree/main/examples

@sandrohanea
Copy link
Owner

Tested it on my mac (pretty old one) without the new ML chips but it was working :)
Indeed, I will probably host the models on hugging face, and create some downloader (same as the regular models). Thanks again for the contribution!

@sandrohanea sandrohanea merged commit 315190b into sandrohanea:main Jun 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants