Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Whisper language detection #1097

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ae9is
Copy link

@ae9is ae9is commented Dec 13, 2024

See #302

Adds support for automatically detecting language to Whisper tasks.

The existing HuggingFace and Whisper implementations in Python were used as reference:
Hugging Face Transformers
Original Whisper

Also updates the existing Whisper test suites, including adding a string similarity check for actual model output (as opposed to just output length). Please note that the "new" development dependency for these tests, "fastest-levenshtein" is already used by "webpack-cli".

@xenova
Copy link
Collaborator

xenova commented Dec 13, 2024

Thanks for the PR! This will certainly be a useful feature. Regarding the implementation, I think it can be greatly simplified as follows:

  • Instead of using .generate, perform a single forward pass of the inputs
  • Then, consider all logits which correspond to the language token ids
  • Choose the language with the highest score

Currently, the implementation seems to perform a full generation step (could be hundreds of forward passes).

@ae9is
Copy link
Author

ae9is commented Dec 15, 2024

Sorry about that, it was simpler to code and the performance impact for my app was minimal! I've reworked things now to only run one pass for language detection.

Thanks for all the work on this library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants