[Voxtral Realtime] Introduce global log mel max - #33574
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a global log_mel_max configuration for Voxtral's realtime audio processing. This change aims to improve transcription stability by normalizing log-mel spectrograms against a global average instead of a short-term frame-based maximum. The implementation correctly adds logic to use this global value in compute_whisper_melspec and updates the model configuration loading to support it. The tests have also been updated to reflect the improved and more consistent transcription outputs, removing previous workarounds. My main feedback is on improving the robustness of the type checking in voxtral.py.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
|
Hi @patrickvonplaten, the pre-commit checks have failed. Please run: uv pip install pre-commit
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, Tip Is
|
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Pai <416932041@qq.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Pai <416932041@qq.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
For true realtime the log_mel should not be normalized based only on very short log_mel frames - instead we should try to normalize over a "global" avg. Some extensive evals show that setting a global value leads to better & more stable transcriptions