Skip to content

[Voxtral Realtime] Introduce global log mel max - #33574

Merged
mgoin merged 5 commits into
vllm-project:mainfrom
patrickvonplaten:pvp/global_log_mel_max
Feb 2, 2026
Merged

mgoin merged 5 commits into
vllm-project:mainfrom
patrickvonplaten:pvp/global_log_mel_max

Conversation

@patrickvonplaten

Copy link
Copy Markdown
Collaborator

For true realtime the log_mel should not be normalized based only on very short log_mel frames - instead we should try to normalize over a "global" avg. Some extensive evals show that setting a global value leads to better & more stable transcriptions

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
@mergify mergify Bot added the multi-modality Related to multi-modality (#4194) label Feb 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a global log_mel_max configuration for Voxtral's realtime audio processing. This change aims to improve transcription stability by normalizing log-mel spectrograms against a global average instead of a short-term frame-based maximum. The implementation correctly adds logic to use this global value in compute_whisper_melspec and updates the model configuration loading to support it. The tests have also been updated to reflect the improved and more consistent transcription outputs, removing previous workarounds. My main feedback is on improving the robustness of the type checking in voxtral.py.

Comment thread vllm/model_executor/models/voxtral.py Outdated
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
@mergify

mergify Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

Hi @patrickvonplaten, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?
mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

@mgoin mgoin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 2, 2026
@mgoin
mgoin merged commit 5019c59 into vllm-project:main Feb 2, 2026
50 checks passed
PiratePai pushed a commit to PiratePai/epd_shm that referenced this pull request Feb 3, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Pai <416932041@qq.com>
PiratePai pushed a commit to PiratePai/epd_shm that referenced this pull request Feb 3, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Pai <416932041@qq.com>
gameofdimension pushed a commit to gameofdimension/vllm that referenced this pull request Feb 5, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants