[Voxtral] Fix speech transcription api

[patrickvonplaten]

Make sure speech transcription api works.

Allows to do standard vllm serve + speech-to-text transcription API

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request aims to fix the speech transcription API for Voxtral streaming models. It updates the speech-to-text configuration and adds the necessary methods to process transcription requests. However, I've found a critical issue in the implementation of get_generation_prompt where several key parameters (task_type, request_prompt, to_language) are ignored, leading to incorrect behavior for translation and prompted transcription. A code suggestion is provided to address the most critical parts of this issue.

[mergify]

Hi @patrickvonplaten, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @patrickvonplaten, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[NickLucche]

Thanks @patrickvonplaten !

[NickLucche]

qq: is offline the right setting here?

[patrickvonplaten]

Yeah it's still quite difficult to test actual "online" streaming given that this PR: #28973 is not yet merged - so for now I've only tested "offline" streaming which means that whole audio file is passed in one go.

Once #28973 is merged we however will need to slightly modify the methods here I think but maybe it's ok to do this after and for now just ensure "offline" streaming, a.k.a pass the whole audio file in one-go, works as expected?

[mergify]

Hi @patrickvonplaten, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint