Support Voxtral audio generation in the Transformers backend

[harshaljanjani]

What does this PR do?

→ Blocked by the companion Transformers PR making VoxtralProcessor.__call__ audio-capable.
→ Enables Voxtral audio generation through the Transformers backend (--model-impl transformers) currently XFAILed. Stops forwarding tokenizer to MistralCommonBackend (rejects the kwarg), and forces the HF config and weights (not Mistral's params.json / consolidated) when the backend is explicitly requested.
→ Verified no regressions on the vLLM-side with tests/models/multimodal/processing/test_transformers_audio.py (Voxtral un-XFAILed) and tests/models/multimodal/generation/test_transformers_audio.py (Voxtral added, matches HF via check_logprobs_close); full audio suite green.
→ I've tried to explain why each non-trivial change was made in a one-line comment given much of this involved unwrapping failure modes layer by layer and uncovering deeper issues in the call chain :)

cc: @hmellor

Before (vLLM Voxtral Generation)

After (vLLM Voxtral Generation)

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[harshaljanjani]

Keeping this as a draft for now until the Transformers-side PR is merged; looking forward to the reviews!

[hmellor]

Does it? Transformers supports mistral-common

[harshaljanjani]

It does and the tokenizer loads fine either way, but this due to the model config. As in if we keep config_format=auto it resolves to mistral for Voxtral (ships params.json), and adapt_config_dict gives the backend a generic PreTrainedConfig. AutoModel.from_config resolves by config class not model type, so even with model_type=voxtral and architectures=['VoxtralForConditionalGeneration'] present (left the instrumentation in the screenshot, in_model_mapping=False) we hit the ValueError. Definitely could've been clearer with the comments though, made edits!

[hmellor]

Ok so would I be right in saying that:

• This forces vLLM to use the HF config
• Then HF may load the Mistral config inside a HF class

[harshaljanjani]

So yes this forces vLLM to use the HF config, and tmk no there isn't a scenario where it can fall back to the Mistral here, the reason being that Voxtral ships both an HF config.json and a Mistral params.json and the PR just swaps which one we read (auto reads params.json and gives PreTrainedConfig; HF reads config.json and gives VoxtralConfig). The forced path only ever fetches config.json and Transformers only reads params.json in its offline convert scripts, not at load time.

[hmellor]

For the config sure, but for the tokenizers/processors Transformers will wrap the classes in mistral-common if it is installed.

[harshaljanjani]

In that case yup it's AutoProcessor gives VoxtralProcessor with a MistralCommonBackend tokenizer and we leave that untouched here, we only pin the config/weights to HF and the processor stays on mistral-common. If the comment reads more broadly than intended (the config) I'm happy to reword it :)