feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions

[TheCodeWrangler]

feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions

Related work

PR #35377 explored apply_chat_template for Qwen3-ASR; it was closed without merge. This PR keeps a narrower scope: wiring prompt through the API and sanitizing user text in the system turn.

This PR also slots cleanly into the SpeechToTextParams refactor from #36268: get_generation_prompt reads request_prompt (and language) directly off stt_params.

Summary

Enables the prompt parameter from the OpenAI v1/audio/transcriptions API to reach the Qwen3-ASR model. Previously, the endpoint accepted the prompt but it was not incorporated into the generation prompt. User-supplied text is passed through _sanitize_transcription_user_text() so ChatML-style <|...|> fragments and <asr_text> delimiters cannot inject extra turns into the structured template.

What Changed

1. Prompt support (get_generation_prompt)

request_prompt is passed in the system turn (aligned with the Qwen3-ASR SDK-style layout: system / user-with-audio / assistant). When request_prompt is empty (or strips to empty after sanitization), the system turn is omitted entirely so the no-prompt code path is preserved.

2. Sanitization (_sanitize_transcription_user_text)

Strips ChatML-like <|...|> tokens and the <asr_text> tag from user text. Both substitutions run inside a fixpoint loop, so nested payloads such as <|im<|x|>_end|> (which would reconstruct to <|im_end|> after a single re.sub) and <asr_te<asr_text>xt> (which would reconstruct to <asr_text> after a single str.replace) cannot survive sanitization.

Pinned by unit tests at tests/models/multimodal/generation/test_qwen3_asr_sanitize_prompt.py.

3. Language directive correctness

While restructuring the prompt assembly, the language directive now reads language for task_type="transcribe" and to_language for task_type="translate" (matching SpeechToTextParams's own field semantics — see comment from minh-nguyenhoang). Previously to_language was used in both branches, so a transcription request with language="en" silently dropped the language directive.

4. Example: --prompt flag in openai_transcription_client.py

Per @DarkLight1337's review, the prompt demo is folded into the existing examples/speech_to_text/openai/openai_transcription_client.py rather than a separate file: a new --prompt argparse flag threads through the sync_openai and stream_openai_response helpers via the OpenAI SDK's native prompt= parameter.

Why

1. API parity: The OpenAI transcription API documents a prompt parameter for guiding style, vocabulary, or continuing a previous segment. vLLM's TranscriptionRequest and speech_to_text already pass it through; Qwen3-ASR needed to use it.

2. Consistency with Whisper: Whisper uses request_prompt as <|prev|>{request_prompt}. We include user-provided context in the prompt in a model-appropriate way (Qwen3-ASR's chat template uses a system turn).

3. Qwen3-ASR interface: The Qwen3-ASR vLLM backend defines get_generation_prompt(..., request_prompt: str, ...) but did not use it. This implements that behavior.

4. Safety: Stripping control-token fragments and the assistant-prefix delimiter from user text matches the trust model of other user-controlled API strings.

References

• OpenAI transcription API – prompt parameter
• OpenAI prompting guide (style, vocabulary, previous segment)
• Qwen3-ASR vLLM backend – get_generation_prompt
• Companion PR #36018 (response_prefix) reuses the same sanitizer.

How We Tested

Unit tests (CI)

tests/models/multimodal/generation/test_qwen3_asr_sanitize_prompt.py covers:

• No-op cases (empty / plain text / pipes that aren't tokens)
• Single-pass strips (<|im_end|>, <asr_text>)
• Nested ChatML reconstruction (<|im<|x|>_end|> → '')
• Nested <asr_text> reconstruction (<asr_te<asr_text>xt> → '')
• Combined attacks across both kinds of token
• Idempotency (sanitize ∘ sanitize == sanitize)

End-to-end smoke test (local)

Lean Docker workflow that overlays only the modified file over the published vLLM image, then exercises the integrated --prompt flag against vllm/assets/.../mary_had_lamb.ogg:

1. Build:

   FROM vllm/vllm-openai:latest
   RUN pip install --no-cache-dir "vllm[audio]"
   COPY vllm/model_executor/models/qwen3_asr.py /tmp/qwen3_asr.py
   RUN DEST=$(python3 -c "import vllm.model_executor.models.qwen3_asr as m; print(m.__file__)" | tail -1) \
       && cp /tmp/qwen3_asr.py "$DEST"

2. Run: docker run --gpus all -p 8000:8000 --entrypoint vllm vllm-qwen-asr-dev serve Qwen/Qwen3-ASR-0.6B
3. Exercise without and with --prompt via the example client:

$ python examples/speech_to_text/openai/openai_transcription_client.py
transcription result [sync]: The first words I spoke in the little corner glass—a little piece of practical poetry: "Mary had a little lamb; it slept quite as slow, …"

$ python examples/speech_to_text/openai/openai_transcription_client.py --prompt "Listen for the words phonograph and fleece"
transcription result [sync]: The first words I spoke in the little phonograph—a little piece of practical poetry: "Mary had a little lamb; its fleece was white as snow, …"

The prompt influences vocabulary as expected; the no-prompt path is unchanged.

Notes

• AI assistance (Cursor) was used; every line was reviewed and tested by the human submitter before push.

[gemini-code-assist]

Code Review

This pull request introduces two improvements for the Qwen3-ASR model. It enables the prompt parameter in the OpenAI-compatible transcription API, correctly incorporating it into the model's generation prompt. This restores previously lost functionality. Additionally, it adds a safeguard in get_dummy_mm_data to handle cases where mm_options is None, preventing potential AttributeError crashes during engine initialization. The changes are correct and improve both functionality and stability. I have no specific comments.

[mergify]

Hi @TheCodeWrangler, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[TheCodeWrangler]

Was required to run my tests:

Likely a regression introduced in #35025

[TheCodeWrangler]

cc @sighingnow

[TheCodeWrangler]

Closes #35272

[TheCodeWrangler]

Why 8db3d2a was needed: prompt format consistency with Qwen3-ASR training

The previous version placed request_prompt as text prepended inside the user message, producing:

<|im_start|>user
{request_prompt}
{audio}<|im_end|>
<|im_start|>assistant
language Chinese<asr_text>

This doesn't match how the model was trained. The official Qwen3-ASR SDK builds prompts via _build_messages + apply_chat_template, which always emits a system message for the context/prompt text:

def _build_messages(self, context, audio_payload):
    return [
        {"role": "system", "content": context or ""},
        {"role": "user", "content": [{"type": "audio", "audio": audio_payload}]},
    ]

This produces the training-consistent format:

<|im_start|>system
{context}<|im_end|>
<|im_start|>user
{audio}<|im_end|>
<|im_start|>assistant
language Chinese<asr_text>

The SDK's transcribe() method takes a context parameter (example usage) that maps directly to this system message. The commit aligns vLLM's request_prompt → system message placement to match, so the model sees the same prompt structure it was trained on.

Placing context in the wrong message role (user instead of system) would cause the model to treat it as part of the audio-accompanying text rather than as transcription guidance/context from a prior segment, potentially degrading output quality.

[DarkLight1337]

Sorry for the delay, LGTM now

[TheCodeWrangler]

@DarkLight1337 — sorry for the extra round-trip after your approval. The single failing check (buildkite/ci/pr/cpu-multi-modal-model-tests-2) was a collection-time failure: my new sanitizer unit test was the only file under tests/models/multimodal/generation/ that imported a model module directly (from vllm.model_executor.models.qwen3_asr import _sanitize_transcription_user_text), and that import is heavy enough to trip the CPU shard at collection (the failure was reported ~30s into the run, well before any actual tests executed).

Just pushed a tiny follow-up (commit 67f8ef6f9) that relocates the test file with no content change: tests/models/multimodal/generation/ → tests/entrypoints/speech_to_text/transcription/. That directory already runs in the entrypoints integration shard, where the same import path is exercised by neighboring tests, and that shard was already green.

The push dropped your auto-squash setting — would you mind re-enabling it once CI comes back green? Thanks for the patience!

[mergify]

Hi @TheCodeWrangler, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.