diff --git a/docs/fern/pages/recipes/cli-templates/vllm.mdx b/docs/fern/pages/recipes/cli-templates/vllm.mdx index 003b322194c6..1fa09a95add8 100644 --- a/docs/fern/pages/recipes/cli-templates/vllm.mdx +++ b/docs/fern/pages/recipes/cli-templates/vllm.mdx @@ -48,16 +48,38 @@ The default model is `Qwen/Qwen3-0.6B`. -**Experimental.** Transcribe English audio over `/v1/realtime` using vLLM 0.30.0's native -Qwen3-ASR support. The launch script requires an explicit `--hf-overrides` argument: +**Experimental.** Transcribe audio over `/v1/realtime` using vLLM's native Qwen3-ASR +support. This example was validated with English audio. + +The default model is `Qwen/Qwen3-ASR-0.6B`. The launch script selects its +`Qwen3ASRRealtimeGeneration` architecture automatically. + +Follow the [vLLM installation instructions](../../developer-guide/knowledge-base/modular-components/backends/vllm/overview.md#installation) +to use Dynamo's compatible backend version. + +**Command-line flags:** `--model` sets the model identifier or path. `--hf-overrides` +selects the realtime architecture; it is optional for the default model and required +for other models. + +**Pass-through arguments:** Additional unrecognized arguments are forwarded to the backend worker command. + +Start the default model: + +```bash +./launch/agg_realtime_transcription.sh --max-model-len 4096 --max-num-seqs 2 +``` + +For another realtime-capable model, replace `MODEL_ID` and `REALTIME_ARCHITECTURE` +with its model identifier and realtime architecture: ```bash -./launch/agg_realtime_transcription.sh --model Qwen/Qwen3-ASR-0.6B \ - --hf-overrides '{"architectures":["Qwen3ASRRealtimeGeneration"]}' \ - --max-model-len 4096 --max-num-seqs 2 +./launch/agg_realtime_transcription.sh --model MODEL_ID \ + --hf-overrides '{"architectures":["REALTIME_ARCHITECTURE"]}' ``` -In another terminal, send a speech file longer than five seconds to check incremental delivery: +In another terminal, send a speech file longer than five seconds to check incremental +delivery from the default model. Replace `speech.wav` with your audio file and set +`--model` to the served model if you changed it: ```bash python launch/realtime_audio_client.py \ diff --git a/docs/fern/pages/use-cases/multimodal-serving/voice.md b/docs/fern/pages/use-cases/multimodal-serving/voice.md index dd699edff778..227aa4e1aa1d 100644 --- a/docs/fern/pages/use-cases/multimodal-serving/voice.md +++ b/docs/fern/pages/use-cases/multimodal-serving/voice.md @@ -27,7 +27,7 @@ for a cascaded pipeline, or a vLLM-Omni model for audio input and output. Follow the example's runtime-image requirements; it currently requires compatible source-built Dynamo images, not the published 1.5.0 images. - **Separate vLLM/vLLM-Omni speech workers:** Use the - [Voxtral realtime transcription launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_realtime_transcription.sh) + [Qwen3-ASR realtime transcription launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_realtime_transcription.sh) and [Qwen3-TTS launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_omni_audio.sh) to exercise each endpoint independently before connecting them to an LLM. - **Audio-in/audio-out with vLLM-Omni:** Run the diff --git a/examples/backends/vllm/launch/agg_realtime_transcription.sh b/examples/backends/vllm/launch/agg_realtime_transcription.sh index 060da9ce52b6..5091b10dbc6a 100755 --- a/examples/backends/vllm/launch/agg_realtime_transcription.sh +++ b/examples/backends/vllm/launch/agg_realtime_transcription.sh @@ -8,8 +8,10 @@ SCRIPT_DIR="$(dirname "$(readlink -f "$0")")" source "$SCRIPT_DIR/../../../common/gpu_utils.sh" source "$SCRIPT_DIR/../../../common/launch_utils.sh" -MODEL="mistralai/Voxtral-Mini-4B-Realtime-2602" +DEFAULT_MODEL="Qwen/Qwen3-ASR-0.6B" +MODEL="$DEFAULT_MODEL" HF_OVERRIDES="" +HF_OVERRIDES_SET=false EXTRA_ARGS=() while [[ $# -gt 0 ]]; do @@ -24,10 +26,12 @@ while [[ $# -gt 0 ]]; do exit 2 fi HF_OVERRIDES="$2" + HF_OVERRIDES_SET=true shift 2 ;; --hf-overrides=*) HF_OVERRIDES="${1#*=}" + HF_OVERRIDES_SET=true shift ;; *) @@ -37,9 +41,13 @@ while [[ $# -gt 0 ]]; do esac done +if [[ "$HF_OVERRIDES_SET" == false && "$MODEL" == "$DEFAULT_MODEL" ]]; then + HF_OVERRIDES='{"architectures":["Qwen3ASRRealtimeGeneration"]}' +fi + if [[ -z "$HF_OVERRIDES" ]]; then echo "Error: select the model's realtime architecture explicitly with --hf-overrides." >&2 - echo "Usage: $0 [--model MODEL] --hf-overrides JSON [vLLM args...]" >&2 + echo "Usage: $0 [--model MODEL] [--hf-overrides JSON] [vLLM args...]" >&2 exit 2 fi