Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 28 additions & 6 deletions docs/fern/pages/recipes/cli-templates/vllm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,38 @@ The default model is `Qwen/Qwen3-0.6B`.
</Accordion>
<Accordion title="Qwen3-ASR Realtime Transcription">

**Experimental.** Transcribe English audio over `/v1/realtime` using vLLM 0.30.0's native
Qwen3-ASR support. The launch script requires an explicit `--hf-overrides` argument:
**Experimental.** Transcribe audio over `/v1/realtime` using vLLM's native Qwen3-ASR
support. This example was validated with English audio.

The default model is `Qwen/Qwen3-ASR-0.6B`. The launch script selects its
`Qwen3ASRRealtimeGeneration` architecture automatically.

Follow the [vLLM installation instructions](../../developer-guide/knowledge-base/modular-components/backends/vllm/overview.md#installation)
to use Dynamo's compatible backend version.

**Command-line flags:** `--model` sets the model identifier or path. `--hf-overrides`
selects the realtime architecture; it is optional for the default model and required
for other models.

**Pass-through arguments:** Additional unrecognized arguments are forwarded to the backend worker command.

Start the default model:

```bash
./launch/agg_realtime_transcription.sh --max-model-len 4096 --max-num-seqs 2
```

For another realtime-capable model, replace `MODEL_ID` and `REALTIME_ARCHITECTURE`
with its model identifier and realtime architecture:

```bash
./launch/agg_realtime_transcription.sh --model Qwen/Qwen3-ASR-0.6B \
--hf-overrides '{"architectures":["Qwen3ASRRealtimeGeneration"]}' \
--max-model-len 4096 --max-num-seqs 2
./launch/agg_realtime_transcription.sh --model MODEL_ID \
--hf-overrides '{"architectures":["REALTIME_ARCHITECTURE"]}'
```

In another terminal, send a speech file longer than five seconds to check incremental delivery:
In another terminal, send a speech file longer than five seconds to check incremental
delivery from the default model. Replace `speech.wav` with your audio file and set
`--model` to the served model if you changed it:

```bash
python launch/realtime_audio_client.py \
Expand Down
2 changes: 1 addition & 1 deletion docs/fern/pages/use-cases/multimodal-serving/voice.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ for a cascaded pipeline, or a vLLM-Omni model for audio input and output.
Follow the example's runtime-image requirements; it currently requires
compatible source-built Dynamo images, not the published 1.5.0 images.
- **Separate vLLM/vLLM-Omni speech workers:** Use the
[Voxtral realtime transcription launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_realtime_transcription.sh)
[Qwen3-ASR realtime transcription launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_realtime_transcription.sh)
and [Qwen3-TTS launcher](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/launch/agg_omni_audio.sh)
to exercise each endpoint independently before connecting them to an LLM.
- **Audio-in/audio-out with vLLM-Omni:** Run the
Expand Down
12 changes: 10 additions & 2 deletions examples/backends/vllm/launch/agg_realtime_transcription.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
source "$SCRIPT_DIR/../../../common/gpu_utils.sh"
source "$SCRIPT_DIR/../../../common/launch_utils.sh"

MODEL="mistralai/Voxtral-Mini-4B-Realtime-2602"
DEFAULT_MODEL="Qwen/Qwen3-ASR-0.6B"
MODEL="$DEFAULT_MODEL"
HF_OVERRIDES=""
HF_OVERRIDES_SET=false

EXTRA_ARGS=()
while [[ $# -gt 0 ]]; do
Expand All @@ -24,10 +26,12 @@ while [[ $# -gt 0 ]]; do
exit 2
fi
HF_OVERRIDES="$2"
HF_OVERRIDES_SET=true
shift 2
;;
--hf-overrides=*)
HF_OVERRIDES="${1#*=}"
HF_OVERRIDES_SET=true
shift
;;
*)
Expand All @@ -37,9 +41,13 @@ while [[ $# -gt 0 ]]; do
esac
done

if [[ "$HF_OVERRIDES_SET" == false && "$MODEL" == "$DEFAULT_MODEL" ]]; then
HF_OVERRIDES='{"architectures":["Qwen3ASRRealtimeGeneration"]}'
fi

if [[ -z "$HF_OVERRIDES" ]]; then
echo "Error: select the model's realtime architecture explicitly with --hf-overrides." >&2
echo "Usage: $0 [--model MODEL] --hf-overrides JSON [vLLM args...]" >&2
echo "Usage: $0 [--model MODEL] [--hf-overrides JSON] [vLLM args...]" >&2
exit 2
fi

Expand Down
Loading