[cherry-pick][release/v0.32.0] Backport 14 milestone fixes and regressions (batch 4)

docker/Dockerfile.xpu

@@ -40,7 +40,10 @@ ENV VLLM_OMNI_TARGET_DEVICE=xpu
 # Keep the upstream oneCCL, torch and Triton packages paired. The final
 # vLLM XPU wheel requires triton==3.8.0+xpu; installing the old triton-xpu
 # overlay would replace that runtime with an incompatible version.
+# The kernels egg_info hook imports kernels_data. Install that runtime before
+# building Omni without isolation in the shared environment.
 RUN --mount=type=cache,target=/root/.cache/uv \
+    uv pip install "kernels-data>=0.14.0.dev1" && \
     uv pip install --no-build-isolation --excludes <(printf "torch\ntriton-xpu\n") \
     ".[dev]"
 

docs/design/feature/prefix_caching.md

@@ -290,7 +290,8 @@ Hit spans come from adapter-classified `STARTED` events for new requests:
   `EXTENDED`, so the manager does not mark a hit: those rows were delivered in
   earlier steps and re-emitting them would duplicate output. A
   `delivered_upto` span for this case is Phase 2.
-- Preemption + reschedule: vLLM resets `num_computed_tokens` to 0 on
+- Preemption + reschedule: for stages with `recompute_preemption="allow"`
+  (the default), vLLM resets `num_computed_tokens` to 0 on
   preemption and re-runs prefix matching on resume, so the resumed request
   can come back with a fresh hit. With the V1 model runner it arrives
   through `scheduled_cached_reqs` (id in `resumed_req_ids`, `new_block_ids`
@@ -303,6 +304,22 @@ Hit spans come from adapter-classified `STARTED` events for new requests:
   hidden states should be sized so preemption does not occur while prefix
   caching is on. Same as before this refactor; tracked for Phase 2.
 
+Stages whose decode state cannot be reconstructed from token IDs declare
+`recompute_preemption="fail"` in their AR pipeline topology. Capacity
+preemption then finishes the request with one terminal ERROR and releases its
+request/KV state instead of requeueing it, including when a KV-delivery
+connector asks to discard stale output. Explicit `reset_prefix_cache` retains
+its existing resume behavior, subject to `supports_running_prefix_cache_reset`;
+its reset scope, rather than the shared `drop_stale_output` flag, identifies
+that exception. Fail-mode preemption also notifies the auxiliary-output
+connector before removing the request, so the worker receives its finished
+notification and subsequent metadata assembly cannot reference a freed request.
+
+Fish Omni retains the default `allow` policy: its MRv2 model state saves the
+generated codebooks before a slot is reused and restores full frame rows on
+resume. Missing codebooks in the token stream alone do not imply that a stage
+cannot replay; model-state replay support must also be considered.
+
 The adapter is the sole owner of started-request observations. Its single
 `translate_step` entry point captures an immutable `PrefixCacheStep`.
 `EXTENDED` is represented by `step.extended_req_ids`, with token counts in

docs/serving/speech_api.md

@@ -978,7 +978,22 @@ See the [Gepard section of the online TTS hub](../user_guide/examples/online_ser
 
 | Model | Description |
 | ------- | ------------- |
-| `k2-fsa/OmniVoice` | Pure-diffusion TTS. Supports voice cloning via `ref_audio` (with optional `ref_text`); no built-in voice presets. |
+| `k2-fsa/OmniVoice` | OmniVoice generates mono speech at 24 kHz. It supports text-to-speech without a reference clip, voice cloning with `ref_audio` and optional `ref_text`, and voice design. It has no built-in voice presets. |
+
+For voice cloning, send `ref_audio`. If `ref_text` is missing or blank, the default deploy config leaves Whisper unloaded until the first such request, then loads `openai/whisper-large-v3-turbo` on the worker device and transcribes the clip. Set `additional_config.omnivoice_asr.load_asr_on_startup` to `true` to load it during worker startup. Set `additional_config.omnivoice_asr.asr_device` to `cuda:0` or `cpu` to override the worker device. Sending `ref_text` avoids ASR and uses the supplied transcript.
+
+OmniVoice prepares the reference clip before it builds the voice cloning input. It performs these steps:
+
+- It converts the clip to mono, resamples it to 24 kHz, and measures its original root mean square (RMS) level.
+- When the original RMS is greater than zero and less than `0.1`, it raises the waveform level to `0.1` and keeps the original RMS for output volume control.
+- When `ref_text` is missing, it trims a clip longer than 20 seconds at a detected silence. Supplied text skips this special long-audio trim.
+- It removes middle silence and trims silence at both edges whether the transcript is automatic or supplied by the caller.
+- It aligns the sample count down to the audio tokenizer hop size.
+- When `ref_text` is missing, it sends the prepared waveform to Whisper.
+- It adds a final period for English text or a Chinese full stop for Chinese text when the reference text has no ending punctuation.
+- It sends the same prepared waveform to the audio tokenizer.
+
+OmniVoice processes voice-cloning audio after decoding. It removes middle gaps of at least 500 milliseconds. It trims edge silence while keeping 100 milliseconds at each edge. It scales cloned output using the original reference RMS when that value is below `0.1`, and leaves the level unchanged when the value is at least `0.1`. It fades the first and last 100 milliseconds, then adds 100 milliseconds of silence at each edge. Text-only output returns the decoder tensor without this voice-cloning postprocessing.
 
 OmniVoice uses packed variable-length attention for batched generator execution. The attention operator accepts FP16 and BF16 inputs, so its query,
 key, and value tensors are evaluated in BF16 even when the stage is configured with `dtype: float32`; the attention output is converted back to the model's

docs/user_guide/examples/offline_inference/text_to_speech.md

@@ -36,7 +36,7 @@ python examples/offline_inference/text_to_speech/<model>/end2end.py \
     --ref-text  "Transcript of the reference audio."
 ```
 
-`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below.
+For most models, `--ref-audio` and `--ref-text` are optional for text-only synthesis but must be provided together for voice cloning. OmniVoice also accepts `--ref-audio` without `--ref-text` and transcribes the reference lazily with Whisper. Qwen3-TTS, Voxtral TTS, and CosyVoice3 accept additional model-specific flags documented in their sections below.
 
 ---
 
@@ -252,15 +252,15 @@ Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24
 
 ## OmniVoice
 
-Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
+OmniVoice creates speech in more than 600 languages. It supports text-to-speech without a reference clip, voice cloning, and voice design.
 
 ### Prerequisites
 
 ```bash
 huggingface-cli download k2-fsa/OmniVoice
 ```
 
-Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
+Voice cloning requires `transformers>=5.3.0`. Automatic voice and voice design use `transformers>=4.57.0`.
 
 ### Quick start (auto voice)
 
@@ -272,6 +272,8 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
 
 ### Voice cloning
 
+With an explicit transcript, voice cloning does not load Whisper:
+
 ```bash
 python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
     --model k2-fsa/OmniVoice \
@@ -280,6 +282,30 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
     --ref-text  "This is the reference transcription."
 ```
 
+If `--ref-text` is omitted, OmniVoice loads `openai/whisper-large-v3-turbo` on the worker device and transcribes the reference audio after it prepares the waveform. The first request downloads and loads Whisper, so it takes longer and uses more device memory. Set `additional_config.omnivoice_asr.load_asr_on_startup` to `true` to load Whisper during worker startup, or set `additional_config.omnivoice_asr.asr_device` to use another GPU or the CPU.
+
+```bash
+python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
+    --model k2-fsa/OmniVoice \
+    --text "hello" \
+    --ref-audio trump_ref.wav
+```
+
+### Reference audio processing
+
+OmniVoice prepares the reference clip before it transcribes the clip or builds the voice cloning input. It performs these steps:
+
+- It converts the clip to mono and resamples it to 24 kHz.
+- It measures the original root mean square (RMS) level. When the RMS is greater than zero and below `0.1`, it raises the waveform level to `0.1` and keeps the original RMS for output volume control.
+- When `--ref-text` is omitted, it trims a clip longer than 20 seconds at a detected silence.
+- It removes middle silence and trims silence at both edges whether the transcript is automatic or supplied by the caller.
+- It aligns the sample count down to the audio tokenizer hop size.
+- When `--ref-text` is omitted, it sends the prepared waveform to Whisper.
+- It adds a final period for English text or a Chinese full stop for Chinese text when the reference text has no ending punctuation.
+- It sends the same prepared waveform to the audio tokenizer and uses the resulting reference tokens to estimate the target duration.
+
+OmniVoice processes voice cloning audio after decoding. It removes middle gaps of at least 500 milliseconds and trims edge silence while keeping 100 milliseconds at each edge. It uses the original reference RMS when the RMS is below `0.1`, and it leaves the level unchanged when the RMS is at least `0.1`. It fades the first and last 100 milliseconds, then adds 100 milliseconds of silence at both edges. Text only output returns the decoder tensor without this processing.
+
 ### Voice design
 
 ```bash
@@ -302,6 +328,11 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
 
 - Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
 - Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
+- By default, the first request with reference audio and no `--ref-text` downloads
+  and loads `openai/whisper-large-v3-turbo`, so it has extra latency and GPU
+  memory use. Set `additional_config.omnivoice_asr.load_asr_on_startup` to `true`
+  to move that load to worker startup.
+- Generated audio remains mono at 24 kHz.
 
 ---
 

docs/user_guide/examples/online_serving/text_to_speech.md

@@ -448,15 +448,15 @@ bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh
 
 ## OmniVoice
 
-Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline.
+OmniVoice creates speech in more than 600 languages. It supports text-to-speech without a reference clip, voice cloning, and voice design.
 
 ### Prerequisites
 
 ```bash
 huggingface-cli download k2-fsa/OmniVoice
 ```
 
-Voice cloning (offline) needs `transformers>=5.3.0`; auto voice works with `transformers>=4.57.0`.
+Voice cloning needs `transformers>=5.3.0`. When `ref_text` is omitted, OmniVoice loads `openai/whisper-large-v3-turbo` on the worker device to transcribe the reference audio. Set `additional_config.omnivoice_asr.load_asr_on_startup` to `true` to load Whisper during worker startup, or set `additional_config.omnivoice_asr.asr_device` to use another GPU or the CPU.
 
 ### Launch
 
@@ -472,13 +472,35 @@ vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
 cd examples/online_serving/text_to_speech/omnivoice
 python speech_client.py --text "Hello, how are you?"
 python speech_client.py --text "Bonjour, comment allez-vous?" --language French
+python speech_client.py --text "hello" --ref-audio /path/to/ref_audio.wav
 ```
 
-The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`.
+The client supports `--api-base`, `--model`, `--text`, `--response-format`,
+`--language`, `--voice`, `--ref-audio`, `--ref-text`, `--instructions`,
+`--seed`, and `--output`.
 
 ### Notes
 
-- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#omnivoice).
+- The explicit `--ref-text` path is faster because it does not load Whisper.
+- By default, the first request with reference audio and no reference text loads
+  `openai/whisper-large-v3-turbo` on the worker device, which adds latency and
+  device memory use. Set `additional_config.omnivoice_asr.load_asr_on_startup` to
+  `true` to move that load to worker startup. Generated audio is mono at 24 kHz.
+
+### Reference audio and output processing
+
+When a request includes `ref_audio`, OmniVoice prepares the reference clip before it transcribes the clip or builds the voice cloning input. It performs these steps:
+
+- It converts the clip to mono and resamples it to 24 kHz.
+- It measures the original root mean square (RMS) level. When the RMS is greater than zero and below `0.1`, it raises the waveform level to `0.1` and keeps the original RMS for output volume control.
+- When `ref_text` is omitted, it trims a clip longer than 20 seconds at a detected silence.
+- It removes middle silence and trims silence at both edges whether the transcript is automatic or supplied by the caller.
+- It aligns the sample count down to the audio tokenizer hop size.
+- When `ref_text` is omitted, it sends the prepared waveform to Whisper.
+- It adds a final period for English text or a Chinese full stop for Chinese text when the reference text has no ending punctuation.
+- It sends the same prepared waveform to the audio tokenizer.
+
+After decoding, OmniVoice processes voice cloning output by removing middle gaps of at least 500 milliseconds and trimming edge silence while keeping 100 milliseconds at each edge. It uses the original reference RMS when the RMS is below `0.1`, and it leaves the level unchanged when the RMS is at least `0.1`. It fades the first and last 100 milliseconds, then adds 100 milliseconds of silence at both edges. Text only output returns the decoder tensor without this processing.
 
 ---
 

examples/offline_inference/text_to_speech/README.md

@@ -44,7 +44,7 @@ python examples/offline_inference/text_to_speech/<model>/end2end.py \
     --ref-text  "Transcript of the reference audio."
 ```
 
-`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below. Qwen3-TTS in particular uses its own argparse surface (`--query-type`, `--audio-path`, etc.) and does not follow the common shape; see its section.
+For most models, `--ref-audio` and `--ref-text` are optional for text-only synthesis but must be provided together for voice cloning. OmniVoice also accepts `--ref-audio` without `--ref-text` and transcribes the reference lazily with Whisper. Qwen3-TTS, Voxtral TTS, and CosyVoice3 accept additional model-specific flags documented in their sections below. Qwen3-TTS uses its own argparse surface with options such as `--query-type` and `--audio-path`, so it does not follow the common command shape.
 
 ---
 
@@ -407,15 +407,15 @@ python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
 
 ## OmniVoice
 
-Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
+OmniVoice creates speech in more than 600 languages. It supports text-to-speech without a reference clip, voice cloning, and voice design.
 
 ### Prerequisites
 
 ```bash
 huggingface-cli download k2-fsa/OmniVoice
 ```
 
-Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
+Voice cloning requires `transformers>=5.3.0`. Automatic voice and voice design use `transformers>=4.57.0`.
 
 ### Quick start (auto voice)
 
@@ -427,6 +427,8 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
 
 ### Voice cloning
 
+With an explicit transcript, voice cloning does not load Whisper:
+
 ```bash
 python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
     --model k2-fsa/OmniVoice \
@@ -435,6 +437,30 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
     --ref-text  "This is the reference transcription."
 ```
 
+Omit `--ref-text` to use automatic transcription. The default OmniVoice deploy config loads Whisper on the worker device when the first request needs it. Set `load_asr_on_startup: true` in `additional_config.omnivoice_asr` to load it during worker startup. Set `asr_device: "cuda:0"` or `asr_device: "cpu"` in the same block to override the worker device:
+
+```bash
+python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
+    --model k2-fsa/OmniVoice \
+    --text "hello" \
+    --ref-audio trump_ref.wav
+```
+
+### Reference audio processing
+
+OmniVoice prepares the reference clip before it transcribes the clip or builds the voice cloning input. It performs these steps:
+
+- It converts the clip to mono and resamples it to 24 kHz.
+- It measures the original root mean square (RMS) level. When the RMS is greater than zero and below `0.1`, it raises the waveform level to `0.1` and keeps the original RMS for output volume control.
+- When `--ref-text` is omitted, it trims a clip longer than 20 seconds at a detected silence.
+- It removes middle silence and trims silence at both edges whether the transcript is automatic or supplied by the caller.
+- It aligns the sample count down to the audio tokenizer hop size.
+- When `--ref-text` is omitted, it sends the prepared waveform to Whisper.
+- It adds a final period for English text or a Chinese full stop for Chinese text when the reference text has no ending punctuation.
+- It sends the same prepared waveform to the audio tokenizer.
+
+After decoding, OmniVoice applies silence removal, RMS scaling, fades, and padding only to voice-cloning output. Text-only output returns the decoder tensor unchanged. Output audio is mono at 24 kHz.
+
 ### Voice design
 
 ```bash
@@ -466,6 +492,11 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
 
 - Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
 - Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
+- The default deploy config uses `openai/whisper-large-v3-turbo` for lazy ASR on
+  the worker device. Set `load_asr_on_startup: true` in `additional_config.omnivoice_asr`
+  to load it during worker startup. Set `asr_device` in the same block to use a
+  different GPU or the CPU. Supplying `--ref-text` avoids ASR.
+- Generated audio remains mono at 24 kHz.
 
 ---
 

examples/offline_inference/text_to_speech/omnivoice/end2end.py

@@ -1,10 +1,11 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """End-to-end OmniVoice TTS inference via vLLM-Omni.
 
 Supports:
 - Auto voice mode: text only → generated speech
 - Voice cloning mode: text + reference audio → cloned voice speech
+- Automatic reference transcription: reference audio without reference text
 
 Usage:
     # Auto voice
@@ -13,6 +14,11 @@
     # Voice cloning
     python end2end.py --model k2-fsa/OmniVoice --text "Hello" \
         --ref-audio ref.wav --ref-text "reference transcription"
+
+    # Automatic reference transcription (lazy by default; set
+    # load_asr_on_startup: true to preload Whisper at worker startup)
+    python end2end.py --model k2-fsa/OmniVoice --text "hello" \
+        --ref-audio trump_ref.wav
 """
 
 import argparse
@@ -113,7 +119,8 @@ def run_e2e():
 
         audio_signal, sr = load_audio(args.ref_audio, sr=None)
         multi_modal_data["audio"] = (audio_signal.astype(np.float32), sr)
-        mm_processor_kwargs["ref_text"] = args.ref_text or ""
+        if args.ref_text is not None:
+            mm_processor_kwargs["ref_text"] = args.ref_text
         mm_processor_kwargs["sample_rate"] = sr
 
     if args.lang: