[akshatvishu]

Purpose

Resolves #5659
OmniVoice voice cloning failed when a request included reference audio without reference text. The pipeline encoded the reference audio but it did not create the transcript that OmniVoice uses with those audio tokens. The model therefore received audio tokens without matching text and the generated audio could be silence or white noise or very low volume.

The official OmniVoice implementation transcribes missing reference text with Whisper. It also prepares the reference waveform before transcription and audio tokenization. The preparation raises quiet audio to the target level, removes silence, aligns the sample count to the tokenizer hop size and keeps the original volume for the final output. The official implementation also adds missing punctuation and applies silence removal, volume adjustment, fades and padding to generated audio.

This branch adds the missing ASR path and the shared audio processing code. It uses the same prepared waveform for transcription and voice tokenization. It also keeps blank reference text absent from offline and online requests so the pipeline can detect that it must transcribe the reference audio.

Test plan

ROCM Version : 7.2.53211-c2d9476115
vLLM Version : 0.26.0
vLLM-Omni Version : 0.1.dev2415+gb9a51592f.rocm (git sha: b9a5159, date: ocm)

Run the CPU tests for reference audio preparation and automatic transcription. The tests use synthetic audio and mocked model components, so they do not need model weights or a GPU.

pytest -q \
    tests/model_executor/models/omnivoice/test_asr.py \
    tests/model_executor/models/omnivoice/test_audio_processing.py

pytest \
    --run-level advanced_model \
    -q tests/e2e/offline_inference/test_omnivoice_expansion.py \
    -k ref_audio_without_ref_text

pytest \
    --run-level advanced_model \
    -q tests/e2e/online_serving/test_omnivoice_expansion.py \
    -k ref_audio_only

pytest \
    --run-level advanced_model \
    -q tests/e2e/online_serving/test_omnivoice_expansion.py \
    -k "not ref_audio_only"

1 failed, 3 passed, 1 deselected, 15 warnings in 1876.42s (0:31:16)

updated fail test pass:

pytest \
    --run-level advanced_model \
    -q -s \
    'tests/e2e/online_serving/test_omnivoice_expansion.py::TestOmniVoiceSeed::test_speech_auto_voice_seed_determinist[omni_server0]'

1 passed, 15 warnings in 650.18s (0:10:50)

pytest \
    --run-level advanced_model \
    -q tests/e2e/offline_inference/test_omnivoice_expansion.py \
    -k "not ref_audio_without_ref_text" \

1 passed, 1 deselected, 16 warnings in 23.77s

Test result

The CPU tests passed with 41 passed tests and 15 warnings.

The offline regression log records 1 passed test, 1 deselected test, and 16 warnings in 275.04 seconds.

The online regression log records 1 passed test, 4 deselected tests, and 15 warnings in 497.61 seconds.

The test_speech_auto_voice_seed_deterministic test was updated because this branch added pydub dependency to prepare and write the audio. Repeated requests can produce the same audio samples while the complete WAV files differ slightly because of sample rounding and file encoding. The old byte by byte check therefore failed even though generation was deterministic. The test now checks the sample rate and length and compares the decoded audio samples with a small tolerance.

Performance

script:https://gist.github.com/akshatvishu/e6f7d14105652216668bdb8b0582f743
The script runs zero-shot voice cloning using tests/assets/qwen3_tts/clone_2.wav as the reference audio, with one warm-up run and three measured runs. The main uses : --ref-text "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

Benchmarked current changes (without --ref-text as we use whisper) v/s main (in main we need to provide -ref-text otherwise voice cloning returns silent .wav outputs)

Benchmark dimension	Supplied ref_text	Missing ref_text, cold	Missing ref_text, warm and uncached	Repeated same audio
ASR behavior	ASR skipped	Model load and transcription	Transcription with the loaded model	Transcript cache hit
End to end latency	0.475 s mean over 3 requests	4.559 s	0.671 s mean over 3 different clips	0.474 s
Individual warm times	Not applicable	Not applicable	0.641, 0.652, and 0.722 s	Not applicable
ASR GPU allocation	None	+1.507 GiB	Already resident	Already resident
ASR parameter placement	Not loaded	cuda:0	cuda:0	cuda:0

The warm benchmark used three different reference clips, so every measured request ran Whisper and created a new cache entry. No measured request used the transcript cache. The cold request was excluded from the 0.671s mean.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[linyueqian]

Thanks for chasing this down. The duration hardcode is a real bug and I agree the cloning path was broken independently of ASR, so the _estimate_target_len part is worth landing on its own.

On the Whisper fallback, some context on why it was not in the original port: in vllm-omni every cloning TTS takes ref_text as an explicit request field (CosyVoice3, Fish Speech, GLM-TTS, higgs v2, IndexTTS2), and upstream's ASR fallback lives in the OmniVoice.from_pretrained() convenience wrapper rather than in the model. The port implemented the engine and left the transcript to the caller. That is a scope call, not a correctness claim, so I am not against adding it, but it needs to be opt-in and it needs to not sit on the request path unconditionally. Details inline.

[Med] On the perf table: the 16.7s is worth a second look before we accept 33x as the cost. Steady-state whisper-large-v3-turbo on a few-second clip should be a few hundred ms, and self._asr_pipeline is cached on the instance, so either the warm-up run is not actually warming the ASR pipeline or something is falling back badly on ROCm. Could you re-measure on CUDA and report warm-only numbers? The 0.0 MB peak VRAM delta also looks off given a second model is loaded on self.device.

[linyueqian]

[High] Make ASR opt-in and configurable rather than always-on with a hardcoded model. Upstream landed the same knobs after the initial release: k2-fsa/OmniVoice#221 (asr_model_name) and k2-fsa/OmniVoice#224 (asr_device). As written, every OmniVoice deployment silently grows a second model on the same device as the diffusion worker plus an outbound HF download that only happens when the first ref-audio-without-ref-text request arrives, so an offline or air-gapped deployment fails at request time instead of at startup, and the engine's memory-utilization budget no longer reflects what is actually resident.

Suggest a deploy-yaml block (enable flag, model name, device) defaulting to off, and resolving it at init so a misconfiguration surfaces during startup.

[akshatvishu]

Added and validated load_asr, asr_model_name and asr_device. The default load_asr: false skips startup loading, while a request with reference audio and no ref_text still loads Whisper when needed. I kept this behavior because upstream uses load_asr to choose eager or lazy loading and not to disable automatic transcription but in-case we wanna move away from this then I can remove it!

[linyueqian]

[High] The transcript is only reused when voice_name is set, since _speaker_cache is not consulted otherwise. The repro in #5659 posts inline base64 ref_audio with no uploaded voice, so it takes this branch and re-transcribes on every single request, which is exactly the throughput cliff in your perf table.

Two options that avoid it: transcribe once at voice-upload time (/v1/audio/voices already carries a ref_text field, so the auto-filled transcript can be stored alongside the speaker entry), and/or key a small transcript cache on a hash of the prepared waveform so the inline path also gets reuse.

[akshatvishu]

Added a bounded cache with eight entries for inline reference audio. The key includes the prepared waveform, its original RMS and whether the request used automatic or supplied text. Each entry stores the transcript and audio tokens, so repeated requests skip both Whisper and audio tokenization.

[linyueqian]

[Med] This adds a repo-wide dependency (plus audioop-lts on 3.13) for one model's audio postprocessing. Can the silence trim, RMS match, fade and padding be done with numpy and torchaudio, which are already hard deps? Worth noting openai-whisper is already in this file, so ASR itself is not the new-dependency concern here, pydub is.

[akshatvishu]

Removed both proposed dependencies. The port now uses NumPy and torchaudio as suggested and follows the same silence window, range merge, edge trim, RMS, fade and padding rules as upstream.

[linyueqian]

[Med] This runs unconditionally, so it also changes output on the auto-voice path where there is no reference audio at all. That is a behavior change for users who are not doing voice cloning, and it is why test_speech_auto_voice_seed_deterministic had to be relaxed from a byte comparison to a tolerance comparison.

Preference would be to scope the postprocess to the cloning path so the existing auto-voice output stays bit-identical and the determinism test can stay strict. If it genuinely needs to apply everywhere to match upstream, please call that out explicitly in the PR description as an intentional output change.

[akshatvishu]

I changed the logic so the new processing runs only for voice-cloning requests and not for text-only generation. Text-only generation now returns the decoder tensor before that processing runs and I added a CPU test that fails if the text-only path reaches the voice-cloning logic.

I also restored the exact WAV byte assertion as requested. On the available ROCm MI300X GPU, that assertion is flaky and fails on both the unchanged main branch (the original test) and this branch.

[linyueqian]

[Nit] Good catch. The old estimate_duration(text, "Nice to meet you.", 25) meant the cloning path never used the real reference length or transcript, so target duration was wrong for every voice-clone request regardless of whether ref_text was supplied. This part is a correctness fix and stands on its own, so if the ASR discussion above drags on it would be reasonable to split this hunk into its own PR and land it first.

[akshatvishu]

Thanks for the quick review @linyueqian. You were rightl; he original numbers came from an older branch where Whisper was still on CPU and the 0.0 MB metric did not measure ASR memory.

After fixing device placement, I repeated the test on the available AMD mi300x single GPU. PyTorch reports the device as cuda:0, and the log confirms that all Whisper parameters are on that device.

I removed the old 33.7x latency and 97% throughput claims. I also removed the generator latency claim because generator-only time was not measured separately. The output duration and RMS rows were removed because the reference clips and conditioning text differed and RMS alone does not measure output quality.

The observed difference between the separate explicit text and warm automatic transcription benchmarks is about 0.197s but I would not present that number as isolated Whisper latency because the two benchmarks used different reference clips.

[linyueqian]

Thanks, this addressed everything and the perf correction was the right call. For the record on the numbers: withdrawing the 33.7x latency and -97% throughput rows rather than defending them, and dropping the output-duration and RMS comparisons because the two runs used different reference clips, is exactly the right handling. A ~0.197s warm delta that you decline to present as isolated ASR latency is more useful than a confident number that is not comparable.

The other four are resolved as far as I can tell from reading: pydub and audioop-lts are out of requirements/common.txt with the silence detection reimplemented on numpy and torchaudio, the postprocess is scoped to the cloning path so text-only output stays as it was, and the inline reference cache is content-addressed on a SHA-256 of the prepared waveform with an 8-entry LRU, which closes the re-transcribe-every-request path from #5659. One follow-up on the config block inline.

[linyueqian]

[Nit] The name reads like a master switch but the semantics are startup-eager versus lazy: with load_asr: false a request carrying ref_audio and no ref_text still pulls openai/whisper-large-v3-turbo onto the worker device on first use, so there is no configuration that declines ASR outright.

I think the default behavior is defensible, since the only requests it fires on are the ones that produce broken audio today, so I am not asking to change it. But load_asr is easy to read as "ASR off" by an operator sizing GPU memory, and they would then be surprised by a lazy load. Either rename to something like preload_asr / load_asr_on_startup, or keep the name and add a third state where ASR is genuinely disabled and the request fails with a clear message telling the caller to supply ref_text. The docs you added describe the lazy behavior correctly, so this is only about the key name.

[akshatvishu]

I agree, changed the variable name from load_asr to load_asr_on_startup + add a brief comment to describe the behavior!

[Gaohan123]

[P2] RMS gain can exceed normalized waveform range

A quiet clip with an isolated peak can be amplified far beyond [-1, 1] here; a reproduced RMS=0.0119 input produced a peak of 8.4018 before both Whisper and HiggsAudio tokenization. Upstream's int16 AudioSegment conversion clips at this boundary. Clamp after the gain and add a peaky-quiet-waveform regression test.

[akshatvishu]

fixed in c4766e4 !

[linyueqian]

Re-reviewed the delta at c4766e4. Since the round 3 approval at 768a8f9 the only substantive change is the reference-audio clamp, and it is correct.

The gain-up branch scales a quiet reference by 0.1 / original_rms to reach the target RMS, which is safe for a waveform whose energy is spread out but not for a peaky one. Your new test is the exact worst case: a single unit sample in 7056 zeros has an RMS of about 0.0119, so the gain is roughly 8.4 and the peak lands near 8.4 before this change, well outside the range the downstream stages assume. Clamping after the multiply bounds that at 1.0. Worth noting explicitly that clipping necessarily gives up the RMS target for such inputs, since you cannot both normalize energy and stay in range here; bounding the peak is the right trade for reference conditioning, and the alternative of peak-normalizing instead would change the loudness of every ordinary reference rather than just the pathological ones.

The remaining changes in the delta are the two SPDX headers moved to the vLLM-Omni project and one annotation on the fake ASR's inputs list.

Approving again at this head. DCO, both wheel builds, docs, and pre-commit are green, and GitHub reports the branch mergeable.

Validation: static review only. I did not execute the fork head or run the audio tests, so the clamp's numerical behaviour rests on your added test rather than my own run.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 137b33d4b7a1 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[linyueqian]

This is approved and the only thing between it and merging is a rebase.

The branch went conflicting against main, and the conflict took CI with it: all four lanes are now red at c4766e42, general build 14652, AMD 11405, Intel 7807, and NPU 6402. That is not four separate problems, it is one merge state that no lane can build from, so there is nothing here worth triaging until the branch merges cleanly again.

Please rebase onto current main. That will clear the conflict and give all four lanes a tree they can actually build, and I will re-check the results at the new head. If anything real surfaces once they run properly I will raise it then, but I am not expecting it: the approval covered the ASR fallback and the audio preprocessing alignment, and nothing in those areas is implicated by a merge conflict.

Worth knowing while you are in there: the ready label is already on this PR, and pushing to a branch that already carries it does not re-fire per-PR CI on its own. The rebase itself will produce a fresh run, but if you ever need to re-run without a new commit, the label has to be cycled rather than simply left in place.

[linyueqian]

Correction to my last note: the merge conflict does not explain all four CI failures. General 14652 and AMD 11405 both ran tests and caught final_output.request_output in the new regression test at tests/e2e/offline_inference/test_omnivoice_expansion.py:105. That accessor no longer exists, so the test cannot reach its audio assertions. Please update it and the nested-output fallback when rebasing.

Intel 7807 and the NPU Platforms job fail separately on the old create_error_response import. AMD also has two diffusion attention tolerance failures. A rebase is still needed, but we need to check the test fix and fresh CI before merging.

[akshatvishu]

@linyueqian Thanks for the correction. I’ve rebased and pushed the accessor fix, including removing the redundant nested-output fallback. Local targeted tests pass! Can you please re-run the CI test again (The Intel/NPU import errors and AMD attention tolerance failures remain separate concerns at previous CI run)

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@akshatvishu this pull request has had no human commit, comment or review since 2026-09-16. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 14 days

@akshatvishu this pull request has had no human commit, comment or review since 2026-09-16. Please consider marking this PR as draft until work can resume. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[vllm-omni-review-bot]

Omni ReviewBot review

6 actionable finding(s).

CI at 85bcc610876c (2026-10-10T00:57:34.533562+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

Full review analysis

Scan:

Category	Result
Tests / verification	2 finding(s) below
Security	no finding reported
Docs / comments	3 finding(s) below
Behavior / compatibility	1 finding(s) below
Correctness	no finding reported

Validated:

• [resolved] Gaohan123 RMS-gain clip: prepare_reference_audio clips to [-1,1] after gain (audio.py:326) and test_reference_audio_normalization_clips_peaky_quiet_waveform covers it. Residual: RMS is still taken before silence trim, so a quiet-but-peaky clip can keep a low original_rms and then scale cloned output down.
• [resolved] load_asr renamed to load_asr_on_startup with comment that false is lazy, not off (omnivoice.yaml:16-18). Residual: no knob disables ASR; a ref_audio-only request still loads Whisper.
• [claim-verified] #5659 path: missing/blank ref_text with ref_audio now transcribes via Whisper then encodes the same prepared waveform (pipeline_omnivoice.py:456-517, serving_speech.py:2392, tts_adapters/omnivoice.py:38, end2end.py:121).
• [claim-verified] additional_config.omnivoice_asr is not a StageDeployConfig field; unknown stage YAML keys merge into engine_extras (stage_config.py:700) then engine_args.update (1074) and extract_diffusion_stage_config_kwargs keeps OmniDiffusionConfig.additional_config. Defaults still ASR-on-demand if the block is dropped.
• [claim-verified] first-element test indexes are after fake ASR append; numpy/torch waveform branches both run via parametrize
• [claim-verified] same prepared waveform for ASR and tokenizer: test_pipeline_uses_one_prepared_waveform_for_asr_and_tokenizer.

6 actionable finding(s).

Verdict: REQUEST CHANGES

Findings

• [P1] Default OmniVoice lazy-loads Whisper inside an already-packed request batch — vllm_omni/diffusion/models/omnivoice/pipeline_omnivoice.py

Evidence for Default OmniVoice lazy-loads Whisper inside an already-packed request batch

Default OmniVoice lazy-loads Whisper inside an already-packed request batch. _load_asr_pipeline uses vanilla hf_pipeline(...) plus asr_pipeline.model.to(target_device) (pipeline_omnivoice.py:276-287), invoked from _prepare_request_input via _resolve_ref_text (510-515) while forward() walks req.requests (811-814). With deploy defaults load_asr_on_startup: false, max_num_seqs: 8, request_batch_max_wait_ms: 10, the first missing-ref_text clone stalls every other seq in that batch, including text-only siblings, for Hub download and the device move. Those Whisper tensors are allocated after load_model's CuMem weights pool has exited, and self._asr_pipeline is never cleared, so DiffusionWorker.sleep() / empty_cache() cannot reclaim them; later text-only traffic on the same worker keeps paying that VRAM. Load or fail Whisper before the batch is assembled, and drop _asr_pipeline on worker sleep/abort.

Evidence: Trigger: default omnivoice.yaml load_asr_on_startup: false with max_num_seqs: 8 and request_batch_max_wait_ms: 10, plus OmniVoicePipeline.supports_request_batch = True, so a first clone that sends ref_audio without ref_text can share a DiffusionRequestBatch with text-only siblings. Adverse effect: that clone's Whisper Hub download and .to(device) run inside the packed forward() prepare loop, blocking those siblings, then _asr_pipeline keeps the weights on the worker device outside the CuMem pool so sleep/empty_cache cannot reclaim them for later text-only traffic. pipeline_omnivoice.py:276-287 asr_pipeline = hf_pipeline(\n "automatic-speech-recognition",\n model=self._asr_model_name,\n dtype=asr_dtype,\n device=asr_device,\n ) … asr_pipeline.model = asr_pipeline.model.to(target_device). pipeline_omnivoice.py:510-515 if ref_audio_tokens is None and needs_asr: / ref_text = self._resolve_ref_text(. pipeline_omnivoice.py:811-814 for i, request in enumerate(req.requests): / prepared = self._prepare_request_input(prompt, extra). pipeline_omnivoice.py:293 self._asr_pipeline = asr_pipeline. Unchanged by this diff, present in the PR-time tree: diffusion_model_runner.py:342-349 with get_memory_context(): / self.pipeline = model_loader.load_model( — CuMem weights pool wraps only load_model, not later forward(). Unchanged by this diff, present in the PR-time tree: diffusion_worker.py:844-847 offload_tags = ("weights",) if level == 1 else tuple() / allocator.sleep(offload_tags=offload_tags) / current_omni_platform.empty_cache() — sleep never drops _asr_pipeline.

- **[P2] Correct the PR Test result paragraph: it says test_speech_auto_voice_seed_deter…** — `tests/model_executor/models/omnivoice/test_omnivoice_expansion.py:191`

Evidence for Correct the PR Test result paragraph: it says test_speech_auto_voice_seed_deter…

Correct the PR Test result paragraph: it says test_speech_auto_voice_seed_deterministic moved off byte equality because this branch added pydub. At this head that is false — tests/e2e/online_serving/test_omnivoice_expansion.py:191 still asserts r1.audio_bytes == r2.audio_bytes, and audio.py imports numpy/torch/torchaudio only. Clone-path tests are the ones using assert_allclose. Rewrite that paragraph to match this head.

Evidence: Unchanged GitHub metadata for PR 5784 (not in the diff): Test result says The test_speech_auto_voice_seed_deterministic test was updated because this branch added pydub dependency and The test now checks the sample rate and length and compares the decoded audio samples with a small tolerance. Current head contradicts that. tests/e2e/online_serving/test_omnivoice_expansion.py:191 assert r1.audio_bytes == r2.audio_bytes — auto-voice seed still compares raw WAV bytes. vllm_omni/diffusion/models/omnivoice/audio.py:10-12 import numpy as np / import torch / import torchaudio — no pydub import (only a pydub-compatibility docstring at audio.py:115). Clone path tests/e2e/online_serving/test_omnivoice_expansion.py:241 np.testing.assert_allclose(repeated_audio, audio, rtol=1e-5, atol=1e-4). Trigger: a reviewer of the Test result believes this branch added pydub and weakened the auto-voice seed check. Adverse effect: that evidence contract does not match head 85bcc61.

- **[P2] This PR makes online OmniVoice cloning a supported path (`OmniVoiceAdapter` / `…** — `recipes/k2-fsa/OmniVoice.md`

Evidence for This PR makes online OmniVoice cloning a supported path (`OmniVoiceAdapter` / `…

This PR makes online OmniVoice cloning a supported path (OmniVoiceAdapter / _create_diffusion_speech omit blank ref_text; pipeline lazy-loads Whisper) and updates the TTS hubs and examples, but unchanged recipes/k2-fsa/OmniVoice.md still advertises the old contract at L10 (online serving for automatic voice) and L89 (Online serving currently supports automatic voice:). Sync the Mode line and Online serving section, including a --ref-audio-only example and the lazy Whisper / load_asr_on_startup note from docs/serving/speech_api.md.

Evidence: Trigger: a user following the in-repo OmniVoice recipe for online serving reads that only automatic voice is supported. Adverse effect / unmet requirement: this PR’s user-visible online contract now includes voice cloning (ref_audio with optional ref_text; omitted/blank ref_text transcribes via Whisper) and the hubs/examples were updated, but the recipe was not. Unchanged by this diff, present in the PR-time tree: recipes/k2-fsa/OmniVoice.md:10 - Mode: Offline inference for all modes and online serving for automatic voice. Unchanged by this diff, present in the PR-time tree: recipes/k2-fsa/OmniVoice.md:89 Online serving currently supports automatic voice:. New contract in this diff: docs/serving/speech_api.md:836 | \k2-fsa/OmniVoice` | OmniVoice generates mono speech at 24 kHz. It supports text-to-speech without a reference clip, voice cloning with `ref_audio` and optional `ref_text`, and voice design. It has no built-in voice presets. |. docs/serving/speech_api.md:838 For voice cloning, send `ref_audio`. If `ref_text` is missing or blank, the default deploy config leaves Whisper unloaded until the first such request, then loads `openai/whisper-large-v3-turbo` ... Set `additional_config.omnivoice_asr.load_asr_on_startup` to `true` .... Serving path in this diff: vllm_omni/entrypoints/openai/tts_adapters/omnivoice.py:38 if request.ref_text and request.ref_text.strip():. Parallel online hub example in this diff: docs/user_guide/examples/online_serving/text_to_speech.md:420 python speech_client.py --text "hello" --ref-audio /path/to/ref_audio.wav`.

Suggestion: Online serving supports automatic voice, voice cloning (ref_audio with optional ref_text; omitted ref_text transcribes with Whisper), and voice design:

- **[P2] This PR documents cloned output as removing middle gaps of at least 500 ms (`do…** — `docs/serving/speech_api.md`

Evidence for This PR documents cloned output as removing middle gaps of at least 500 ms (`do…

This PR documents cloned output as removing middle gaps of at least 500 ms (docs/serving/speech_api.md:851; same claim at docs/user_guide/examples/offline_inference/text_to_speech.md:262 and docs/user_guide/examples/online_serving/text_to_speech.md:448). postprocess_generated_audio calls remove_silence(..., middle_silence_ms=500), and remove_silence sets keep_silence_ms=middle_silence_ms. With keep=500, _split_on_silence expands each nonsilent range by 500 ms on both sides, so 500–1000 ms gaps are fully preserved; only gaps longer than 1 s shrink, by (gap − 1000) ms. That keep math is already tested with keep=200 in test_remove_silence_preserves_middle_gap_boundary (200 ms gap kept, 500 ms gap loses 100 ms). Fix the three sentences, or change keep_silence_ms if 500 ms pauses are supposed to disappear.

Evidence: Trigger: voice-cloning output is passed through postprocess_generated_audio, which this PR documents as removing middle gaps of at least 500 ms. Unmet requirement: those new user-visible sentences do not match the keep/detect contract. docs/serving/speech_api.md:851 It removes middle gaps of at least 500 milliseconds. (same wording at docs/user_guide/examples/offline_inference/text_to_speech.md:262 and docs/user_guide/examples/online_serving/text_to_speech.md:448). vllm_omni/diffusion/models/omnivoice/audio.py:381 middle_silence_ms=500,. vllm_omni/diffusion/models/omnivoice/audio.py:242 keep_silence_ms=middle_silence_ms,. vllm_omni/diffusion/models/omnivoice/audio.py:194 output_ranges = [[start - keep_silence_ms, end + keep_silence_ms] for start, end in ranges]. tests/model_executor/models/omnivoice/test_audio_processing.py:162 (199, 0), (200, 0), (500, 100) with middle_silence_ms=200 — detection window is not the amount removed; keep=window means a gap must exceed 2×window to shrink. Adverse effect: a 600 ms cloned pause is fully kept, contradicting the docs; only pauses >1 s are shortened, leaving 1 s.

Suggestion: OmniVoice processes voice-cloning audio after decoding. It shortens middle silences longer than 1 second (500 ms detected, 500 ms kept on each side). It trims edge silence while keeping 100 milliseconds at each edge.

- **[P2] This PR rewrites the OmniVoice online section to say cloning is supported and a…** — `docs/user_guide/examples/online_serving/text_to_speech.md`

Evidence for This PR rewrites the OmniVoice online section to say cloning is supported and a…

This PR rewrites the OmniVoice online section to say cloning is supported and adds python speech_client.py --text "hello" --ref-audio /path/to/ref_audio.wav with no --ref-text, but the Supported Models hub table at the top of the same file still lists | OmniVoice | \k2-fsa/OmniVoice` | (offline only) |. That is the only remaining (offline only)cell in the tree;examples/online_serving/text_to_speech/README.mdalready shows✓, and docs/serving/speech_api.mddocuments optionalref_text`. Update the hub row so the first table users check matches the new online-cloning contract.

Evidence: Trigger: a reader of the online TTS hub Supported Models table (first table in the file this PR edited). Adverse effect: OmniVoice voice cloning is still labeled offline-only, contradicting this PR's new online contract (optional ref_text, Whisper ASR). Unchanged by this diff, present in the PR-time tree: docs/user_guide/examples/online_serving/text_to_speech.md:25 | OmniVoice | k2-fsa/OmniVoice | (offline only) | — | — | — |. Same file, changed by this PR: docs/user_guide/examples/online_serving/text_to_speech.md:396 It supports text-to-speech without a reference clip, voice cloning, and voice design.; docs/user_guide/examples/online_serving/text_to_speech.md:420 python speech_client.py --text "hello" --ref-audio /path/to/ref_audio.wav. Sibling table on this head: examples/online_serving/text_to_speech/README.md:27 | OmniVoice | k2-fsa/OmniVoice | ✓ | — | — | — |. API docs changed by this PR: docs/serving/speech_api.md:836 | k2-fsa/OmniVoice| OmniVoice generates mono speech at 24 kHz. It supports text-to-speech without a reference clip, voice cloning withref_audioand optionalref_text, and voice design. It has no built-in voice presets. |.

Suggestion: | OmniVoice | k2-fsa/OmniVoice | ✓ (ref_audio, optional ref_text) | — | — | — |

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[vllm-omni-review-bot]

[P2] The new clone-without-ref-text path Hub-loads openai/whisper-large-v3-turbo (`_…

Evidence and suggested fix

The new clone-without-ref-text path Hub-loads openai/whisper-large-v3-turbo (_ASR_MODEL_NAME / yaml asr_model_name) with no local_files_only. Both OmniVoice Prerequisites still only run huggingface-cli download k2-fsa/OmniVoice (online:401, offline:215), then immediately give a --ref-audio command with no --ref-text. After that prefetch, an HF_HUB_OFFLINE/airgapped first ASR request fails because Whisper was never fetched. Add huggingface-cli download openai/whisper-large-v3-turbo beside the OmniVoice fetch in both Prerequisites blocks (same gap at examples/online_serving/text_to_speech/README.md:685, examples/offline_inference/text_to_speech/README.md:369, recipes/k2-fsa/OmniVoice.md:25).

Evidence: Trigger: follow the rewritten Prerequisites, then the new copy-paste clone-without-ref-text example, with Whisper absent from the HF cache (airgapped / HF_HUB_OFFLINE). Adverse effect: _load_asr_pipeline Hub-loads the default ASR id and the first request fails because Prerequisites never fetched it. docs/user_guide/examples/online_serving/text_to_speech.md:401 huggingface-cli download k2-fsa/OmniVoice. Same file:420 python speech_client.py --text "hello" --ref-audio /path/to/ref_audio.wav. docs/user_guide/examples/offline_inference/text_to_speech.md:215 huggingface-cli download k2-fsa/OmniVoice. Same file:240 The first request downloads and loads Whisper, so it takes longer and uses more device memory. vllm_omni/diffusion/models/omnivoice/pipeline_omnivoice.py:66 _ASR_MODEL_NAME = "openai/whisper-large-v3-turbo". Same file:276-280 asr_pipeline = hf_pipeline(\n "automatic-speech-recognition",\n model=self._asr_model_name,\n dtype=asr_dtype,\n device=asr_device,\n ) — no local_files_only. vllm_omni/deploy/omnivoice.yaml:19 asr_model_name: "openai/whisper-large-v3-turbo".

Suggestion: huggingface-cli download k2-fsa/OmniVoice
huggingface-cli download openai/whisper-large-v3-turbo

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] Default OmniVoice lazy-loads Whisper inside an already-packed request batch — vllm_omni/diffusion/models/omnivoice/pipeline_omnivoice.py

See the review for details. If you are the PR author and disagree, react 👎 here; the maintainer will see your disagreement.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 30 days

@akshatvishu this pull request has had no human commit, comment or review since 2026-09-16. Please consider closing this PR if the work is no longer planned. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[linyueqian]

Re-approving at 137b33d. The branch was rewritten after my approval at c4766e4, so I went through the author changes since then. The main one is the merge with OmniVoice request batching and step execution: reference_rms now lives on each prepared request, both the step path and the batched path post-process per request, text-only requests skip post-processing, cache hits keep the RMS, and an ASR failure becomes a per-request error in a batch. The other changes since then are small fixes (ASR pipeline cached only after the device move succeeds, stereo mixed down before the empty check). The request_output accessor failure from the first CI run is fixed.

I also ran it end to end on one L20X against the merge base (bcd8eb6), default omnivoice.yaml except whisper-large-v3 instead of turbo. Voice clone with ref_audio only: base drops the first three words (Whisper coverage 0.85), this head speaks the full text (1.00), speaker similarity 0.933 vs 0.937. With ref_text supplied, output changes as expected from the new preprocessing, with coverage 1.00 on both and similarity within 0.001. No tracebacks in either server log.

The two notes inline are not blocking. Buildkite has not run on this head yet, so I will retrigger it; the merge waits for that.

[linyueqian]

[suggestion] With the lazy default, the first ref_audio-only request loads Whisper inside the batch loop, and every request that arrives during the load waits for it. On a fresh server (one L20X, whisper-large-v3 because turbo was not cached there), text-only requests sent 0.5 to 2 s after that clone request took 2.7 to 4.2 s instead of 0.43 s, and GPU memory went from 6127 to 8861 MiB with a 10127 MiB peak during the load. That memory also sits outside the weights pool that sleep mode releases. I would make load_asr_on_startup: true the default here, or say this trade-off in the docs next to the option. Not blocking.

[linyueqian]

[suggestion] "removes middle gaps of at least 500 milliseconds" reads as if a 600 ms gap is removed. In audio.py the threshold is 500 ms and 500 ms is kept on each side, so only gaps longer than 1 s shrink, and they shrink to 1 s. The same sentence appears in the offline and online text-to-speech example docs. It would also help to say there that the ASR model is downloaded on first use, so offline (HF_HUB_OFFLINE=1) setups need to pre-download it like k2-fsa/OmniVoice.