[WIP]: Add Ming-omni-tts dense 0.5B pipeline#2906
[WIP]: Add Ming-omni-tts dense 0.5B pipeline#2906akshatvishu wants to merge 11 commits intovllm-project:mainfrom
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
I guess everyone is suffering under the new limits (╥_╥) |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
This PR is marked as [WIP] and is substantial (~10,500 lines / 47 files).
Could you please run the L3 tests locally and paste the results here? This helps validate the integration on your end before we proceed with full review.
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
please make changes accordingly after #2383 merged. For the model usage, I suggest to write a model recipe under vllm_omni/recipes using the template. It seems there are some duplicate/dead codes as well, can you try to compress it first?
|
I also recommend you to use the add-tts-models skill |
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Thanks for the thorough test matrix and the warm-cache RTF numbers, those are the right kind of evidence for a model-add PR. At 10.5k additions the PR is hard to review carefully. I think it can stay as one PR if we condense it by reusing modules that already live in the repo. Inline comments below on the specific files, ordered roughly by expected line savings.
Not blocking merge, flagging for the author and maintainers.
| @@ -0,0 +1,868 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MAJOR] This file is 868 lines and mixes a Qwen2 AR backbone, the Aggregator, FlowLoss head, stop head, and latent patch emission. Two asks:
- The Qwen2 backbone should import from upstream vLLM (
from vllm.model_executor.models.qwen2 import Qwen2Model) rather than being reimplemented here. qwen3_omni Thinker and qwen3_tts Talker both follow that pattern. Estimated saving: 300 to 400 lines. - After that, please split the remainder into
backbone.py,aggregator.py,flowloss_head.py,patch_emission.py. Our coding-style guideline targets 200 to 400 lines per file with 800 as a hard cap.
| @@ -0,0 +1,207 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MAJOR] cosyvoice3/code2wav_core/cfm.py (325 lines) already implements Conditional Flow Matching. This PR adds fm/cfm.py (207) plus fm/modules.py (147), for roughly 350 lines of duplicated logic.
Suggestion: promote the cosyvoice3 CFM plus a DiT base to vllm_omni/model_executor/modules/flow_matching/, have Ming import it, and keep only fm/dit.py (Ming-specific conditioning) and fm/flowloss.py here.
This is a cross-model refactor, fine to land as a prerequisite PR owned by a maintainer or cc @yuanheng-zhao rather than blocking Ming on it. Worth an issue link from the PR body at minimum.
| @@ -0,0 +1,291 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MAJOR] The chunk-accumulate-and-final-flush logic here is the same pattern used by stage_input_processors/qwen3_tts.py for talker2code2wav_async_chunk. Extract a helper to stage_input_processors/_chunk.py (or _chunk_transfer.py) and have both models call it. Likely 80 to 120 lines saved across the two files plus easier future maintenance when the SharedMemoryConnector contract changes.
| @@ -0,0 +1,188 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MINOR] Pure math. qwen3_tts/tokenizer_25hz/ and voxtral_tts/voxtral_tts_audio_tokenizer.py also ship an iSTFT. Recommend opening a follow-up issue to migrate all three to a shared vllm_omni/model_executor/modules/audio/stft.py. Not a blocker on this PR, but please file the issue so this does not go cold.
| @@ -0,0 +1,66 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MINOR] Small file, quick check: does cosyvoice3 already load the CampPlus 192-d speaker embedder somewhere under cosyvoice3/utils.py or cosyvoice3/tokenizer.py? If yes, share the loader.
| @@ -0,0 +1,581 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MINOR] 581 lines reads as two concerns: the top-level two-stage dispatcher and the weight loader. Split the loader into loader.py and keep the dispatcher plus registry wiring in __init__.py + ming_tts.py at about 200 lines each.
| @@ -0,0 +1,364 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MINOR] 364 lines of constants, runtime keys, token IDs, stop-head defaults, and sample-rate validation. Split into constants.py and validation.py.
| @@ -0,0 +1,86 @@ | |||
| async_chunk: true | |||
There was a problem hiding this comment.
[MAJOR] #2383 (config refactor 2/N) replaces stage_configs/*.yaml with a two-layer PipelineConfig (Python) plus deploy/<model>.yaml split. Follow-up 2c will remove --stage-configs-path and the legacy ModelPipeline path entirely.
If this PR lands ahead of 2c, please open a migration task on your side so Ming ships a pipeline.py plus deploy/ming_tts.yaml immediately after. Preferable: rebase onto #2383 once it merges and ship directly on the new schema to avoid a rewrite. cc @lishunyang12 for coordination.
| @@ -0,0 +1,654 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
[MINOR] 654 lines is almost entirely an 11-case dispatch ladder. Please move the case definitions to cases.yaml (prompt, ref_audio, speaker, expected sample rate, etc.) and keep a roughly 100-line driver that parametrizes off that file. CI can then iterate the same cases. Expected saving: about 500 lines here alone.
| @@ -0,0 +1,217 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
[NIT] 217 lines of curl examples read as documentation rather than a runnable script. Prefer fenced code blocks inside README.md, keep run_curl.sh as a short "here are three sanity checks" helper.
|
@akshatvishu It seems there're a lot added files that could re-use modules from the talker of I'll update #2890 later today and try to merge it ASAP and then you might want to rebase |
|
@yuanheng-zhao Sure, I will wait for #2890 to get merge and will then start working on the suggestion left by @linyueqian as it seems like I can borrow a lot from |
|
Hey @akshatvishu , the git rebase --onto main the-talker-branch your-current-branchNote to fetch and have latest main and my branch on your local |
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
akshatvishu
force-pushed
the
feat/ming-omni-tts-dense
branch
from
d949ec7 to
9add4ef
Compare
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
…s signature Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
…tecture Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
…to-detection fails Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
4 participants
Purpose
Add Ming-omni-tts dense 0.5B support to vLLM-Omni via a two-stage AR+Flow → Audio VAE pipeline.
Original repo : https://github.com/inclusionAI/Ming-omni-tts
Resolves:
#1461
Changes:
Model files (
vllm_omni/model_executor/models/ming_tts/)ming_tts.py— top-level two-stage dispatcher and weight-loading entry pointming_tts_llm.py— Stage-0 Qwen2 AR backbone with inline Aggregator, FlowLoss, stop head, and latent patch emissionming_tts_audio_vae.py— Stage-1 Audio VAE decoder producing 44.1 kHz mono waveform outputconfig_ming_tts.py— Ming dense constants, runtime keys, latent sizes, token IDs, stop-head defaults, and sample-rate validationconfiguration_ming_dense.py— Hugging Face config adapter forinclusionAI/Ming-omni-tts-0.5Bprompt_builder.py— prompt construction for speech, music, instructions, TTA, prompt waveform, and speaker embeddingsingress.py— first-stage prompt ingestion for the disaggregated pipelinespeaker_extractor.py— CampPlus 192-d speaker embedding extraction for reference audiofm/— Flow Matching modules used by Stage-0 latent generationaudio_tokenizer/— Ming Audio VAE tokenizer and decoder support modulesRegistry
MingTTSForConditionalGeneration,MingLLMModel, andMingAudioVAEModelinvllm_omni/model_executor/models/registry.pyStage config & input processors
vllm_omni/model_executor/stage_configs/ming_tts.yaml— sequential two-stage AR+Flow → Audio VAE pipelinevllm_omni/model_executor/stage_configs/ming_tts_async_chunk.yaml— async chunk pipeline with SharedMemoryConnector,latent_chunk_size: 25, andmax_num_seqs: 1vllm_omni/model_executor/stage_input_processors/ming_tts.py— Stage-0 → Stage-1 latent patch transfer forllm2audio_vaeandllm2audio_vae_async_chunk, including final partial chunk flushOffline examples
examples/offline_inference/ming_tts/end2end.py— end-to-end Omni example covering 11 cookbook cases:style,ip,bgm,tta,emotion,basic,dialect,zero_shot,podcast,speech_bgm,speech_soundexamples/offline_inference/ming_tts/README.md— offline launch notes for sequential and async chunk runsOnline serving
vllm_omni/entrypoints/openai/serving_speech.py— Ming prompt builder for OpenAI-compatible/v1/audio/speech, with structured instructions,voice→ IP,language→ dialect, reference audio, 192-d speaker embeddings, podcast multi-speaker conditioning, and streaming PCM/WAV outputexamples/online_serving/ming_tts/run_server.sh— async chunk server launch scriptexamples/online_serving/ming_tts/openai_speech_client.py— API client covering Ming controls and streaming outputexamples/online_serving/ming_tts/run_curl.sh— curl examples for/v1/audio/speechexamples/online_serving/ming_tts/README.mdanddocs/user_guide/examples/online_serving/ming_tts.md— online serving documentationArchitecture:
Known limitations / follow-ups:
/v1/audio/speechdoes not yet exposeprompt_mode=music/ttaor FlowLoss controls (cfg,sigma,temperature); online BGM and TTA require a future prompt-mode API extension.max_num_seqs: 1; multi-request batching is not yet validated.latent_chunk_size: 5improves online TTFP significantly but diverges onpodcastin the offline async matrix; repo YAML stays on the validatedlatent_chunk_size: 25default until that is resolved.Test Plan
Validation was performed on an NVIDIA L4 GPU (Colab).
Offline sequential — full 11-case cookbook matrix :
Offline async_chunk — full 11-case cookbook matrix:
python examples/offline_inference/ming_tts/end2end.py \ --case <case> \ --streaming \ --stage-configs-path vllm_omni/model_executor/stage_configs/ming_tts_async_chunk.yaml \ --enforce-eagerOnline serving —
/v1/audio/speechasync_chunk checks:Test Result
Offline correctness — sequential vs. async_chunk (
latent_chunk_size: 25):All 11 cases produced identical frame counts and Stage-1 total patch counts between sequential and default async_chunk, confirming correct Stage-0 → Stage-1 handoff and final partial chunk flush.
styleipbgmttaemotionbasicdialectzero_shotpodcastspeech_bgmspeech_soundUpstream FlashAttention comparison (cold, single-request, L4):
Upstream: torch 2.6.0+cu124, FlashAttention 2.7.4.post1. vLLM-Omni VAE stage runs through SDPA, not upstream FlashAttention. Integration comparison, not kernel parity benchmark.
styleipbgmemotionbasicdialectzero_shotpodcastspeech_bgmspeech_soundvLLM-Omni matches or beats upstream RTF on
bgm; async25 is near-parity onstyle,zero_shot, andpodcast. Cold single-request numbers include engine startup and first-request lazy setup costs.Warm-cache RTF vs upstream (L4, post-warmup, 1 warmup + 1 measured request):
Warm-cache removes first-request lazy setup. Fairer per-request comparison against upstream.
styleipbgmemotionbasicdialectzero_shotpodcastspeech_bgmspeech_soundWarm vLLM-Omni sequential beats upstream FlashAttention RTF across all 10 measured cases. Async25 further reduces RTF for longer/reference-conditioned cases and the zero-ref
style/bgmruns.Warm-cache offline benchmark (L4, 1 warmup + 1 measured request):
styleipbgmzero_shotpodcastttabasicAsync chunk benefits longer/reference-conditioned cases; overhead roughly cancels the overlap benefit for short speech cases.
Online serving benchmark (10 prompts, concurrency 1, eager, L4):
sequential_eagerasync_chunk_eager(chunk=25)async_chunk_bench(chunk=5)latent_chunk_size: 5reduces mean TTFP by ~73% and E2E by ~11% vs. sequential, but remains experimental pending podcast offline finalization.Online
/v1/audio/speechvalidation (async_chunk, all speech-mode cases):All cases returned valid WAV at 44.1 kHz. Streaming PCM returned progressive chunks. Reference audio, speaker embedding, and podcast multi-reference checks passed.
styleipbasicemotiondialectzero_shotpodcastspeech_bgmspeech_soundstreamingBEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)