[Model][TTS] Add dots.tts (rednote-hilab): continuous-AR 48kHz TTS by Moore-Z · Pull Request #4765 · vllm-project/vllm-omni

Moore-Z · 2026-06-27T08:28:19Z

Summary

Closes #4218.

Adds dots.tts (rednote-hilab) — continuous-AR 48 kHz TTS built on
Qwen2.5-1.5B + autoregressive flow-matching DiT head + 48 kHz AudioVAE.

Follows the VoxCPM2 pipeline pattern (vLLM-native AR backbone +
side-path acoustic/vocoder modules), adapted for:

Qwen2.5-1.5B-Base LM instead of MiniCPM4
Single DiT flow-matching head with 10-step Euler ODE (no FSQ / residual LM)
48 kHz AudioVAE with BigVGAN-style causal decoder
CAM++ x-vector speaker encoder (zero-shot path only this PR)

Implementation overview

New files

vllm_omni/model_executor/models/dots_tts/

dots_tts_talker.py — main pipeline (LLM forward → DiT N-step Euler → patch_encoder AR loopback → AudioVAE decode → eos_proj)
dots_tts_vocoder.py — vendored AudioVAE
dots_tts_dit.py — vendored DiT
dots_tts_patch_encoder.py — vendored semantic encoder
dots_tts_speaker_encoder.py — vendored CAM++ x-vector
dots_tts_prompt.py — prompt template
pipeline.py — single-stage AR pipeline config

Plus transformers_utils/configs/dots_tts.py, deploy/dots_tts.yaml,
examples/offline_inference/text_to_speech/dots_tts/end2end.py, and
3 registry entries.

Vendor strategy

Vendored from upstream @ `a393d2e` (release 2026-06-12). Single-file
per module, full public API preserved (ming_flash_omni convention).
Upstream not on PyPI — vendoring is the only path.

Diagnostics (env-gated, 0 production overhead)

`DOTS_TTS_BETA_TRACE=1` — per-patch rms/max trace
`DOTS_TTS_HIDDEN_DUMP=` — per-step LLM hidden `.pt` dump

Test plan

Functional smoke test

Requires `rednote-hilab/dots.tts-soar` cached or accessible via HF.

```bash
python examples/offline_inference/text_to_speech/dots_tts/end2end.py \
--text "Hello, this is a test of dots TTS running on vLLM Omni."
```

Expected: writes `output_audio/output.wav` (~3–5s, 48kHz mono),
audible English speech matching the input text.

Reference (upstream) for A/B comparison

```bash

upstream is not on PyPI; clone + install from source

git clone https://github.com/rednote-hilab/dots.tts && cd dots.tts
pip install -e .
dots.tts --model-name-or-path rednote-hilab/dots.tts-soar \
--text "Hello, this is a test of dots TTS running on vLLM Omni." \
--output /tmp/upstream.wav --seed 42
```

Checkpoints

`rednote-hilab/dots.tts-soar` — tested
`rednote-hilab/dots.tts-base` — same architecture, not validated this PR
`rednote-hilab/dots.tts-mf` — MeanFlow 2-4 step, deferred to future PR

Known follow-up (still WIP)

~16% speech-rate regression vs upstream + mild swallow on first word.
Hidden-state diff (`DOTS_TTS_HIDDEN_DUMP`) shows early decode cos ≈ 0.87
seed-aligned, drops past step 7 — likely bf16 cumulative drift / DiT
numerical implementation differences.
Voice cloning (`--prompt-audio`): CAM++ encoder integrated but not
wired into prompt builder.
CUDA Graph capture deferred to step 8.

chatgpt-codex-connector · 2026-06-27T08:28:24Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

…ncy) PR vllm-project#4765 ran the DiT+VAE side-path per-request, so concurrency regressed (N=16 -> 0.58x aggregate realtime; the soar-expensive DiT/VAE serialized). The DiT recomputes full attention each euler step (no incremental KV), so requests batch cleanly: - _run_dit_n_step_euler_batched: pad M requests' fm_sequences to a common max_hist, noise patch at a fixed tail, per-request attention masks (padding fully masked -> bit-exact independent attention), one DiT head call on [2M,L] per euler step. - _finish_decode_batched: one DiT-euler + one batched AudioVAE decode ([D,128,4]-> [D,1,7680]) + batched eos across all in-flight requests; per-request patch_encoder loopback kept (small, streaming-KV). - forward(): 1 request -> validated single path; >1 -> batched. Measured on RTX 5090 (DiT=10): N=1 0.55x, N=2 1.04x, N=4 1.77x, N=8 2.63x, N=16 3.37x streams-realtime (was 0.58x at N=16 unbatched -> ~5.8x). Concurrent audio verified valid (no NaN, no cross-contamination between streams). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

tzhouam

No modification on the modelrunner or worker. Looks fine to me.

linyueqian

Thanks for the contribution, the vendoring hygiene here is genuinely good (I spot-checked ~10 upstream line citations in the module headers against studio-dots-ai/dots.tts@a393d2e and they all match), and the pipeline wiring follows the voxcpm2 conventions closely.

I ran this end to end on a Hopper-class GPU (bf16, eager, vllm 0.24.0, PR merged into current main). Functional summary:

Offline example works: clean 48 kHz speech, whisper-medium transcribes correctly apart from the first word.
Your "mild swallow on first word" reproduces reliably: "Hello, this is a test..." transcribed as "So, this is a test...", and under concurrency "The weather today..." came back as "Weather today...". See the inline comment on the vocoder call for a likely mechanism.
Warm sequential RTF 1.12-1.20 (short/medium/long English texts). c=4 simultaneous submit: 16.0 s wall for 13.8 s of total audio, i.e. no throughput gain over c=1. Staggered arrivals: every request got its own text (no cross-request contamination, ASR-verified), but the long request slowed 35% with three short requests in flight.
Note the branch itself still imports supports_xccl (pre-#4709 base), so it only runs against vllm 0.24 after merging with current main; a rebase would be good before undrafting.

Inline comments carry the main findings (2 High, 3 Medium, 1 Low). Secondary items, all fine to defer but worth tracking:

DotsTTSConfig parses dit_config / audio_vae_config / speaker_encoder_config from the checkpoint but the talker builds everything from hardcoded _build_soar_*_config(), so pointing --model at dots.tts-base or dots.tts-mf silently builds soar shapes. Same concern as raised on #3838 (config.json should win, or the model should fail loudly on non-soar checkpoints).
The module docstring's step log contradicts the code in places ("Still stubbed: ... make_omni_output / compute_logits" while both are implemented), several comments cite sections of dots_tts_notes.md which this PR gitignores, and the DOTS_TTS_HIDDEN_DUMP block does per-step torch.save, which no other in-tree model ships. A cleanup sweep before undrafting would help reviewers a lot.
The five dots_tts_runtime_config knobs in the deploy yaml are not read anywhere yet; see the optimization notes below, two of them are exactly where the concurrency win is.
No tests, and dots_tts is missing from examples/offline_inference/text_to_speech/README.md and docs/models/supported_models.md; expected for WIP but blocking for merge-ready.
FYI the output processor logs Error concatenating tensor for key sr; keeping last tensor once per request; harmless but noisy.

Suggested optimization path

Since CUDA graphs and runtime tuning are on your roadmap, some techniques that map directly onto this architecture, mostly with in-repo precedent:

fp32 ODE integration by default (inline comment at the Euler update). Quality first: it is a few casts per step and the DiT matmuls stay bf16 under your existing autocast. Precedent: #4341.
Batch the DiT across requests. The per-request side path serializes 10 Euler steps per request per token step, which is why c=4 shows no scaling. Stacking per-request (z, cond) into one batched DiT forward per ODE step is the biggest concurrency lever; this is what voxcpm2's enable_batched_meanflow does, and your yaml already reserves the analogous knobs.
CUDA-graph the decode path. DiT step loop and patch encoder are static-shape at decode. voxcpm2's unified decode graph is the template (#4606, 2.5x at bs64 on the same class of hardware). One heads-up when you get there: #4606 also changed prepare_runner_inputs to receive tensor/ndarray slices instead of Python lists, so copy the post-#4606 pattern, not the one from this branch's merge-base.
Amortize the vocoder. vae_decode_every=N with the streaming decode state cuts AudioVAE invocations by N, and the streaming API doubles as the fix for the boundary artifacts (inline comment). Coalescing per-patch D2H copies (coalesce_audio_d2h) removes a sync per step.
Cut ODE steps. The mf checkpoint (MeanFlow, 2-4 steps) takes 2.5-5x off the dominant per-token cost once supported; even for soar a num_steps WER/MOS sweep may find 10 is conservative.
Seeded per-request noise (inline comment) is free and makes every A/B above deterministic. #4869 is the qwen3-tts analog.

Happy to re-run the same bench matrix (warm sequential RTF, c=4 batch, staggered arrivals with ASR verification) on a rebase.

… ODE integration, seeded FM noise, light package init Review fixes from PR vllm-project#4765 (linyueqian): * High-1: prefill branch now pushes a (req_id, None) placeholder onto _results_queue so compute_logits' positional pairing holds for mixed prefill+decode batches (voxcpm2 pattern). Verified: unit tests on the pairing contract + 3-request e2e, all pass. * High-2: Euler ODE integration state (z, times, CFG blend) now fp32; DiT matmuls stay bf16 under autocast, result cast back to sequence dtype (same fix as Ming CFM, PR vllm-project#4341). * Medium-1: FM noise drawn from a per-request per-step blake2b-seeded Generator instead of the global CUDA RNG (voxcpm2 parity). Verified: in-session replay is bit-identical. * Medium-3: drop eager talker import from package __init__ (registry imports submodules directly; voxcpm2 convention). Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

First contribution toward integrating rednote-hilab/dots.tts (Qwen2.5-1.5B base LM + DiTAR/MeanFlow flow-matching head + 48 kHz BigVGAN-style AudioVAE + CAM++ x-vector cloning). This commit covers steps 1-3b of the planned 8-step roll-out (see dots_tts_notes.md §5.1): Skeleton (steps 1-2): - Register DotsTTSConfig / DotsTTSForConditionalGeneration / dots_tts pipeline in the three vllm-omni registries (model / pipeline / HF config). - Single-stage AR pipeline.yaml + deploy yaml (V1 optimisations all off). - Talker shell wires vLLM-native Qwen2Model as the base LM; forward() returns scaffold hidden states; protocol methods stubbed for later steps. AudioVAE vendor (step 3b): - dots_tts_vocoder.py merges six upstream files (bigvgan.py, config.py, alias_free_{filter,resample,act}.py, backbone/layers.py:Conv1d/ConvTranspose1d) pinned at studio-dots-ai/dots.tts@a393d2e. Follows the ming_flash_omni audio_vae.py single-file pattern (continuous AudioVAE category, no _-prefix rename, no behaviour simplification). - AudioVAEConfig diverges from upstream: replaced pydantic ConfigBase with stdlib @DataClass after verifying soar/mf checkpoint configs use only the 17 declared fields (extra="allow" never exercised); added .get() helper so bigvgan.py call sites stay unmodified. - Talker __init__ instantiates a random-init AudioVAE via a hardcoded _build_soar_audio_vae_config() factory (step 3c will swap to hf_config parsing and load real vocoder.safetensors weights). Verified: standalone smoke (random latent -> 48 kHz wav) instantiates 180.9 M parameter AudioVAE and produces the expected (1, 1, frames*hop_size) output tensor with matching duration. Still TODO (steps 3c+): real vocoder weight loading, DiT flow-matching head, CAM++ speaker encoder, preprocess / make_omni_output / compute_logits, per-request state, CUDA Graph captures. Refs: GitHub issue vllm-project#4218 (initial announcement). Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

- step 3c: load_weights() routes vocoder.safetensors into AudioVAE via AutoWeightsLoader; __init__ calls remove_weight_norm() on decoder to match upstream's inference-fused export layout (951 keys 1:1). - step 4a: vendor backbone/{dit,layers}.py into dots_tts_dit.py (~470 lines, upstream pin a393d2e). Conv1d/ConvTranspose1d skipped (already in dots_tts_vocoder.py). - step 4b: talker __init__ constructs DiT(1024 -> 128, flow_matching) using a 13-field _DiTConfig dataclass that mirrors the soar config.json DiT block and exposes a to_dict() helper for upstream's interface. - step 4c: load_weights() extended to a two-branch routing (AudioVAE + DiT), single iterator pass; DiT uses exact prefix match on velocity_field_predictor.* (244 keys 1:1, no remap needed). Verified via strict=True load_state_dict and forward diff. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Lands the full dots.tts integration from skeleton through the first intelligible vllm-omni TTS wav. Spans step 5a (patch_encoder vendor + 5 projectors + Qwen2 LM config hoist) through step 7e-r4a (offline serving via Omni engine + first text-driven, recognizable wav). Talker (dots_tts_talker.py) * AR loopback: DiT N-step Euler + patch_encoder + AudioVAE decode + eos_proj real stop signal. fm_sequence / cfg_sequence per-request state with autocast-wrapped DiT solver (mirrors upstream runtime.py:280). * 5 vendored side modules wired with weight loading (AudioVAE + DiT + patch_encoder + 5 projectors + CAM++ speaker encoder, 2.4B params across 2615 tensors). * Capability flags: have_multimodal_outputs / has_preprocess / has_postprocess — without these gpu_model_runner silently bypasses preprocess and the AR side-path never runs. * Pin AudioVAE + speaker_encoder to fp32 (upstream runtime.py:81 only casts model.core to bf16; vocoder + xvector_extractor stay fp32 for conv1d / fbank precision). * _IOHelper: dtype-aware normalize/denormalize so bf16 latents from DiT survive the round trip into patch_encoder / VAE. Vendored leaves (commit a393d2e from rednote-hilab/dots.tts) * dots_tts_patch_encoder.py — VAESemanticEncoder + SemanticEncoderDecode State (AR loopback head). * dots_tts_speaker_encoder.py — CAM++ x-vector extractor (3D-Speaker), fbank + downsample stack folded into one leaf file. Config + serving * DotsTTSConfig: 16 Qwen2 LM fields hoisted to top level (vllm Qwen2Model reads them via get_text_config()). * dots_tts_prompt.py: build_dots_tts_prompt — template "[文本]{text}[文本对应语音]<|audio_gen_start|>" returning real Qwen2 token IDs (no placeholders; our preprocess embeds input_ids directly). * examples/offline_inference/text_to_speech/dots_tts/end2end.py: Omni() engine offline runner mirroring voxcpm2/end2end.py. Verified: 13.44s text-driven wav from "Hello, this is a test of dots TTS running on vLLM Omni." — main content intelligible at zero-shot bf16; RTF 8.7x in eager mode. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

… ODE integration, seeded FM noise, light package init Review fixes from PR vllm-project#4765 (linyueqian): * High-1: prefill branch now pushes a (req_id, None) placeholder onto _results_queue so compute_logits' positional pairing holds for mixed prefill+decode batches (voxcpm2 pattern). Verified: unit tests on the pairing contract + 3-request e2e, all pass. * High-2: Euler ODE integration state (z, times, CFG blend) now fp32; DiT matmuls stay bf16 under autocast, result cast back to sequence dtype (same fix as Ming CFM, PR vllm-project#4341). * Medium-1: FM noise drawn from a per-request per-step blake2b-seeded Generator instead of the global CUDA RNG (voxcpm2 parity). Verified: in-session replay is bit-identical. * Medium-3: drop eager talker import from package __init__ (registry imports submodules directly; voxcpm2 convention). Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Replace per-patch isolated AudioVAE.inference_from_latents with the upstream streaming decode path (init_stream_state / stream_step / stream_flush, model.py:1881-1900): - _RequestState grows vocoder_stream_state (lazy-allocated, reset on prefill); the per-request BigVGANStreamState carries the decoder's LSTM hidden + sliding 33-frame latent window, so every patch is decoded with real left conv context instead of zero-padded isolated boundaries. - Prefill's DiT patch #0 now reaches the vocoder: upstream zero-shot emits every decode-loop patch (the drop at model.py:1778 only applies under prompt prefill / voice clone), and our prefill patch corresponds to upstream's first decode-loop patch. This restores the missing first 160 ms (whisper: first word back to 'Hello', was 'So'). - Stop path drains the decoder's 2-frame lookahead tail via stream_flush in the same step is_stopping fires. Verified: sample conservation 22 x 7680 exact (first chunk 3840 + 21 x 7680 + flush 3840); duration 3.52s = previous baseline 3.36s + patch #0 + tail; stream_step 45-60 ms/patch vs 29 ms before (fp32 window decode, ~8x frames per step); e2e RTF 1.78 on RTX 5080/WSL2. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Reviewer (Low): local session/tracking files (dots_tts_notes.md, runpod_voxcpm2_setup.md, scratch_*) don't belong in the repo's .gitignore — moved to .git/info/exclude locally. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

- Drop the DOTS_TTS_FP32_SIDE experiment hook (obsoleted by fp32 ODE integration landing as default) and the DOTS_TTS_HIDDEN_DUMP regression-diff hook (diagnosis complete). DOTS_TTS_BETA_TRACE stays as the model's env-gated debugging instrument. - Rewrite the module docstring from an incremental step changelog to an architecture summary; retire stale step tags / references to local session notes throughout comments and log strings; translate the remaining Chinese comments (prompt-template literals stay). - dots_tts.yaml: remove the dots_tts_runtime_config block — it never had a consumer (the runtime-config class was cut early on); translate header comments. - end2end.py: drop the mm-dump / per-patch-rms diagnostic prints. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Replace the hardcoded _build_soar_*_config() factories with parsers over the checkpoint's config.json blocks: - vocoder / DiT blocks map 1:1 onto AudioVAEConfig / _DiTConfig declared fields, so an unknown or missing key raises instead of being silently dropped; a missing block raises ValueError up front (_require_config_block) rather than falling back to another checkpoint's values. - PatchEncoder keeps the upstream behaviour of consuming only the six keys SuperviseEncoder reads (the rest are swallowed there too). - Module dims (DiT in/out, patch_encoder in/out, the five projectors, CAM++ embedding size / max seconds, speaker-encoder sample rate) now derive from the config instead of literals. - The FM-workspace module constants are cross-checked against the config via _validate_architecture_constants — a checkpoint that disagrees is refused instead of generating garbage. - DotsTTSConfig: drop the never-populated declared params (dit_config / audio_vae_config / bigvgan_config / etc.) — the real config.json sub-blocks arrive via **kwargs as attributes. Verified standalone: factories off the real soar config build AudioVAE (180,898,397 params), DiT (344M), patch_encoder (270 keys), and all three raise on a config missing their block. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

- docs/models/supported_models.md: add DotsTTSForConditionalGeneration row (NVIDIA GPU, zero-shot TTS via rednote-hilab/dots.tts-soar). - tests/model_executor/models/dots_tts/test_talker.py: pure-CPU bare- talker tests (voxcpm2 test pattern) covering the stop-signal contract — positional _results_queue pairing in a mixed prefill/decode batch, the prefill None placeholder, the 0.8-threshold force-continue semantics — and deferred per-request state eviction. 5 tests, no GPU / weights / engine required. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

…oy_config_name DOTS_TTS_PIPELINE never set default_deploy_config_name, so vllm_omni/deploy/dots_tts.yaml was never auto-loaded (unlike voxcpm2, which sets this). enforce_eager and enable_prefix_caching: false silently never took effect. Under current upstream/main, enabling prefix caching routes multimodal output through _build_combined_prefix_cache_mm_payload, which — unlike the non-prefix-cache path — does not skip the meta.req_id/meta.sparse_audio keys, so they leak onto the wire as bare strings, get dropped by _ensure_tensor_values, and the sparse_audio marker is lost. That broke sparse-audio routing and truncated generation to a single 160ms patch. Verified: e2e run now shows enforce_eager=True, enable_prefix_caching=False in the engine config, no more "Dropping non-tensor multimodal output key" warnings, and Duration=3.52s (matching the expected streaming-vocoder chunk math) instead of 0.08s. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Adds dots.tts to examples/offline_inference/text_to_speech/README.md (table row + per-model section, mirroring voxcpm2) and a new recipes/rednote-hilab/dots.tts.md recipe with real RTX 5080 16GB verification numbers (3.52s / RTF 1.569 single-request run) and an explicit known-limitations section: no online-serving adapter yet, voice cloning not wired, dots.tts-mf unsupported, no CUDA graph capture, no concurrent-request scaling, and SamplingParams.seed not threaded to the DiT noise (voxcpm2 parity). Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Moore-Z · 2026-08-11T08:13:16Z

@linyueqian — following up since it's been a while since your review. Status:

All 6 inline findings closed (H1/H2/M1/M2/M3/Low) — commits landed a while back, listed in the earlier reply threads.
Rebased onto current upstream/main (was 196 commits behind). Caught and fixed a real bug in the process: dots_tts/pipeline.py was missing default_deploy_config_name, so dots_tts.yaml was never actually auto-loading — enforce_eager and enable_prefix_caching: false silently weren't taking effect. Under the newer vLLM (0.26.0) that combination surfaced a truncated-output bug via a prefix-cache multimodal-output path that landed upstream after this branch was first written (_build_combined_prefix_cache_mm_payload doesn't preserve the sparse_audio marker this model needs). Fixed, re-verified: clean 3.52s / RTF 1.569 offline run, no more truncation.
Replied on the seed thread with the three-track breakdown — landing on voxcpm2 parity for this PR, generic SamplingParams.seed threading as a follow-up (needs a shared gpu_model_runner.py change, didn't want to bundle that into an already-large diff).
Added the offline README section + a recipe doc (recipes/rednote-hilab/dots.tts.md) with real hardware numbers.
One more gap surfaced while writing the recipe: dots.tts isn't registered in vllm_omni/entrypoints/openai/tts_adapters/, so /v1/audio/speech doesn't know how to build request params for it yet — offline Omni() is the only working path right now. Documented as a known limitation; happy to hear whether you'd want that blocking undraft or fine as a fast-follow.

No pressure on timeline — just wanted to flag this hadn't fallen off radar, and happy to get you a rebased build to re-run your bench matrix against.

linyueqian

Round-2 review at ffc98058. All six findings from my July review are fixed, and I re-ran the model end to end on an H20 to get runtime evidence for the two that needed it.

Verified on GPU (vLLM 0.26.0, dots.tts-soar, whisper-medium for transcription):

Seeded FM noise: audio is now bit identical across two separate engine restarts, sha256 match on 3/3 outputs. In July the same probe gave different durations run to run.
Streaming vocoder: the first-word swallow is gone. 10/10 transcripts came back word exact, including the two texts that reproduced it in July ("Hello everyone, ..." and "The weather forecast ...").
fp32 ODE, prefill queue placeholder, light package init, .gitignore hunk: all check out on read, and test_talker.py now pins the prefill/decode row pairing that the High finding was about, including the sub-threshold force-continue case.
Perf: warm RTF 1.10 to 1.14 sequential, so the full-window vocoder re-decode did not cost anything measurable against the 1.12 to 1.20 I measured in July. At c=4 there is no cross talk and no throughput gain, which matches what the recipe already documents.
The per-request Error concatenating tensor for key sr warning from July no longer appears.

The checkpoint-driven side-path config with fail-loud validation is a nice upgrade over the hardcoded soar values, and the recipe is unusually honest about what is not wired yet. One robustness issue to fix before merge and two small suggestions, all inline.

The branch is CONFLICTING right now: examples/offline_inference/text_to_speech/README.md collides with #5957, which landed on main after your morning merge. Both sides only add a model section, so it is a keep-both resolution.

One process note: tzhouam's approval is from the July head and predates this rewrite by a month, so it is worth a fresh look before this goes in.

…eed, fix docstring Review fixes from PR vllm-project#4765 (linyueqian), round 2 at ffc9805: * [important] _finish_decode raised RuntimeError past the per-request FM buffer capacity, which is engine-fatal, not request-fatal — it takes every other in-flight and future request down with it. Now checks capacity before the step's append and stops the request gracefully (is_stopping=True, stream_flush, real stop signal), same path as a model-decided stop. Also fixed the _MAX_AUDIO_PATCHES comment (~85s was off by 2x; actual is ~164s). * [suggestion] Dropped seed: 42 from dots_tts.yaml — dead with temperature: 0.0 (greedy), and SamplingParams.seed has no path to the DiT flow-matching noise regardless (recipe already documents this). * [suggestion] Fixed _run_patch_encoder_loopback's docstring: return shape is [1, 1, llm_hidden], not [1, _PATCH_ENCODER_OUT_DS_RATE, llm_hidden] — the two encoder positions fold into the feature axis before out_proj, not into separate output positions. Verified the overflow fix on GPU (RTX 5080): patched _MAX_AUDIO_PATCHES to 8 locally, a normal request truncates cleanly at exactly 8 patches (1.28s duration) instead of raising, and a second generate() on the same engine still succeeds. Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

# Conflicts: # examples/offline_inference/text_to_speech/README.md Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Moore-Z · 2026-08-13T06:04:54Z

@linyueqian — round 2 items are all closed out:

FM buffer overflow, dead seed: 42, and the _run_patch_encoder_loopback docstring shape — all fixed and replied inline, threads resolved.
Merged current upstream/main to pick up [New Model] Support IndexTTS 2.5 #5957 (IndexTTS-2.5), which had landed a README conflict with the dots.tts section — resolved (kept both sections) and pushed.

One thing I'd like your read on: tzhouam's approval is from the July head, about a month before this round's rewrite. GitHub still counts it toward merge since nothing dismissed it, but it doesn't reflect the current diff. Would you rather I ask tzhouam for a fresh pass, or are you comfortable with your own review standing in for that given how deep round 2 went?

linyueqian

lgtm

The CUDA ready pipeline selects tests with -m 'core_model and cpu', so the unmarked dots.tts talker tests were deselected and never ran (0 collected, 5 deselected). Add the markers the sibling voxcpm2 talker test already carries, which brings them into the per-PR CPU test step. Verified with the pipeline's own selector against tests/model_executor: 5 collected, 5 passed (was 0 collected). Signed-off-by: Yueqian Lin <linyueqian@outlook.com>

This branch's `configs/__init__.py` referenced `dots_tts` in three places, the `_CLASS_TO_MODULE` map, `__all__`, and the eager-import block, but `configs/dots_tts.py` only exists on main via vllm-project#4765, which landed after this branch's merge base. The eager import therefore raised on any `import vllm_omni`: ImportError: cannot import name 'dots_tts' from partially initialized module 'vllm_omni.transformers_utils.configs' The "circular import" wording is Python's default guess and is misleading here; the module is simply absent. Because `vllm_omni/__init__.py` imports the configs package eagerly, nothing in the tree could be imported, so every pytest run died during plugin collection rather than at a recognizable place. GitHub Actions stayed green throughout: the wheel builds and pre-commit never import the package, and Buildkite had not run on this branch. dots.tts is not part of this PR, so the references are removed rather than the module vendored in. A later rebase onto main picks up both the module and these lines from vllm-project#4765. Signed-off-by: Yueqian Lin <linyueqian@outlook.com>

…TTS (vllm-project#4765) Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Moore-Z requested review from Gaohan123, ZeldaHuang, hsliuustc0106, linyueqian, lishunyang12, princepride, tzhouam, yuanheng-zhao and ywang96 as code owners June 27, 2026 08:28

Moore-Z force-pushed the feat/dots-tts branch from 0e40ceb to e582cde Compare June 28, 2026 05:37

Moore-Z mentioned this pull request Jul 1, 2026

[New Model]: dots.tts (rednote-hilab) — continuous-AR 48kHz zero-shot voice cloning #4218

Closed

1 task done

tzhouam approved these changes Jul 8, 2026

View reviewed changes

linyueqian reviewed Jul 8, 2026

View reviewed changes

Moore-Z requested review from Sy0307, alex-jw-brooks, amy-why-3459 and gcanlin as code owners July 14, 2026 04:12

Moore-Z force-pushed the feat/dots-tts branch from de6f5c2 to 5880173 Compare July 15, 2026 08:51

Moore-Z requested review from david6666666 and yenuo26 as code owners July 15, 2026 08:51

hsliuustc0106 added enhancement New feature or request new model add new model tts code related to tts models labels Jul 17, 2026 — with ChatGPT Codex Connector

Moore-Z added 3 commits August 11, 2026 00:37

Moore-Z added 6 commits August 11, 2026 00:37

Moore-Z force-pushed the feat/dots-tts branch from 9f2b109 to 0d03fd1 Compare August 11, 2026 06:42

Moore-Z requested a review from NickCao as a code owner August 11, 2026 06:42

Moore-Z added 2 commits August 11, 2026 01:30

Merge branch 'main' into feat/dots-tts

ffc9805

linyueqian reviewed Aug 12, 2026

View reviewed changes

Comment thread vllm_omni/model_executor/models/dots_tts/dots_tts_talker.py

Comment thread vllm_omni/deploy/dots_tts.yaml Outdated

Comment thread vllm_omni/model_executor/models/dots_tts/dots_tts_talker.py Outdated

Moore-Z added 2 commits August 12, 2026 23:59

Merge remote-tracking branch 'upstream/main' into feat/dots-tts

d65edb7

# Conflicts: # examples/offline_inference/text_to_speech/README.md Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

linyueqian approved these changes Aug 13, 2026

View reviewed changes

linyueqian enabled auto-merge (squash) August 13, 2026 17:08

linyueqian added the ready label to trigger buildkite CI label Aug 13, 2026

linyueqian added this to the v0.28.0 milestone Aug 13, 2026

linyueqian added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Aug 13, 2026

linyueqian merged commit 90a08c0 into vllm-project:main Aug 13, 2026
7 of 9 checks passed

mo-ke-ke mentioned this pull request Aug 13, 2026

[Model][Performance] Optimize MiniMax-H3 strict Ulysses boundaries #6173

Merged

linyueqian changed the title ~~[Model][TTS][WIP] Add dots.tts (rednote-hilab) — continuous-AR 48kHz TTS~~ [Model][TTS] Add dots.tts (rednote-hilab): continuous-AR 48kHz TTS Aug 13, 2026

linyueqian mentioned this pull request Aug 13, 2026

[CI/Build] Add dots.tts weekly e2e coverage #6174

Merged

sphinxkkkbc mentioned this pull request Aug 16, 2026

[Feat][dots.tts] Add Online Serving Support #6235

Merged

khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026

[Model][TTS][WIP] Add dots.tts (rednote-hilab) — continuous-AR 48kHz …

93bdae9

…TTS (vllm-project#4765) Signed-off-by: Moore-Z <zhumoore9661996@gmail.com>

Navigation Menu

Repository navigation

[Model][TTS] Add dots.tts (rednote-hilab): continuous-AR 48kHz TTS - #4765

[Model][TTS] Add dots.tts (rednote-hilab): continuous-AR 48kHz TTS#4765
linyueqian merged 18 commits into
vllm-project:mainfrom
Moore-Z:feat/dots-tts

Moore-Z commented Jun 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 27, 2026

Uh oh!

tzhouam left a comment

Uh oh!

linyueqian left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Moore-Z commented Aug 11, 2026

Uh oh!

linyueqian left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Moore-Z commented Aug 13, 2026

Uh oh!

linyueqian left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Moore-Z commented Jun 27, 2026

Summary

Implementation overview

New files

Vendor strategy

Diagnostics (env-gated, 0 production overhead)

Test plan

Functional smoke test

Reference (upstream) for A/B comparison

upstream is not on PyPI; clone + install from source

Checkpoints

Known follow-up (still WIP)

Uh oh!

chatgpt-codex-connector Bot commented Jun 27, 2026

Uh oh!

tzhouam left a comment

Choose a reason for hiding this comment

Uh oh!

linyueqian left a comment

Choose a reason for hiding this comment

Suggested optimization path

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Moore-Z commented Aug 11, 2026

Uh oh!

linyueqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Moore-Z commented Aug 13, 2026

Uh oh!

linyueqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants