[New Model] Ming-3-TTS (inclusionAI/Ming-3.0-tts-tiny) Support

benchmarks/tts/model_configs.yaml

@@ -34,6 +34,13 @@ models:
       voice_clone:
         task_type: Base
 
+  inclusionAI/Ming-3.0-tts-tiny:
+    supported_tasks: [voice_clone]
+    backend: openai-audio-speech
+    endpoint: /v1/audio/speech
+    task_extra_body:
+      voice_clone: {}
+
   IndexTeam/IndexTTS-2.5:
     supported_tasks: [voice_clone]
     backend: openai-audio-speech

docs/models/supported_models.md

@@ -93,6 +93,7 @@ th {
 | `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
 | `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
 | `MingTTSForConditionalGeneration` | Ming-omni-tts dense 0.5B | `inclusionAI/Ming-omni-tts-0.5B` | ✅︎ | | | ✅︎ | — |
+| `Ming3TTSForConditionalGeneration` | Ming-3.0-TTS Tiny (Ling V3 MoE) | `inclusionAI/Ming-3.0-tts-tiny` | ✅︎ | | | ✅︎ | — |
 | `GLMTTSForConditionalGeneration` | GLM-TTS | `zai-org/GLM-TTS` | ✅︎ | | | | — |
 | `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenMOSS/MOSS-TTS-Nano-NPU.md) |
 | `MossTTSDelayModel` | MOSS-TTS, MOSS-TTSD, MOSS-SoundEffect, MOSS-VoiceGenerator | `OpenMOSS-Team/MOSS-VoiceGenerator` | ✅︎ | | | | — |

recipes/inclusionAI/Ming-3.0-TTS.md

@@ -0,0 +1,160 @@
+# Ming-3.0-TTS Tiny
+
+> Voice cloning, speech control, voice design, and multi-speaker Podcast.
+
+## Summary
+
+- Vendor: inclusionAI
+- Model: [`inclusionAI/Ming-3.0-tts-tiny`](https://huggingface.co/inclusionAI/Ming-3.0-tts-tiny)
+- Runtime: vLLM-Omni two-stage online serving
+- API: OpenAI-compatible `/v1/audio/speech`
+
+## Architecture
+
+Stage 0 uses the Ling V3 MoE backbone (18 KDA / 6 MLA layers) and a Flow decoder
+to generate continuous audio latents. Stage 1 converts them to streaming audio
+with AudioVAE. The pipeline uses [ming3_tts.yaml](../../vllm_omni/deploy/ming3_tts.yaml).
+
+## CUDA
+
+### Environment
+
+- OS: Linux
+- Python: 3.12.3
+- CUDA: 13.0
+- PyTorch: 2.13.0+cu130; Transformers: 5.19.0
+- vLLM version: 0.32.0
+- vLLM-Omni commit: `9eb1adcc0`
+- GPU: 1x H100 / H200
+
+### Reference audio
+
+Run from the repository root. These examples use two
+[Seed-TTS evaluation](https://github.com/BytedanceSpeech/seed-tts-eval) clips from
+a [pinned HF mirror](https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval/tree/8f5e1aa2a35d42f42e940074c1983358b9491f89).
+Download both WAVs before starting the server; requests use the local copies.
+Commands require `curl` and `jq`.
+
+```bash
+REF_DIR="$PWD/.cache/ming3_tts/references"
+BASE=https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval/resolve/8f5e1aa2a35d42f42e940074c1983358b9491f89/zh/prompt-wavs
+mkdir -p "$REF_DIR"
+curl --fail --location --show-error "$BASE/10002287-00000094.wav" --output "$REF_DIR/speaker_1.wav"
+curl --fail --location --show-error "$BASE/10002290-00000094.wav" --output "$REF_DIR/speaker_2.wav"
+```
+
+| File | Exact transcript |
+| --- | --- |
+| `speaker_1.wav` | 在此奉劝大家别乱打美白针。 |
+| `speaker_2.wav` | 对此，乔宝云认为有夸大的成分。 |
+
+### Commands
+
+Start the server in one terminal using the uv `.venv`. The HF ID selects the
+pipeline automatically. Run the request examples in another terminal from the
+same repository root.
+
+```bash
+MODEL=inclusionAI/Ming-3.0-tts-tiny
+REF_DIR="$PWD/.cache/ming3_tts/references"
+CUDA_VISIBLE_DEVICES=0 .venv/bin/vllm serve "$MODEL" \
+  --omni --port 8000 --allowed-local-media-path "$REF_DIR"
+```
+
+Use `--revision <HF-commit>` to pin a checkpoint. The HF model weights download
+automatically on first launch.
+
+## Voice cloning
+
+Use the downloaded `speaker_1.wav` and its exact transcript. Omit `ref_text`
+for speaker-only conditioning.
+
+```bash
+REF_DIR="$PWD/.cache/ming3_tts/references"
+jq -n --arg ref_audio "file://$REF_DIR/speaker_1.wav" \
+  '{
+    "model": "inclusionAI/Ming-3.0-tts-tiny",
+    "input": "傍晚的风从山口吹过来，远处的云慢慢染成了金色。",
+    "ref_audio": $ref_audio,
+    "ref_text": "在此奉劝大家别乱打美白针。",
+    "language": "Chinese",
+    "response_format": "wav",
+    "max_new_tokens": 200,
+    "extra_params": {"temperature": 0.0, "ming3_cfg": 1.0},
+    "seed": 1234
+  }' |
+curl --fail-with-body --silent --show-error http://127.0.0.1:8000/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  --data-binary @- --output ming3_tiny_clone.wav
+```
+
+## Speech control and voice design
+
+Pass structured controls as a JSON string in `instructions`, or use freeform
+text. Speaker-only control tasks take reference audio without `ref_text`.
+
+| Task | Control | Reference audio |
+| --- | --- | --- |
+| Emotion | `情感`, e.g. `高兴` | Required |
+| Dialect | `方言`, e.g. `广粤话` | Required |
+| Speed / pitch / volume | `语速` / `基频` / `音量` | Required |
+| Freeform | Natural-language instructions | Required |
+| Voice design | `风格`, e.g. `年轻女声，声音清晰自然。` | Omit |
+| Built-in voice | `IP`: vendor voice name | Omit |
+
+Dialect example:
+
+```bash
+REF_DIR="$PWD/.cache/ming3_tts/references"
+jq -n --arg ref_audio "file://$REF_DIR/speaker_1.wav" \
+  '{
+    "model": "inclusionAI/Ming-3.0-tts-tiny",
+    "input": "今日天气几好，我哋食完饭一齐去海边行下啦。",
+    "ref_audio": $ref_audio,
+    "instructions": "{\"方言\":\"广粤话\"}",
+    "response_format": "wav",
+    "extra_params": {"temperature": 0.0, "ming3_cfg": 1.0},
+    "seed": 1234
+  }' |
+curl --fail-with-body --silent --show-error http://127.0.0.1:8000/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  --data-binary @- --output ming3_tiny_dialect.wav
+```
+
+## Podcast
+
+Provide one reference clip and exact transcript per speaker, in speaker order.
+Each dialogue line starts with `speaker_N:`. The example uses both downloaded
+WAVs; replace audio and transcripts together when using your own clips.
+
+```bash
+REF_DIR="$PWD/.cache/ming3_tts/references"
+jq -n \
+  --arg ref_a "file://$REF_DIR/speaker_1.wav" \
+  --arg ref_b "file://$REF_DIR/speaker_2.wav" \
+  '{
+    "model": "inclusionAI/Ming-3.0-tts-tiny",
+    "input": "speaker_1:你好，今天我们聊聊语音合成。\nspeaker_2:好呀，我们开始吧。",
+    "ref_audio": [$ref_a, $ref_b],
+    "ref_text": "speaker_1:在此奉劝大家别乱打美白针。\nspeaker_2:对此，乔宝云认为有夸大的成分。",
+    "response_format": "wav",
+    "extra_params": {"temperature": 0.0, "ming3_cfg": 1.0},
+    "seed": 1235
+  }' |
+curl --fail-with-body --silent --show-error http://127.0.0.1:8000/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  --data-binary @- --output ming3_tiny_podcast.wav
+```
+
+## Notes
+
+- Output is mono 44.1 kHz WAV. For PCM streaming, set `response_format: "pcm"` and `stream_format: "audio"`.
+- CFM CUDA Graphs follow Stage 0's configuration: `enforce_eager: true` or `compilation_config.cudagraph_mode: NONE` disables their compilation and capture.
+- Context limit: 4096 tokens. Both stages allow 16 scheduled sequences;
+  For chunked prefill, speaker and reference-audio conditioning is cached per request and sliced using the scheduled token offset, including single-token prefill slices.
+  AudioVAE currently decodes requests individually within a scheduled batch.
+- Reference clips must be 1–30 seconds. Upload clips through `/v1/audio/voices` and reuse their names in `voice`; built-in IP names use `instructions`.
+- Flow temperature defaults to 1.5; the examples explicitly use 0.0.
+  `ming3_cfg=1` corresponds to native CFG scale 2. The legacy `lingv3_cfg` alias and checkpoint `lingv3_tts` metadata remain supported.
+- Podcast infers language from the dialogue and does not accept `instructions`. Decode bounds are derived from the turn text; an explicit `max_new_tokens` must not be below the derived minimum.
+- The Podcast example has a second-speaker similarity warning. With the same references, temperature 1.5 / seed 1234 / `language: "Chinese"` matched both intended speakers in the sampling control. Voice quality still needs listening.

tests/entrypoints/openai_api/test_tts_adapter.py

@@ -33,6 +33,7 @@
     detect_tts_model_type,
     resolve_adapter,
 )
+from vllm_omni.entrypoints.openai.tts_adapters import ming3_tts as ming3_tts_module
 from vllm_omni.entrypoints.openai.tts_adapters.auk import AuKAdapter
 from vllm_omni.entrypoints.openai.tts_adapters.base import TTSCapabilities, resolve_stage_model_path
 from vllm_omni.entrypoints.openai.tts_adapters.covo_audio import CovoAudioAdapter
@@ -42,6 +43,7 @@
     IndexTTS25Adapter,
     indextts2_conditioning_cache_salt,
 )
+from vllm_omni.entrypoints.openai.tts_adapters.ming3_tts import Ming3TTSAdapter
 from vllm_omni.entrypoints.openai.tts_adapters.ming_flash_omni_tts import MingFlashOmniTTSAdapter
 from vllm_omni.entrypoints.openai.tts_adapters.moss_tts import (
     MossTTSAdapter,
@@ -77,6 +79,7 @@
     "omnivoice",
     "covo_audio",
     "ming_tts",
+    "ming3_tts",
     "moss_tts_nano",
     "moss_tts",
     "higgs_audio_v2",
@@ -1037,6 +1040,91 @@ def test_qwen3_tts_rejects_only_length_finished_base_audio(mocker):
         adapter.validate_generation(params, stage0_finish_reason="length", output_tokens=191)
 
 
+@pytest.mark.parametrize("architecture", ["Ming3TTSForConditionalGeneration", "LingV3TTSForConditionalGeneration"])
+def test_detects_ming3_by_architecture(architecture):
+    assert detect_tts_model_type("llm", architecture) == "ming3_tts"
+
+
+def test_ming3_build_accepts_resolver_cache_key(mocker):
+    server = SimpleNamespace(_tts_tokenizer=object())
+    server._resolve_ref_audio = mocker.AsyncMock(return_value=([0.1], 24000, "cache-key"))
+    server._get_resolved_ref_audio_artifact_key = mocker.Mock(return_value="artifact-key")
+    mocker.patch.object(
+        ming3_tts_module,
+        "build_ming3_prompt",
+        return_value={
+            "prompt_token_ids": [1],
+            "prompt": "prompt",
+            "text": "hello",
+            "additional_information": {},
+        },
+    )
+    request = OpenAICreateSpeechRequest(
+        input="hello",
+        ref_audio="file:///reference.wav",
+        ref_text=None,
+        max_new_tokens=200,
+    )
+
+    prepared = asyncio.run(Ming3TTSAdapter(SpeechServingContext(server=server)).build(request, [], False))
+
+    server._resolve_ref_audio.assert_awaited_once_with(request.ref_audio)
+    server._get_resolved_ref_audio_artifact_key.assert_called_once_with("cache-key")
+    assert prepared.prompt["additional_information"]["speaker_audio_keys"] == ["artifact-key"]
+    assert prepared.model_type == "ming3_tts"
+
+
+def test_ming3_sampling_overrides_attach_request_rng_state(mocker):
+    server = mocker.Mock()
+    server.engine_client.model_config.hf_config.tts_config = {}
+    adapter = Ming3TTSAdapter(SpeechServingContext(server=server))
+    defaults = [SamplingParams(max_tokens=801)]
+    prompt: dict[str, Any] = {"additional_information": {}}
+    request = OpenAICreateSpeechRequest(input="hello", max_new_tokens=200, seed=1234)
+
+    overridden = adapter.apply_sampling_overrides(defaults, request, prompt, "request-1")
+
+    assert overridden[0].max_tokens == 201
+    assert prompt["additional_information"] == {
+        "ming3_request_id": "request-1",
+        "tts_local_seed": 1234,
+    }
+
+
+def test_ming3_rejects_transcript_without_reference_audio():
+    request = OpenAICreateSpeechRequest(input="hello", ref_text="Reference transcript.")
+    server = SimpleNamespace(_apply_uploaded_speaker=lambda request: None)
+    adapter = Ming3TTSAdapter(SpeechServingContext(server=server))
+
+    assert adapter.validate(request) == "'ref_text' requires reference audio"
+
+
+def test_ming3_accepts_audio_backed_uploaded_voice(mocker):
+    def resolve_uploaded_voice(request):
+        request.ref_audio = "data:audio/wav;base64,AA=="
+        request.ref_text = "Reference transcript."
+        return None
+
+    server = SimpleNamespace(
+        _apply_uploaded_speaker=mocker.Mock(side_effect=resolve_uploaded_voice),
+        _validate_ref_audio_format=lambda ref_audio: None,
+    )
+    request = OpenAICreateSpeechRequest(
+        input="hello",
+        voice="alice",
+        ref_audio=None,
+        ref_text=None,
+        instructions=None,
+        speaker_embedding=None,
+        max_new_tokens=None,
+    )
+
+    assert Ming3TTSAdapter(SpeechServingContext(server=server)).validate(request) is None
+    server._apply_uploaded_speaker.assert_called_once_with(request)
+    assert request.ref_audio == "data:audio/wav;base64,AA=="
+    assert request.ref_text == "Reference transcript."
+
+
 def test_indextts_adapters_are_versioned():
     assert resolve_adapter("indextts2") is IndexTTS2Adapter
     assert resolve_adapter("indextts2_5") is IndexTTS25Adapter

tests/entrypoints/openai_api/test_tts_detection.py

@@ -287,14 +287,21 @@ def test_pipeline_stage_list_is_complete():
 
 
 def test_entry_stage_archs_are_explicit():
-    """Only Ming dense and AuK identify their entry stage by architecture.
+    """Ming dense, AuK, and Ming3 identify entry stages by architecture.
 
-    Both use a generic ``model_stage`` (``llm`` and ``encoder``) that other
+    They use generic ``model_stage`` values (``llm`` and ``encoder``) that other
     pipelines share. Widening this set would change which stage
     ``_find_tts_stage`` selects in mixed deployments — notably VoxCPM2, which
     declares ``model_archs`` but is found by its ``latent_generator`` stage key.
     """
-    assert tts_entry_stage_archs() == frozenset({"MingTTSForConditionalGeneration", "AuKForConditionalGeneration"})
+    assert tts_entry_stage_archs() == frozenset(
+        {
+            "MingTTSForConditionalGeneration",
+            "AuKForConditionalGeneration",
+            "Ming3TTSForConditionalGeneration",
+            "LingV3TTSForConditionalGeneration",
+        }
+    )
 
 
 def test_minimax_h3_encoder_is_not_a_tts_stage():