[Qwen3TTS][Feat] Streaming output

[JuanPZuluaga]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Simple streaming output implementation for Qwene3TTS models for the latest disaggregated inference pipeline.

one can test it with:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice  --host 0.0.0.0 \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
--omni --port 8091 --trust-remote-code \
--enforce-eager --gpu-memory-utilization 0.3

then:

curl -X POST http://0.0.0.0:8091/v1/audio/speech  -H "Content-Type: application/json" \
-d '{"input": "Hello, how are you? I am Pablo and I am happy to contribute to vLLM omni.", "voice": "vivian", "language": "English", "stream": true, "response_format": "pcm", "instructions": "speak ultra slow"}' \
--no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Thanks.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please providing the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please pasting the results comparison before and after, or e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec1c7f2941

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

Streaming output for Qwen3-TTS

This PR adds streaming PCM output for the TTS endpoint and an offline streaming example via AsyncOmni. The overall approach is sound — constraining streaming to raw PCM, adding pydantic validators, extracting shared helpers, and providing tests. A few issues worth addressing before merge:

1. Silent failure on streaming errors: The _generate_pcm_chunks generator swallows exceptions with only a log message, so the client receives a truncated 200 response with no indication of failure. At minimum, raising after logging (or yielding an error sentinel) would be better.
2. AudioResponse import removed from serving_speech.py but the class is still used in the non-streaming path via self.create_audio() return type — verify this doesn't break type checking or downstream usage.
3. _estimate_prompt_len called but not defined in the offline example's _build_inputs helper — this will crash at runtime when --txt-prompts is used.
4. Hardcoded IP address in the PR description curl example (10.16.59.44) — should be localhost or a placeholder.
5. Several minor nits on robustness and style noted inline.

[hsliuustc0106]

Streaming output for Qwen3-TTS

Good feature addition overall — the streaming path, protocol validation, and documentation are well-structured. A few issues to address before merging:

1. Bug: CancelledError silently swallowed in streaming generator — the except asyncio.CancelledError block logs but does not re-raise, so FastAPI cannot detect client disconnection and clean up properly.
2. Bug: _estimate_prompt_len is undefined — _build_inputs references it in the --txt-prompts path, but the function is never defined or imported. The --txt-prompts CLI argument itself is also missing from parse_args().
3. Inconsistency: _extract_audio_output has two different contracts — the diff introduces a version returning (dict, key) for both paths, but the streaming and non-streaming paths then each manually re-extract sr and audio_tensor with duplicated logic that could diverge.
4. finished flag only added to non-image branch in async_omni.py — the image branch still defaults to finished=True, which is inconsistent.

See inline comments for details.

[hsliuustc0106]

@vllm-omni-reviewer

[github-actions]

🤖 VLLM-Omni PR Review

Code Review: [Qwen3TTS][Feat] Streaming output

1. Overview

This PR implements streaming output support for Qwen3TTS models in the disaggregated inference pipeline. The changes include:

• Protocol layer: Adding stream field with validation constraints to OpenAICreateSpeechRequest
• Serving layer: Implementing _generate_pcm_chunks() for progressive PCM audio delivery
• Async infrastructure: Adding finished flag to OmniRequestOutput
• Examples & docs: Comprehensive documentation for both online serving and offline inference streaming modes
• Tests: Unit and integration tests for streaming validation and response handling

Overall Assessment: Positive - This is a well-structured implementation with good documentation, appropriate validation, and solid test coverage. The design follows existing patterns in the codebase.

---

2. Code Quality

Strengths

• Clean separation between streaming and non-streaming code paths
• Proper use of Pydantic validators for input validation
• Good error handling with asyncio.CancelledError support
• Consistent code style with the existing codebase

Potential Issues

vllm_omni/entrypoints/openai/serving_speech.py:218-261 - The _generate_pcm_chunks method handles two different audio output modes (cumulative list vs per-step tensor). While the logic appears correct, the dual-mode handling could lead to confusion:

if isinstance(audio_val, list):
    # Cumulative mode: each update grows the list; emit only new tail.
    new_chunks = audio_val[prev_count:]
    prev_count = len(audio_val)
else:
    # Per-step mode: each update is a single tensor; emit directly.
    new_chunks = [audio_val] if audio_val is not None else []

Consider adding a docstring explaining when each mode is used.

vllm_omni/entrypoints/openai/protocol/audio.py:72-73 - The validator mutates self.speed which is a side effect that could be surprising:

if self.speed is None:
    self.speed = 1.0

This is acceptable but worth documenting that the validator normalizes the value.

examples/offline_inference/qwen3_tts/end2end.py:258-261 - The _save_wav function handles both list and tensor audio data, but the list concatenation assumes all chunks have compatible dimensions:

audio_tensor = torch.cat(audio_data, dim=-1) if isinstance(audio_data, list) else audio_data

This should be safe given the streaming implementation, but defensive programming could add a check.

---

3. Architecture & Design

Integration

• The finished flag addition to OmniRequestOutput is minimal and non-breaking
• The streaming path correctly uses StreamingResponse from FastAPI
• The _extract_audio_output static method properly abstracts the audio extraction logic

Design Patterns

• Good use of async generators for streaming
• Proper separation of concerns between protocol validation and serving logic

Potential Improvements

vllm_omni/entrypoints/openai/serving_speech.py:263-277 - The _extract_audio_output method could benefit from a more explicit return type:

@staticmethod
def _extract_audio_output(res) -> tuple[dict | None, str | None]:

Consider using a TypedDict or NamedTuple for clearer semantics, or at minimum add a docstring with examples.

vllm_omni/entrypoints/async_omni.py:520 & 527 - The finished field is added in two branches. Consider if there's a way to consolidate this logic to avoid duplication.

---

4. Security & Safety

Input Validation

• ✅ Proper validation that stream=true requires response_format="pcm"
• ✅ Speed adjustment is correctly blocked during streaming
• ✅ SSE stream format is explicitly blocked

Resource Management

• ✅ asyncio.CancelledError is properly caught and re-raised in _generate_pcm_chunks
• ✅ Exceptions are logged with request ID for traceability

No Critical Security Concerns

The streaming implementation only returns generated audio data with no user input directly reflected in the output.

---

5. Testing & Documentation

Test Coverage

• ✅ Unit tests for protocol validation (TestStreamingProtocolValidation)
• ✅ Integration tests for streaming response (TestStreamingResponse)
• ✅ Test for non-streaming path unchanged

Suggestion: Consider adding a test case for the asyncio.CancelledError path to ensure proper cleanup.

Documentation

• ✅ Comprehensive documentation in multiple README files
• ✅ Clear constraints documented (PCM format, no speed adjustment)
• ✅ Working curl examples provided
• ✅ Limitations section updated to remove "No streaming"

---

6. Specific Suggestions

vllm_omni/entrypoints/openai/serving_speech.py:218

Add a more detailed docstring explaining the two audio output modes:

async def _generate_pcm_chunks(self, generator, request_id: str):
    """Generate PCM audio chunks for streaming response.
    
    Handles two audio output modes from the engine:
    - Cumulative mode (list): Engine returns growing list of chunks; 
      we emit only the new tail on each iteration.
    - Per-step mode (tensor): Engine returns single tensor per iteration;
      we emit it directly.
    
    Args:
        generator: Async generator from the engine
        request_id: Request identifier for logging
        
    Yields:
        Raw PCM bytes for each audio chunk
    """

vllm_omni/entrypoints/openai/serving_speech.py:325-327

Consider adding a comment explaining why VoiceDesign requires non-streaming mode:

# VoiceDesign requires non_streaming_mode because it performs voice
# cloning operations that need the complete audio context.
# CustomVoice and Base rely on model defaults (True and False respectively).
if params["task_type"][0] == "VoiceDesign":
    params["non_streaming_mode"] = [True]

examples/offline_inference/qwen3_tts/end2end.py:243

The _build_inputs function references _estimate_prompt_len which isn't shown in the diff. Ensure this function exists and is properly imported or defined.

tests/entrypoints/openai_api/test_serving_speech.py:490-493

Consider adding a test for the cancelled request scenario:

def test_streaming_cancelled(self, streaming_app):
    """Streaming request cancellation is handled gracefully."""
    client = TestClient(streaming_app)
    # Simulate client disconnect - would need async test client
    # This is a placeholder for future async test implementation

---

7. Approval Status

LGTM with suggestions

The PR is well-implemented with:

• Clean code structure following existing patterns
• Comprehensive documentation and examples
• Appropriate input validation
• Good test coverage

The suggestions above are minor improvements for code clarity and maintainability, not blocking issues. The implementation is ready to merge after addressing any documentation nits.

Recommended actions before merge:

1. Consider adding the suggested docstrings for clarity
2. Verify _estimate_prompt_len is properly available in end2end.py
3. Ensure the PR description checklist items are completed

---

This review was generated automatically by the VLLM-Omni PR Reviewer Bot
using glm-5.

[JuanPZuluaga]

should be ready @hsliuustc0106.

[hsliuustc0106]

any test result with your suggested commands?

[JuanPZuluaga]

1. launched the model
2. I ran a couple of warmup requests
3. then ran a couple of experiments:

Non streaming request:

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you? I am Pablo and I am happy to contribute to vLLM omni.", "voice": "vivian", "language": "English", "stream": false, "response_format": "pcm"}' \
  -o ./output_nonstream.pcm \
  -w "Total time: %{time_total}s\n"

same request sent 2 times:

TTFB: 4.947408s | Total: 4.953832s
TTFB: 5.184883s | Total: 5.196970s

Streaming output

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you? I am Pablo and I am happy to contribute to vLLM omni.", "voice": "vivian", "language": "English", "stream": true, "response_format": "pcm"}' \
  -o ./output_stream.pcm \
  --no-buffer \
  -w "TTFB: %{time_starttransfer}s | Total: %{time_total}s\n"

same request sent 2 times:
TTFB: 0.015514s | Total: 5.563862s
TTFB: 0.013207s | Total: 4.442476s

@hsliuustc0106 TTFA is quite low. In another PR, we can tune a desired TTFA based on a given max_seqs and we can change the code2wav logic. note that when you send a streaming request, you might not listen to the audio right away due to the silence generated by the model (this should be normal and expected).

EDIT: Added a fix where the first audio token was repeated twice.

[hsliuustc0106]

1. launched the model
2. I ran a couple of warmup requests
3. then ran a couple of experiments:

Non streaming request:

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you? I am Pablo and I am happy to contribute to vLLM omni.", "voice": "vivian", "language": "English", "stream": false, "response_format": "pcm"}' \
  -o ./output_nonstream.pcm \
  -w "Total time: %{time_total}s\n"

same request sent 2 times:

TTFB: 4.947408s | Total: 4.953832s TTFB: 5.184883s | Total: 5.196970s

Streaming output

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you? I am Pablo and I am happy to contribute to vLLM omni.", "voice": "vivian", "language": "English", "stream": true, "response_format": "pcm"}' \
  -o ./output_stream.pcm \
  --no-buffer \
  -w "TTFB: %{time_starttransfer}s | Total: %{time_total}s\n"

same request sent 2 times: TTFB: 0.015514s | Total: 5.563862s TTFB: 0.013207s | Total: 4.442476s

@hsliuustc0106 TTFA is quite low. In another PR, we can tune a desired TTFA based on a given max_seqs and we can change the code2wav logic. note that when you send a streaming request, you might not listen to the audio right away due to the silence generated by the model (this should be normal and expected).

EDIT: Added a fix where the first audio token was repeated twice.

so, the accuracy will not be tested here?

[JuanPZuluaga]

so, the accuracy will not be tested here?

@hsliuustc0106 makes sense. I'll run a simple benchmark and post the results later here. So we know the audio is the same.

[JuanPZuluaga]

@hsliuustc0106 so i ran a test with:

non streaming:

(is not in the PR: examples/offline_inference/qwen3_tts/test_prompts.txt)

python examples/offline_inference/qwen3_tts/end2end.py \
  --query-type CustomVoice \
  --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
  --txt-prompts examples/offline_inference/qwen3_tts/test_prompts.txt \
  --output-dir output/nonstreaming

streaming:

python examples/offline_inference/qwen3_tts/end2end.py \
  --query-type CustomVoice \
  --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
  --txt-prompts examples/offline_inference/qwen3_tts/test_prompts.txt \
  --output-dir output/streaming \
  --streaming 

I got a local script to compare similarity based in some metrics and produced this:

Comparing 10 pairs

File                               Corr        MSE  MaxDiff   Dur-S  Dur-NS  LenDiff   RMS-S  RMS-NS
----------------------------------------------------------------------------------------------------
output_0.wav                     1.0000   0.000000  0.00000   6.24s   6.24s        0  0.0847  0.0847
output_1.wav                     1.0000   0.000000  0.00000   5.68s   5.68s        0  0.0400  0.0400
output_2.wav                     1.0000   0.000000  0.00000   6.00s   6.00s        0  0.0667  0.0667
output_3.wav                     1.0000   0.000000  0.00000   5.60s   5.60s        0  0.1028  0.1028
output_4.wav                     1.0000   0.000000  0.00000   5.68s   5.68s        0  0.0702  0.0702
output_5.wav                     1.0000   0.000000  0.00000   4.16s   4.16s        0  0.0537  0.0537
output_6.wav                     1.0000   0.000000  0.00000   5.52s   5.52s        0  0.0989  0.0989
output_7.wav                     1.0000   0.000000  0.00000   5.76s   5.76s        0  0.0953  0.0953
output_8.wav                     1.0000   0.000000  0.00000   5.36s   5.36s        0  0.0941  0.0941
output_9.wav                     1.0000   0.000000  0.00000   4.80s   4.80s        0  0.0729  0.0729
----------------------------------------------------------------------------------------------------
MEAN                             1.0000   0.000000            5.48s   5.48s      0.0  0.0779  0.0779
STD                              0.0000   0.000000            0.57s   0.57s      0.0  0.0198  0.0198

meaning the 10 audios are identical.

I can add the script, but i think it's not necessary.

[hsliuustc0106]

fix conflicts

[JuanPZuluaga]

@hsliuustc0106 fixed. Thanks.

[JuanPZuluaga]

Fixed again conflicts, could we merge? @hsliuustc0106

[hsliuustc0106]

Fixed again conflicts, could we merge? @hsliuustc0106

I believe so, let me run the CI

[hsliuustc0106]

fix precommits please

[JuanPZuluaga]

@hsliuustc0106 done. Thanks.