Make speech streaming default to SSE

[syd520zy]

Summary

Make /v1/audio/speech streaming requests default to SSE.

This implements Option A from #4612:

• stream=True now returns OpenAI-style speech.audio.* SSE events.
• stream_format="sse" continues to return SSE events.
• stream_format="audio" remains available as an explicit opt-in for raw PCM/WAV byte streaming.
• Non-streaming speech requests remain unchanged.
• Updated in-repo raw audio streaming consumers, docs, benchmarks, and e2e request configs to explicitly use stream_format="audio".

Motivation

SSE provides structured stream events, which makes it possible to carry stream-level metadata such as usage, done events, and error events. Raw audio byte streaming cannot represent this metadata cleanly.

Issue #4612 discusses the server-side default behavior for speech streaming format. This PR follows the selected Option A direction: make speech streaming default to SSE while keeping raw audio streaming available through an explicit stream_format="audio" request.

Behavior

The updated behavior is:

Request	Behavior
no stream, no stream_format	non-streaming audio response
stream=True	SSE streaming
stream_format="sse"	SSE streaming
stream=True + stream_format="sse"	SSE streaming
stream_format="audio"	raw audio streaming
stream=True + stream_format="audio"	raw audio streaming

word_timestamps=true is still rejected for HTTP speech streaming, because word timestamp streaming is currently implemented by the WebSocket /v1/audio/speech/stream path.

Notes

This intentionally changes the compatibility behavior introduced by #4490, where stream=True preferred raw audio. Raw audio streaming remains available through stream_format="audio".

Tests

Ran:

• pre-commit run ruff-format --files ... on the modified speech API, raw-streaming example/client, benchmark, helper, and test files
• pre-commit run ruff-check --files ... on the same modified Python files
• git diff --check
• python -m py_compile ... on the modified Python files

Not run:

• Full e2e tests. The e2e request configs were updated semantically to pass stream_format="audio" for raw audio streaming, but model/hardware-backed e2e validation was not run locally.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[linyueqian]

Code review

The routing logic itself is correct. I checked the full stream x stream_format truth table and every combination resolves cleanly: stream_format="audio" still wins over the new SSE default, nothing is double-classified, and hoisting the word_timestamps guard to is_streaming() is sound. Making SSE the default for stream=True (issue #4612 "Option A") is the intended direction.

The remaining ask is to migrate the in-repo consumers and docs that still assume the old stream=True -> raw-PCM contract, since none are updated here (the PR only touches 3 files).

blocking: example/demo scripts that POST stream=true and read raw PCM

• fish_speech/speech_client.py, plus the same pattern in cosyvoice3/speech_client.py and ming_tts/openai_speech_client.py: they set payload["stream"] = True + response_format="pcm", then write resp.iter_bytes() to a .pcm file.
• qwen3_tts/tts_common.py (stream=True by default, consumed as raw PCM chunks) and the voxcpm2 / moss_tts_nano gradio demos (proxy raw PCM to a browser AudioWorklet).
• README curl pipes "stream": true straight into play -t raw ...: README.md (two examples, L89 and L529).

After this change all of these receive text/event-stream SSE where they expect raw PCM, producing garbage/silent audio. Please update them to pass stream_format="audio" for the raw path.

blocking: docs are now inaccurate

• docs/serving/speech_api.md L125 still documents stream | bool | false | Stream raw PCM chunks as they are decoded. After this PR, stream=true streams SSE, not raw PCM. The table (and any raw-stream snippets) should be updated, with raw PCM now described under stream_format="audio".

not affected: e2e CI. The streaming TTS e2e tests route through send_audio_speech_request, which uses stream only to choose the client-side with_streaming_response read mode and never forwards it to the server, so they exercise the unchanged non-streaming path. The PR's own updated unit tests in test_serving_speech.py cover the new SSE behavior, so unit/lint CI stays green.

It would also help to note in the PR description that this intentionally reverses the stream=True -> raw-audio contract that #4490 introduced (its test_stream_true_prefers_raw_audio_over_sse asserted exactly that, and is renamed/inverted here).

Minor wording/docstring notes left inline.

[linyueqian]

blocking (root cause): this block is where stream=True switches from raw-audio to SSE. The logic is correct and this is the intended Option A default. What is missing is the matching update to the in-repo consumers that still expect stream=True to return raw PCM. They are not in this PR's diff (so GitHub can't offer one-click suggestions on them); the exact edits are below.

Example / demo clients — add stream_format="audio" next to stream:

fish_speech/speech_client.py (identical in cosyvoice3/speech_client.py and ming_tts/openai_speech_client.py):

    if args.stream:
        payload["stream"] = True
        payload["stream_format"] = "audio"  # keep raw PCM streaming
        payload["response_format"] = "pcm"

qwen3_tts/tts_common.py:

    payload: dict = {
        "input": text.strip(),
        "response_format": "pcm" if stream else response_format,
        "stream": stream,
    }
    if stream:
        payload["stream_format"] = "audio"

voxcpm2/gradio_demo.py and moss_tts_nano/gradio_demo.py: add "stream_format": "audio" to the streaming payload dict the same way.

README (examples/online_serving/text_to_speech/README.md, both curl blocks, L89 and L529):

         "stream": true,
+        "stream_format": "audio",
         "response_format": "pcm"

Docs (docs/serving/speech_api.md L125) — the stream row is now inaccurate:

-| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
+| `stream` | bool | false | When true, stream OpenAI `speech.audio.*` SSE events (requires `response_format="pcm"` or `"wav"`). For raw PCM/WAV byte streaming, set `stream_format="audio"`. |
+| `stream_format` | string | null | `"audio"` = raw PCM/WAV bytes; `"sse"` (or `stream=true`) = OpenAI `speech.audio.*` SSE events. |

[linyueqian]

nit: the two lines just below this (Each Code2Wav chunk is yielded as raw audio bytes ... / For WAV format, a header with placeholder size values is emitted first.) now describe only the stream_format='audio' path, not the new stream=True SSE default. Worth scoping them to the raw-audio case so the docstring isn't read as applying to SSE.

[linyueqian]

nit: Omit for non-streaming, or use stream=true to stream SSE events reads as slightly self-contradictory, since omitting stream_format no longer implies non-streaming once stream=true. Minor wording.

[syd520zy]

I updated the PR to address the review feedback around the stream=True default behavior change.

Changes made:

• Added stream_format="audio" to the in-repo raw PCM/WAV streaming clients and demos that still consume /v1/audio/speech as raw bytes:

  • Fish Speech / CosyVoice3 / Ming CLI clients
  • Qwen3-TTS shared tts_common.py
  • VoxCPM2 and MOSS-TTS-Nano Gradio demos
  • GLM-TTS client and Ming curl script paths that also write raw PCM output

• Updated docs/examples so raw PCM streaming is now documented as:

  • stream=true
  • stream_format="audio"
  • response_format="pcm"

• Updated docs/serving/speech_api.md to document the new default:

  • stream=true now returns OpenAI speech.audio.* SSE events
  • stream_format="audio" explicitly opts into raw PCM/WAV byte streaming

• Scoped the serving_speech.py docstring so the Code2Wav raw-byte/WAV-header wording only applies to the raw audio path.

• Clarified the stream_format field description to avoid implying that omitting stream_format always means non-streaming.

I also updated the raw streaming e2e request configs to include stream_format="audio". As noted in the review, these may not be required for the current CI behavior, but semantically they should match the API contract after this PR: tests that expect raw audio bytes should explicitly opt into the raw audio stream format.

Note: this intentionally changes the compatibility behavior introduced by #4490, where stream=True preferred raw audio. Raw audio streaming remains available through stream_format="audio".

[linyueqian]

Re-review (round 2). The contract flip itself is correct and the round-1 comments are all addressed: the audio.py is_*_stream() truth table is sound (verified all six stream x stream_format combinations), the word_timestamps guard is now hoisted to is_streaming() so it also covers the SSE path, and the new unit/e2e coverage is solid. Forwarding stream_format in runtime.py also means the e2e streaming tests now actually exercise the raw-audio path instead of silently running non-streaming.

Two things still block merge.

[blocking] Merge conflict against current main. main has moved past this branch's last merge and renamed tests/e2e/online_serving/test_voxtral_tts.py to test_voxtral_tts_expansion.py, so this PR's edit to the old path is a modify/delete conflict (that is the current CONFLICTING state). See the inline note for the resolution.

[important] The Option A migration is incomplete. The description says consumers, docs, and benchmarks were all updated, but a few raw-audio consumers still send stream=true without stream_format="audio" and will now receive SSE instead of raw bytes:

• recipes/Qwen/Qwen3-TTS.md L150-152: stream:true + pcm piped into play -t raw (and the prose at L166). Now feeds SSE into the player.
• recipes/mistralai/Voxtral-TTS.md L82-83: stream:true + pcm written to voxtral_stream.pcm, then ffmpeg -f s16le. SSE text lands in the .pcm.
• recipes/OpenMOSS/MOSS-TTS.md L99-109: stream:true + wav written to output.wav. SSE text lands in the .wav.
• benchmarks/fish-speech/bench_speaker_cache.py L203-204 and L242-243: posts stream=True, response_format=pcm over aiohttp expecting raw PCM. benchmarks/patch/patch.py was fixed, but this standalone script was missed.

Each just needs "stream_format": "audio" added next to stream.

[suggestion] tests/e2e/online_serving/test_indextts2.py L77-82: the stream=True streaming test has no stream_format. It will not fail CI (the send_audio_speech_request helper never forwards stream to the server, so it runs non-streaming), but it no longer exercises streaming and is inconsistent with the e2e tests this PR did update. Worth adding "stream_format": "audio" for parity.

Non-blocking: diffusion-mode TTS (_create_diffusion_speech) ignores streaming entirely, but that is pre-existing and out of scope here.

[linyueqian]

[blocking] main renamed this file to test_voxtral_tts_expansion.py after this branch's last merge, so this edit is now a modify/delete conflict (the current CONFLICTING state). When you merge main, drop this file and instead add "stream_format": "audio" to test_speech_english_streaming in test_voxtral_tts_expansion.py -- its streaming payload still has stream=True, response_format=pcm with no stream_format.

[syd520zy]

Re-review (round 2). The contract flip itself is correct and the round-1 comments are all addressed: the audio.py is_*_stream() truth table is sound (verified all six stream x stream_format combinations), the word_timestamps guard is now hoisted to is_streaming() so it also covers the SSE path, and the new unit/e2e coverage is solid. Forwarding stream_format in runtime.py also means the e2e streaming tests now actually exercise the raw-audio path instead of silently running non-streaming.

Two things still block merge.

[blocking] Merge conflict against current main. main has moved past this branch's last merge and renamed tests/e2e/online_serving/test_voxtral_tts.py to test_voxtral_tts_expansion.py, so this PR's edit to the old path is a modify/delete conflict (that is the current CONFLICTING state). See the inline note for the resolution.

[important] The Option A migration is incomplete. The description says consumers, docs, and benchmarks were all updated, but a few raw-audio consumers still send stream=true without stream_format="audio" and will now receive SSE instead of raw bytes:

• recipes/Qwen/Qwen3-TTS.md L150-152: stream:true + pcm piped into play -t raw (and the prose at L166). Now feeds SSE into the player.
• recipes/mistralai/Voxtral-TTS.md L82-83: stream:true + pcm written to voxtral_stream.pcm, then ffmpeg -f s16le. SSE text lands in the .pcm.
• recipes/OpenMOSS/MOSS-TTS.md L99-109: stream:true + wav written to output.wav. SSE text lands in the .wav.
• benchmarks/fish-speech/bench_speaker_cache.py L203-204 and L242-243: posts stream=True, response_format=pcm over aiohttp expecting raw PCM. benchmarks/patch/patch.py was fixed, but this standalone script was missed.

Each just needs "stream_format": "audio" added next to stream.

[suggestion] tests/e2e/online_serving/test_indextts2.py L77-82: the stream=True streaming test has no stream_format. It will not fail CI (the send_audio_speech_request helper never forwards stream to the server, so it runs non-streaming), but it no longer exercises streaming and is inconsistent with the e2e tests this PR did update. Worth adding "stream_format": "audio" for parity.

Non-blocking: diffusion-mode TTS (_create_diffusion_speech) ignores streaming entirely, but that is pre-existing and out of scope here.

Updated the remaining raw-audio streaming consumers from the round-2 review.

Changes made:

• Added stream_format="audio" to the raw streaming examples in:

  • recipes/Qwen/Qwen3-TTS.md
  • recipes/mistralai/Voxtral-TTS.md
  • recipes/OpenMOSS/MOSS-TTS.md

• Added stream_format="audio" to both raw PCM payloads in:

  • benchmarks/fish-speech/bench_speaker_cache.py

• Updated the Qwen recipe note to describe the new raw-audio streaming request shape:

  • stream=true
  • stream_format="audio"
  • response_format="pcm"

I also did another pass for the same raw-byte streaming pattern and fixed two additional affected examples:

• recipes/OpenBMB/VoxCPM2.md
• examples/online_serving/text_to_speech/README.md

These examples pipe or write raw PCM/WAV bytes, so they should explicitly opt into the raw audio stream format now that stream=true defaults to SSE.

Validation run:

• git diff --check
• python -m py_compile benchmarks/fish-speech/bench_speaker_cache.py
• pre-commit run ruff-format --files benchmarks/fish-speech/bench_speaker_cache.py
• pre-commit run ruff-check --files benchmarks/fish-speech/bench_speaker_cache.py

For the two outdated-file items:

• tests/e2e/online_serving/test_voxtral_tts.py: this was resolved after merging current main. The old file was dropped, and the raw streaming case in tests/e2e/online_serving/test_voxtral_tts_expansion.py now includes stream_format="audio".

• tests/e2e/online_serving/test_indextts2.py: this was also updated. Its raw streaming request now includes stream_format="audio".

Could you please re-check these two items against the latest PR head?

[linyueqian]

Approving. All round-2 items are resolved: the voxtral merge conflict is fixed (old file dropped for upstream's rename, stream_format="audio" re-applied in test_voxtral_tts_expansion.py), and the remaining raw-audio consumers (recipes Qwen/Voxtral/MOSS/VoxCPM2, the MOSS-TTS-Nano block in the examples README, and bench_speaker_cache.py) plus test_indextts2.py now pass stream_format="audio".

Re-verified this pass: the is_*_stream() routing and the word_timestamps guard in serving_speech.py are unchanged and correct (all six stream x stream_format combinations check out), a full-tree sweep finds no remaining consumer that streams raw audio without stream_format, and the lightweight CI (DCO, build 3.11/3.12, docs, pre-commit) is green. The only caveat is the unrun full e2e suite already noted in the description; the e2e configs themselves are updated correctly.

[syd520zy]

Approving. All round-2 items are resolved: the voxtral merge conflict is fixed (old file dropped for upstream's rename, stream_format="audio" re-applied in test_voxtral_tts_expansion.py), and the remaining raw-audio consumers (recipes Qwen/Voxtral/MOSS/VoxCPM2, the MOSS-TTS-Nano block in the examples README, and bench_speaker_cache.py) plus test_indextts2.py now pass stream_format="audio".

Re-verified this pass: the is_*_stream() routing and the word_timestamps guard in serving_speech.py are unchanged and correct (all six stream x stream_format combinations check out), a full-tree sweep finds no remaining consumer that streams raw audio without stream_format, and the lightweight CI (DCO, build 3.11/3.12, docs, pre-commit) is green. The only caveat is the unrun full e2e suite already noted in the description; the e2e configs themselves are updated correctly.

The failing test was still mocking the old _coerce_sampling_params_streaming helper, but the current implementation now calls coerce_param_message_types from serving_speech.py.

I updated the test mock target to match the current production path. I also checked for other stale _coerce_sampling_params_streaming references and did not find any remaining usages.