feat(audio): stream OpenAI TTS through the proxy to cut time-to-first-audio by TheCodeWrangler · Pull Request #33976 · BerriAI/litellm

TheCodeWrangler · 2026-07-20T14:58:02Z

Relevant issues

Fixes #33974

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Captured end-to-end against a live proxy hitting the real OpenAI API (gpt-4o-mini-tts). The key metric is time-to-first-byte (TTFB) vs total: streaming means first audio arrives well before generation finishes

Plain request, no stream_format (the common case, matching OpenAI's with_streaming_response examples). Before, at 1ebf2a7 the proxy buffered the whole clip; after 7286b36 it streams incrementally, matching a direct call to OpenAI

# after, via litellm proxy, response_format=pcm
$ curl -sN -X POST http://localhost:4000/v1/audio/speech -H "Authorization: Bearer sk-1234" \
    -d '{"model":"gpt-4o-mini-tts","input":"<~250 char passage>","voice":"coral","response_format":"pcm"}' \
    -D - -o /dev/null -w '[timing] ttfb=%{time_starttransfer}s total=%{time_total}s'
content-type: audio/pcm
[timing] ttfb=0.553630s total=2.922757s

# same request straight to OpenAI (us.api.openai.com), for comparison
content-type: audio/pcm
[timing] ttfb=0.978970s total=4.748323s

stream_format="sse" request, gpt-4o-mini-tts; frames arrive incrementally as text/event-stream

$ curl -sN ... -d '{... ,"stream_format":"sse"}'   # per-frame arrival, relative to request start
+1.148s  frame#1  delta
+1.157s  frame#2  delta
+1.242s  frame#3  delta
+3.807s  frame#59 done
total frames: 60
content-type: text/event-stream; charset=utf-8

A model that ignores stream_format (tts-1) is not mislabeled; the proxy forwards the provider's content-type

$ curl -s ... -d '{"model":"tts-1", ... ,"stream_format":"sse"}' -D -
content-type: audio/mpeg

Type

🆕 New Feature

Changes

The proxy /v1/audio/speech awaited the full clip (HttpxBinaryResponseContent) before sending anything, so time-to-first-audio equaled full-generation time. OpenAI's /v1/audio/speech actually streams over chunked transfer for every request (verified: gpt-4o-mini-tts pcm returns first byte at ~0.55s of a ~2.9s clip), so a client reading incrementally got no benefit through litellm

The OpenAI handler can now open the upstream with with_streaming_response and return a SpeechStreamingResponse whose iterator forwards the provider bytes as they arrive. The proxy asks for this on every speech request and forwards the frames labeled with the provider's actual content-type: audio/* for a normal request, or text/event-stream when the caller sets stream_format="sse" (OpenAI's speech.audio.delta frames). There is no hardcoded model list and no payload difference from calling OpenAI directly; a model that ignores stream_format (e.g. tts-1) just streams a correctly-labeled audio clip. This applies to any openai-compatible provider routed through the OpenAI handler (hosted_vllm, etc); other providers keep returning a buffered response

The streaming toggle is an internal stream_audio flag set by the proxy, deliberately kept distinct from "stream" so it does not trip _is_streaming_request and skip cost tracking. litellm.speech()/aspeech() still return the buffered HttpxBinaryResponseContent by default (stream_audio defaults False), so the SDK contract is unchanged. SpeechStreamingResponse is recognized in the success-logging path so streaming TTS records response_cost from the input characters exactly like the buffered path, closing a budget-bypass gap

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

greptile-apps · 2026-07-20T15:04:49Z

Greptile Summary

This PR adds incremental streaming for the proxy's /v1/audio/speech endpoint so that time-to-first-audio matches calling OpenAI directly, instead of buffering the full clip before sending anything. The OpenAI handler now opens upstream requests with with_streaming_response and wraps the result in a new SpeechStreamingResponse type that carries the raw byte iterator and the provider's response headers; the proxy passes the stream directly to FastAPI's StreamingResponse with the provider's actual content-type (or falls back to application/octet-stream when absent).

New SpeechStreamingResponse NamedTuple (litellm/types/llms/openai.py) holds the async/sync byte iterator and provider headers; it is deliberately not a BaseModel so it is not mistaken for a chat response by existing routing logic.
Cost tracking gap closed: SpeechStreamingResponse is added to _is_recognized_call_type_for_logging, so _process_hidden_params_and_response_cost fires and computes cost from input characters before the stream is consumed — the iterator is never iterated at logging time.
SDK back-compat preserved: stream_audio defaults to False in litellm.speech()/aspeech(), so direct SDK callers continue to receive the buffered HttpxBinaryResponseContent; only the proxy sets stream_audio=True.

Confidence Score: 5/5

Safe to merge — the change is additive, the SDK contract is unchanged, non-OpenAI providers are unaffected, and cost tracking is correctly wired before the stream is consumed.

The streaming path is only activated by the proxy via stream_audio=True; litellm.speech() default is unchanged. The with_streaming_response context-manager lifecycle is handled correctly. Cost is computed from input characters before the iterator is touched, so budget enforcement is not bypassed. All added tests are mock-only and cover both content-type passthrough paths and the no-content-type fallback.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
litellm/types/llms/openai.py	Adds `SpeechStreamingResponse` NamedTuple with `stream_iterator` and `headers` fields; clean, minimal type definition that threads through the async/sync handler correctly
litellm/llms/openai/openai.py	Adds `async_audio_speech_streaming` and `audio_speech_streaming` that open upstream with `with_streaming_response`, read headers eagerly (load-bearing for content-type pass-through), and forward bytes via a generator whose `finally` closes the context manager on both happy and error paths
litellm/proxy/proxy_server.py	Sets `stream_audio=True` unconditionally for all speech requests (only reaches the OpenAI handler), then routes `SpeechStreamingResponse` directly into FastAPI `StreamingResponse` with provider's content-type; falls back to `application/octet-stream` when header is absent
litellm/litellm_core_utils/litellm_logging.py	Adds `SpeechStreamingResponse` to `_is_recognized_call_type_for_logging` so `_process_hidden_params_and_response_cost` runs and records cost from input characters before the stream is consumed — closing the budget-bypass gap correctly
litellm/main.py	Adds `stream_format` and `stream_audio` parameters to `speech()`; `stream_audio` is only forwarded to the OpenAI handler branch, not Azure/ElevenLabs/Vertex, preserving buffered behavior for all non-OpenAI providers
tests/test_litellm/litellm_core_utils/test_litellm_logging.py	New regression test proves `SpeechStreamingResponse` triggers cost logging without consuming the stream iterator, and that computed cost matches `input_cost_per_character x character_count`
tests/test_litellm/llms/openai/speech/test_openai_speech_streaming.py	Comprehensive mock-only tests covering async/sync streaming paths, error-path CM cleanup, content-type pass-through, `stream_audio` forwarding for openai-compatible providers, and SDK back-compat (buffered path unchanged)
tests/test_litellm/proxy/proxy_server/test_routes_audio.py	New proxy route tests cover SSE streaming, provider-ignored `stream_format` content-type pass-through, missing content-type fallback to `application/octet-stream`, and assertion that `stream_audio` (not `stream`) is set in the request data

_{Reviews (7): Last reviewed commit: bd339f6 | Re-trigger Greptile}

codecov · 2026-07-20T15:09:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

veria-ai · 2026-07-20T15:12:05Z

PR overview

This PR adds streaming support for OpenAI text-to-speech responses through the LiteLLM proxy to reduce time-to-first-audio. The changes affect the proxy server path that returns TTS audio as a streaming response.

One security issue remains open after one prior issue was addressed. The remaining concern is that TTS streaming requests can release parallel-request limit slots too early, allowing an authenticated client to hold many long-lived streams and exceed configured concurrency limits. This creates a concrete resource-exhaustion risk against proxy and provider connections until stream lifecycle cleanup is tied to completion, cancellation, or failure.

Open issues (1)

Medium: Parallel-request limit bypass — litellm/proxy/proxy_server.py:9498

Fixed/addressed: 1 · PR risk: 6/10

TheCodeWrangler · 2026-07-20T15:22:23Z

@greptileai

TheCodeWrangler · 2026-07-20T18:17:30Z

@greptileai

TheCodeWrangler · 2026-07-21T15:29:51Z

@greptileai

veria-ai · 2026-07-21T15:35:06Z

+        # returned (text/event-stream for speech.audio.delta frames, or audio/* when the
+        # provider ignored stream_format), so nothing is mislabeled.
+        if isinstance(response, SpeechStreamingResponse):
+            return StreamingResponse(


Medium: Parallel-request limit bypass

litellm.aspeech dispatches success callbacks before this iterator is consumed, and both parallel-request limiters release their max_parallel_requests slot from those callbacks. An authenticated client can therefore open many long-lived TTS streams after each request's headers arrive, exceeding its configured concurrency limit and consuming unbounded proxy and provider connections. Treat SpeechStreamingResponse as a streaming lifecycle and release the slot only after iterator completion, cancellation, or failure, while retaining the required TTS cost accounting.

…ith no Content-Type Greptile review of BerriAI#33976: the proxy's streaming media_type fell back to text/event-stream when the provider omitted Content-Type, which would mislabel raw audio bytes as SSE for openai-compatible backends that stream without the header. Fall back to application/octet-stream instead; the provider's header is still forwarded verbatim when present (OpenAI's text/event-stream for speech.audio.delta, or audio/* when stream_format is ignored). Adds a proxy test pinning the no-Content-Type fallback.

TheCodeWrangler · 2026-07-22T12:20:08Z

@greptileai please re-review the latest head d56fbc5799. Addressed the one finding from the prior review: the streaming media_type now falls back to application/octet-stream instead of text/event-stream when a provider streams without a Content-Type header, so raw audio bytes are never mislabeled as SSE. The provider's header is still forwarded verbatim when present. Added a proxy test pinning the fallback

…ith no Content-Type Greptile review of BerriAI#33976: the proxy's streaming media_type fell back to text/event-stream when the provider omitted Content-Type, which would mislabel raw audio bytes as SSE for openai-compatible backends that stream without the header. Fall back to application/octet-stream instead; the provider's header is still forwarded verbatim when present (OpenAI's text/event-stream for speech.audio.delta, or audio/* when stream_format is ignored). Adds a proxy test pinning the no-Content-Type fallback.

TheCodeWrangler · 2026-08-05T12:32:51Z

@mateo-berri would you be open to reviewing this when you get a chance? You own most of the recent audio/speech handler changes so you're the natural reviewer

It makes the proxy /v1/audio/speech forward the upstream stream incrementally instead of awaiting the full clip, so time-to-first-audio reflects first-chunk latency for voice/real-time TTS. Covers OpenAI and openai-compatible providers (hosted_vllm included); other providers keep buffering. Greptile confidence is 5/5

I originally targeted litellm_oss_daily_2026_07_20, which looks abandoned, so I just rebased onto litellm_internal_staging and it merges cleanly. The only red check is the repo-wide osv-scan (mcp dep, already tracked in #33591), unrelated to this change

…ith no Content-Type Greptile review of BerriAI#33976: the proxy's streaming media_type fell back to text/event-stream when the provider omitted Content-Type, which would mislabel raw audio bytes as SSE for openai-compatible backends that stream without the header. Fall back to application/octet-stream instead; the provider's header is still forwarded verbatim when present (OpenAI's text/event-stream for speech.audio.delta, or audio/* when stream_format is ignored). Adds a proxy test pinning the no-Content-Type fallback.

TheCodeWrangler · 2026-08-05T12:42:11Z

Rebased onto current litellm_internal_staging and re-targeted the PR base there; force-pushed a clean two-commit history. @greptileai please re-review the new head

codspeed-hq · 2026-08-05T13:25:25Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing TheCodeWrangler:audio_speech_streaming_sse (262b877) with litellm_internal_staging (0ca0fa2)}

…ith no Content-Type Greptile review of BerriAI#33976: the proxy's streaming media_type fell back to text/event-stream when the provider omitted Content-Type, which would mislabel raw audio bytes as SSE for openai-compatible backends that stream without the header. Fall back to application/octet-stream instead; the provider's header is still forwarded verbatim when present (OpenAI's text/event-stream for speech.audio.delta, or audio/* when stream_format is ignored). Adds a proxy test pinning the no-Content-Type fallback.

TheCodeWrangler · 2026-08-11T13:06:11Z

Friendly bump on this one. It has been open a few weeks now, is rebased onto litellm_internal_staging, merges cleanly, and CI is green aside from the repo-wide osv-scan (mcp dep, already tracked in #33591). Greptile confidence is 5/5

@mateo-berri you own most of the recent audio/speech handler work so you seem like the natural reviewer, though I am happy for anyone on the team to pick it up. It makes the proxy /v1/audio/speech forward the upstream stream incrementally instead of buffering the whole clip, which cuts time-to-first-audio for real-time and voice TTS on OpenAI and every openai-compatible provider (hosted_vllm included)

TheCodeWrangler · 2026-08-11T13:13:14Z

Rebased fresh onto the latest litellm_internal_staging, and it is green and mergeable. Small, isolated change: streams OpenAI TTS through the proxy so time-to-first-audio is the first chunk instead of full generation, fixing #33974. @yuneng-jiang could you take a look when you get a chance? The only red check is osv-scan, which flags the pre-existing h2 and js-yaml advisories in the base lockfiles, nothing from this PR touches dependencies

…-audio The proxy /v1/audio/speech awaited the full clip (HttpxBinaryResponseContent) before sending anything, so time-to-first-audio equaled full generation time even though OpenAI streams /v1/audio/speech over chunked transfer for every request. A client reading incrementally (with_streaming_response) got no benefit through litellm. The OpenAI handler can now open the upstream with with_streaming_response and return a SpeechStreamingResponse whose iterator forwards the provider bytes as they arrive. The proxy asks for this on every speech request and forwards the frames labeled with the provider's actual content-type: audio/* for a normal request, or text/event-stream when the caller sets stream_format="sse" (OpenAI's speech.audio.delta frames). No hardcoded model list and no payload difference from calling OpenAI directly; a model that ignores stream_format (e.g. tts-1) just streams a correctly-labeled audio clip. Applies to any openai-compatible provider routed through the OpenAI handler (hosted_vllm, etc); other providers keep returning a buffered response. The streaming toggle is an internal stream_audio flag set by the proxy, kept distinct from "stream" so it does not trip _is_streaming_request and skip cost tracking. litellm.speech()/aspeech() still return the buffered HttpxBinaryResponseContent by default (stream_audio defaults False), so the SDK contract is unchanged. SpeechStreamingResponse is recognized in the success-logging path so streaming TTS records response_cost from the input characters exactly like the buffered path, closing a budget-bypass gap. Fixes BerriAI#33974

…ith no Content-Type Greptile review of BerriAI#33976: the proxy's streaming media_type fell back to text/event-stream when the provider omitted Content-Type, which would mislabel raw audio bytes as SSE for openai-compatible backends that stream without the header. Fall back to application/octet-stream instead; the provider's header is still forwarded verbatim when present (OpenAI's text/event-stream for speech.audio.delta, or audio/* when stream_format is ignored). Adds a proxy test pinning the no-Content-Type fallback.

greptile-apps Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread litellm/main.py Outdated

Comment thread litellm/llms/openai/openai.py Outdated

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from 4722c78 to 2119812 Compare July 20, 2026 15:06

veria-ai Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread litellm/types/llms/openai.py Outdated

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch 6 times, most recently from 299557a to 5b9dba2 Compare July 20, 2026 15:21

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from 5b9dba2 to 7286b36 Compare July 20, 2026 18:16

TheCodeWrangler changed the title ~~feat(audio): stream OpenAI TTS via stream_format=sse to cut time-to-first-audio~~ feat(audio): stream OpenAI TTS through the proxy to cut time-to-first-audio Jul 20, 2026

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch 2 times, most recently from 2281aff to 11b49d9 Compare July 21, 2026 15:29

TheCodeWrangler requested a review from ryan-crabbe-berri as a code owner July 21, 2026 15:29

TheCodeWrangler requested a review from a team July 21, 2026 15:29

TheCodeWrangler changed the base branch from litellm_oss_daily_2026_07_17 to litellm_oss_daily_2026_07_20 July 21, 2026 15:29

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from 11b49d9 to 5e46b43 Compare July 21, 2026 15:32

greptile-apps Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread litellm/proxy/proxy_server.py Outdated

veria-ai Bot reviewed Jul 21, 2026

View reviewed changes

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from d56fbc5 to 12817b5 Compare August 5, 2026 12:30

TheCodeWrangler changed the base branch from litellm_oss_daily_2026_07_20 to litellm_internal_staging August 5, 2026 12:30

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from 12817b5 to c29dfce Compare August 5, 2026 12:41

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from c29dfce to 09c26ed Compare August 5, 2026 15:03

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from 09c26ed to bd339f6 Compare August 5, 2026 15:19

TheCodeWrangler closed this Aug 5, 2026

TheCodeWrangler reopened this Aug 5, 2026

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch 3 times, most recently from bd339f6 to 1bf0b6b Compare August 7, 2026 00:09

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch 2 times, most recently from 519dd32 to ec2f663 Compare August 11, 2026 13:05

TheCodeWrangler added 2 commits August 12, 2026 08:10

TheCodeWrangler force-pushed the audio_speech_streaming_sse branch from ec2f663 to 262b877 Compare August 12, 2026 13:19

Uh oh!

Uh oh!

Conversation

TheCodeWrangler commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Changes

Final Attestation

Uh oh!

greptile-apps Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

veria-ai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Open issues (1)

Uh oh!

TheCodeWrangler commented Jul 20, 2026

Uh oh!

TheCodeWrangler commented Jul 20, 2026

Uh oh!

TheCodeWrangler commented Jul 21, 2026

Uh oh!

Uh oh!

veria-ai Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

TheCodeWrangler commented Jul 22, 2026

Uh oh!

TheCodeWrangler commented Aug 5, 2026

Uh oh!

TheCodeWrangler commented Aug 5, 2026

Uh oh!

codspeed-hq Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

TheCodeWrangler commented Aug 11, 2026

Uh oh!

TheCodeWrangler commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

TheCodeWrangler commented Jul 20, 2026 •

edited

Loading

greptile-apps Bot commented Jul 20, 2026 •

edited

Loading

codecov Bot commented Jul 20, 2026 •

edited

Loading

veria-ai Bot commented Jul 20, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 5, 2026 •

edited

Loading