fix(proxy): decode bytes and pass-through SSE for Google-native streamGenerateContent (#27444)

[Anai-Guo]

Summary

Fixes #27444 — /v1beta/models/{model}:streamGenerateContent?alt=sse produced corrupted output that no JSON parser (and no @google/genai client) can read:

data: b'data: {"candidates": [...]}'

The bug is in async_data_generator in litellm/proxy/proxy_server.py. The Google-native streaming iterator (AsyncGoogleGenAIGenerateContentStreamingIterator) yields raw SSE bytes from httpx.aiter_bytes(). Those bytes flow into async_data_generator, where yield f\"data: {chunk}\n\n\" calls str(bytes_obj), which renders the Python b'...' repr instead of the decoded string. The original payload is also already SSE-formatted (starts with data:), so wrapping it again would double-prefix.

OpenAI-format (/v1/chat/completions) is unaffected because chunks there are BaseModel instances and take the JSON-serializing branch.

Fix

Add a bytes branch before the existing str.startswith(\"data: \") (error-detection) branch:

1. Decode UTF-8 (errors="replace" so a malformed byte never tears the stream).
2. If the decoded payload already looks like an SSE event (data:, event:, or : comment line), pass it through unchanged — adding a trailing \n\n only if the upstream chunk didn't include one.
3. Otherwise fall through to the existing data: {chunk}\n\n wrapping path.

The existing OpenAI error-detection branch (elif isinstance(chunk, str) and chunk.startswith(\"data: \")) is preserved unchanged.

Test plan

• Reproduction from Bug: Google-native streamGenerateContent wraps each SSE event in Python b'...' bytes literal #27444 (curl to :streamGenerateContent?alt=sse) now returns clean SSE; @google/genai SDK parses it without error.
• OpenAI-format streaming (/v1/chat/completions) still works (BaseModel branch unchanged).
• Existing proxy streaming tests still pass.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR fixes corrupted SSE output for Google-native streaming (/v1beta/.../streamGenerateContent?alt=sse) by adding a bytes decode branch in async_data_generator so raw bytes from httpx.aiter_bytes() are no longer rendered as Python b'...' literals.

• Decodes bytes to UTF-8 (with errors=\"replace\") and passes already-formatted SSE events through unchanged to avoid a double data: prefix.
• The fix is inserted before the existing str.startswith(\"data: \") error-detection branch, preserving all existing OpenAI-format streaming behavior.
• No automated tests were added for the new bytes decode path, leaving the fix unverified beyond manual curl checks mentioned (but not confirmed) in the PR description.

Confidence Score: 4/5

The change is narrowly scoped to a new bytes branch that only activates for raw-byte chunks; all existing OpenAI-format streaming paths are unaffected and the fallthrough logic is correct.

The decode logic is sound and the continue correctly prevents double-yielding. The remaining gap is the `endswith("

")` check that would miss CRLF-terminated SSE events and produce a spurious extra event boundary, though most clients tolerate this gracefully. No automated tests cover the new path.

litellm/proxy/proxy_server.py — specifically the bytes branch in async_data_generator and the SSE terminator check.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Adds a bytes branch in async_data_generator to decode raw SSE bytes from Google-native streaming iterators and pass pre-formatted SSE through unchanged; logic is correct for the happy path but the endswith("\n\n") terminator check doesn't cover CRLF-terminated events.

_{Reviews (2): Last reviewed commit: "chore(lint): silence PLR0915 on async_da..." | Re-trigger Greptile}

[greptile-apps]

Partial chunk may corrupt SSE framing for large payloads

httpx.aiter_bytes() splits by TCP/buffer boundaries, not by SSE event boundaries. If a large Gemini SSE event is split across two aiter_bytes() calls, the first half (e.g. data: {"candidates":... — incomplete JSON) starts with data: and is yielded immediately with \n\n appended, making the client parse it as a complete but malformed event. The continuation bytes (e.g. ...}\n\n) don't start with an SSE prefix, fall through to yield f"data: {chunk}\n\n", and are double-wrapped. For typical small Gemini payloads this won't trigger, but it can occur for responses with long text fields.

[greptile-apps]

Test plan items unchecked — fix is unverified

The PR description lists three test plan items (curl reproduction, OpenAI-format regression, existing proxy streaming tests) and all remain unchecked. Per the team's custom standard, a PR claiming to fix a reported issue should include evidence that the issue is resolved, such as passing tests or concrete verification steps. No automated tests for the bytes decode path appear to have been added.

Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[shekohex]

Status update on this? @Anai-Guo

[Anai-Guo]

Hi @shekohex — PR is ready from my side: tests added, Codecov is green, and Greptile already left a positive automated review. Just waiting on a maintainer pass. Happy to address any feedback once someone takes a look.

[Perfecto23]

@ishaan-berri @Sameerlite @yuneng-berri @mateo-berri @krrish-berri-2 could this get a maintainer pass soon?

This is a real production regression for Google-native streaming clients. After #26914, /v1beta/models/{model}:streamGenerateContent?alt=sse now routes through the generic proxy SSE wrapper, so Gemini's already-formatted SSE chunks are emitted as data: b'data: {...}'. That breaks clients expecting native Gemini SSE, including gemini-cli when routed through LiteLLM.

I verified this still affects 1.84.0, and 1.85.0 does not include this fix. The current workaround is to pin back to 1.83.14, which is not a good long-term answer.

This PR looks scoped to the right fix: decode byte chunks and pass through already-formatted SSE. Checks are green; it seems to be waiting on maintainer review / conflict resolution. Could you please prioritize merging or releasing this fix?

[Sameerlite]

@greptileai re review

[Sameerlite]

@Perfecto23 thanks for the PR, can you resolve the conflict?

[Perfecto23]

@Sameerlite thanks, but I'm not the PR author. I'm just an affected user following this regression and trying to help get the fix landed.

@Anai-Guo could you take a look at the conflict resolution when you get a chance? It looks like that is currently blocking maintainer review / merge.

[Sameerlite]

#28213
this got merged

[Perfecto23]

@Sameerlite thanks ♥️