Skip to content

fix(realtime): stop sending a second Gemini Live setup on follow-up session.update - #5

Closed
mubashir1osmani wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_fix_gemini_realtime_double_setup
Closed

fix(realtime): stop sending a second Gemini Live setup on follow-up session.update#5
mubashir1osmani wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_fix_gemini_realtime_double_setup

Conversation

@mubashir1osmani

@mubashir1osmani mubashir1osmani commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Relevant issues

Voice-bot operators on the proxy realtime websocket reported three symptoms with Gemini Live models: the assistant going silent after the first turn (bot greets, human replies, silence thereafter), random high and compounding per-turn latency, and intermittent 1011 errors

Linear ticket

N/A

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Running a dockerized proxy with a gemini-realtime model (gemini/gemini-3.1-flash-live-preview, mode: realtime) and driving it with a pipecat GA realtime client (which sends several session.updates while configuring the session, as every GA client does)

Before this change the backend closes immediately after setupComplete:

Gemini Realtime: Sending initial setup with tools to backend
Gemini Realtime: Forwarding session.update as follow-up setup
ERROR realtime_streaming.py: Connection closed in backend to client send messages
  - received 1007 (invalid frame payload data) Request contains an invalid argument.

The 1007 is reproducible against Google directly, independent of litellm: open a Live websocket, send setup, receive setupComplete, send a second setup, and the socket closes with 1007 Request contains an invalid argument

After this change the proxy drops the follow-up session.update (Ignoring session.update (setup already sent)) and a multi-turn conversation completes with the tool call firing. Per-turn time-to-first-audio is stable across turns (~600-1000ms, no compounding), and the session shows zero 1007/1011 errors

Type

🐛 Bug Fix

Changes

Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client message. The AI Studio Gemini realtime path forwarded every client session.update after the first as a follow-up setup, on the assumption that Gemini Live treats a follow-up setup as a full session replacement. It does not; a second setup closes the socket with a 1007. Because pipecat and other GA clients send multiple session.updates while configuring the session, the second one tore the session down before the first turn, which is what produced the silence, the reconnect/retry latency churn, and the 1011s

This drops subsequent session.updates once the initial setup has been sent, mirroring what the Vertex subclass already does for the same reason. Tools and instructions must ride on the first session.update before any conversation content, which is how the GA realtime services already drive the proxy. The previous identical-setup dedup was insufficient because a follow-up that differed at all (for example one that registered tools after connect) was still forwarded and still hit the 1007

Regression tests cover the plain follow-up drop, a follow-up that adds tools (the case the old dedup still forwarded), and the guardrail create_response=False warning path. They fail against the prior behavior (which returned a second setup message) and pass with the drop

@mubashir1osmani
fix(realtime): stop sending a second Gemini Live setup on follow-up s…
edb2153 
…ession.update

Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client
message; a second setup closes the socket with 1007 Request contains an invalid
argument. The AI Studio Gemini path forwarded every client session.update after
the first as a follow-up setup, and GA clients (pipecat) send several while
configuring the session, so the second one tore the session down before the
first turn. Callers saw silence after the first response, exponential per-turn
latency from reconnect/retry churn, and intermittent 1011 errors.

Drop subsequent session.updates instead of resending setup, matching what the
Vertex subclass already does. Tools and instructions must ride on the first
session.update before any conversation content.

Adds regression tests covering the plain follow-up, a follow-up that adds tools
(the case the previous identical-only dedup still forwarded), and the guardrail
create_response=False warning path.
@mubashir1osmani
fix(realtime): retry the backend open handshake instead of failing wi…
3c6a9e1 
…th 1011

The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs;
waiting longer never recovers a hung attempt, but a fresh attempt almost always
connects in ~1s. The proxy opened the backend websocket once with the default
open_timeout and no retry, so a single slow handshake surfaced to the caller as
a fatal 1011 internal error and dropped the call.

Bound each open attempt with a short open_timeout and retry; a bounded attempt
that already timed out spaces out the next try, so no backoff is needed.
Deterministic handshake-status rejections (auth/4xx) are not retried, and the
retry only ever wraps the open, never a live session.

Adds tests for retry-then-succeed, raise-after-max-attempts, and
no-retry-on-auth-failure.
@mubashir1osmani
mubashir1osmani force-pushed the litellm_fix_gemini_realtime_double_setup branch from cae2668 to 3c6a9e1 Compare June 27, 2026 05:40
@mubashir1osmani
fix(realtime): close guardrail bypass + surface handshake status; dro…
29ab8d7 
…p obsolete tests

Three review fixes on the Gemini Live realtime path.

Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once
the initial setup is sent the guardrail's automaticActivityDetection.disabled=true
can no longer be delivered as a follow-up session.update. With that follow-up now
dropped, the model's auto-response stayed enabled and a realtime_input_transcription
guardrail was bypassed (the model answered before the proxy could gate the turn).
Fold the disable into the one-and-only setup instead: the handler injects it into
the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it
into the deferred first setup. OpenAI sessions accept follow-up updates and are left
untouched.

Backend handshake status: the open-retry treated only InvalidStatusCode as
deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake,
so a 401/403 fell into the broad WebSocketException branch and was retried before
the caller closed the client with 1011 instead of the upstream status. Treat both as
non-retryable.

Obsolete tests: the four tests asserting a follow-up session.update is merged and
re-sent as a second setup asserted behavior that crashes Gemini Live with 1007
(verified directly against the API). Removed; the drop is covered by new regression
tests.
@mubashir1osmani

mubashir1osmani commented Jun 27, 2026

Copy link
Copy Markdown
Owner Author

Superseded by BerriAI#31519 (same branch, pushed to the BerriAI upstream against litellm_internal_staging).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mubashir1osmani