feat(vertex_ai): Vertex AI Gemini Live via unified /realtime endpoint

[ishaan-jaff]

Relevant issues

Closes: N/A

What this does

Adds Vertex AI Gemini Live (gemini-2.0-flash-live-001) support through LiteLLM's unified /realtime WebSocket endpoint. Clients use the standard OpenAI Realtime protocol — the proxy handles translation to/from Vertex AI's BidiGenerateContent format.

Tested end-to-end: text in/text out + voice in/voice out (mic → 16 kHz PCM16 → Vertex AI → 24 kHz PCM16 → speaker).

Changes

New: litellm/llms/vertex_ai/realtime/transformation.py — VertexAIRealtimeConfig

• Builds correct wss://{location}-aiplatform.googleapis.com/.../BidiGenerateContent URL
• OAuth2 Bearer token auth (not API key)
• Full model path: projects/{project}/locations/{location}/publishers/google/models/{model}
• Ignores session.update — Vertex AI only accepts one setup message per connection; sending a second causes a 1007 disconnect
• Audio MIME type includes sample rate: audio/pcm;rate=16000

realtime_api/main.py — adds vertex_ai branch that resolves the OAuth token via VertexBase._ensure_access_token_async and constructs VertexAIRealtimeConfig

llm_http_handler.py — auto-sends the provider's session setup message before bidirectional_forward() when requires_session_configuration() returns True (needed for Gemini/Vertex AI Live)

gemini/realtime/transformation.py — fixes two crashes:

• transform_response_done_event raised ValueError when turnComplete arrived before any audio (IDs were None); now generates UUIDs instead
• Same fix in transform_content_done_event
• Silently drop response.create (Vertex AI responds automatically)
• Handle conversation.item.create to extract actual user text

realtime_streaming.py — try/except guard in backend loop so a single malformed message doesn't kill the whole session; also fix pre-existing Pyright type annotation errors on _collect_tool_calls_from_response_done / _collect_user_input_from_backend_event

proxy_server.py — add missing import websockets.exceptions (caused NameError on every WebSocket close)

provider_endpoints_support.json — mark vertex_ai with "realtime": true

docs/my-website/docs/providers/vertex_realtime.md — usage docs with Python, Node.js, and OpenAI SDK examples

Pre-Submission checklist

• Added tests (tests/test_litellm/llms/vertex_ai/realtime/test_vertex_ai_realtime_transformation.py) — 6 unit tests covering URL construction, auth headers, session config format, and full text-in/text-out round trip with mocked WebSockets
• make test-unit passes

Type

• Bug Fix
• New Feature
• Refactoring
• Documentation

Changes

• litellm/llms/vertex_ai/realtime/transformation.py
• litellm/llms/vertex_ai/realtime/__init__.py
• litellm/realtime_api/main.py
• litellm/llms/custom_httpx/llm_http_handler.py
• litellm/llms/gemini/realtime/transformation.py
• litellm/litellm_core_utils/realtime_streaming.py
• litellm/proxy/proxy_server.py
• provider_endpoints_support.json
• docs/my-website/docs/providers/vertex_realtime.md

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 26, 2026 6:11am

[greptile-apps]

Greptile Summary

Adds Vertex AI Gemini Live (gemini-2.0-flash-live-001) support through LiteLLM's unified /realtime WebSocket endpoint. The new VertexAIRealtimeConfig extends GeminiRealtimeConfig with Vertex AI-specific WSS URL construction, OAuth2 Bearer token auth, and fully-qualified model paths. The PR also fixes several crashes in the base Gemini realtime transformer (UUID fallbacks for None IDs, proper model path normalization, graceful handling of unknown event types).

• New provider: VertexAIRealtimeConfig in litellm/llms/vertex_ai/realtime/transformation.py handles URL construction, auth headers, audio MIME types with sample rate, and session setup
• Crash fixes: GeminiRealtimeConfig no longer raises ValueError when turnComplete arrives before any audio content — generates UUID fallbacks instead
• Request translation fixes: response.create and unknown OpenAI event types are now silently dropped instead of being forwarded as raw JSON text to the model
• Guardrail integration concern: The new _send_to_backend helper routes guardrail messages through transform_realtime_request, but Vertex AI drops both session.update and response.create — meaning realtime guardrail warning messages and create_response=false injection are silently lost, making guardrails ineffective for Vertex AI sessions
• Tests: 6 mock-only unit tests covering URL construction, auth, session config, and end-to-end text round-trip

Confidence Score: 3/5

• Core Vertex AI realtime functionality is solid, but guardrail integration has a silent failure mode that should be addressed before merge.
• The new Vertex AI realtime feature is well-implemented with proper URL construction, auth, and session setup. However, the _send_to_backend refactor introduces a logic issue: guardrail warning messages and the create_response=false safety mechanism are silently dropped for Vertex AI because transform_realtime_request returns [] for session.update and response.create. This means realtime guardrails won't function correctly for Vertex AI sessions — blocked content won't generate a warning to the user. The crash fixes in GeminiRealtimeConfig and the model path normalization are good improvements.
• litellm/litellm_core_utils/realtime_streaming.py — guardrail messages silently dropped for Vertex AI via _send_to_backend

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/realtime/transformation.py	New VertexAIRealtimeConfig extending GeminiRealtimeConfig. Well-structured with proper URL construction, OAuth2 auth, and session setup. Minor: double JSON parse on non-session.update messages, inline imports in session_configuration_request.
litellm/litellm_core_utils/realtime_streaming.py	New _send_to_backend method routes guardrail messages through provider transform, but this causes guardrail warning messages and create_response=false injection to be silently dropped for Vertex AI, making realtime guardrails ineffective.
litellm/llms/gemini/realtime/transformation.py	Good fixes: UUID fallbacks for None IDs prevent crashes on early turnComplete; proper model path normalization for Vertex AI; response.create and unknown events now handled gracefully instead of forwarding raw JSON as text.
litellm/realtime_api/main.py	Vertex AI branch properly resolves OAuth token, project, and location before creating config. Health check branch added. Follows established patterns from other providers (azure, bedrock, xai).
litellm/llms/custom_httpx/llm_http_handler.py	Clean addition: auto-sends session setup message for providers that require it, then stores it on the RealTimeStreaming instance. No issues found.
tests/test_litellm/llms/vertex_ai/realtime/test_vertex_ai_realtime_transformation.py	6 well-structured unit tests covering URL construction, auth headers, session config format, and full text-in/text-out round trip with mocked WebSockets. All mock-based, no real network calls.

Sequence Diagram

sequenceDiagram
    participant Client as Client
    participant Proxy as LiteLLM Proxy
    participant Transform as VertexAIRealtimeConfig
    participant VertexAI as Vertex AI

    Client->>Proxy: WebSocket /realtime
    Proxy->>Transform: get_complete_url + validate_environment
    Proxy->>VertexAI: WSS connect
    Transform->>VertexAI: setup message with model path and VAD
    VertexAI-->>Transform: setupComplete
    Transform-->>Client: session.created

    Client->>Transform: conversation.item.create
    Transform->>VertexAI: realtime_input.text
    VertexAI-->>Transform: serverContent.modelTurn text
    Transform-->>Client: response.text.delta
    VertexAI-->>Transform: serverContent.generationComplete
    Transform-->>Client: response.text.done
    VertexAI-->>Transform: serverContent.turnComplete
    Transform-->>Client: response.done

    Client->>Transform: input_audio_buffer.append
    Transform->>VertexAI: realtime_input.audio PCM16
    VertexAI-->>Transform: serverContent.modelTurn inlineData
    Transform-->>Client: response.audio.delta

    Client->>Transform: session.update
    Note right of Transform: Silently dropped

Loading

_{Last reviewed commit: 58109ac}

[greptile-apps]

_{10 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Unused import and module-level instantiation

VertexBase is imported and instantiated as vertex_llm_base at module level but is never referenced anywhere in this file. The actual VertexBase usage happens in realtime_api/main.py which has its own instance. This creates an unnecessary object at import time.

Suggested change

	from litellm.llms.gemini.realtime.transformation import GeminiRealtimeConfig
	from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
	vertex_llm_base = VertexBase()
	from litellm.llms.gemini.realtime.transformation import GeminiRealtimeConfig

[greptile-apps]

Inline imports in method body

Per the CLAUDE.md style guide: "Avoid imports within methods — place all imports at the top of the file (module-level)." The imports of BidiGenerateContentSetup and GeminiResponseModalities should be moved to the top of the file.

# At the top of the file, add:
from litellm.types.llms.gemini import BidiGenerateContentSetup
from litellm.types.llms.vertex_ai import GeminiResponseModalities

Context Used: Context from dashboard - CLAUDE.md (source)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

Unrecognized message types sent as raw text to Gemini

The else branch sends the entire raw JSON string (e.g. {"type":"input_audio_buffer.commit","event_id":"..."}) as a text input to the model. For OpenAI Realtime events that have no Gemini equivalent (like input_audio_buffer.commit, input_audio_buffer.clear), this will cause the model to receive the raw JSON as a user text message, which is likely not the intended behavior.

Consider either explicitly handling known no-op events (returning []) or at minimum logging a warning, similar to how response.create and session.update are handled:

Suggested change

	else:
	# Pass through any other message types as raw text
	realtime_input_dict["text"] = message
	else:
	# Drop unrecognized OpenAI event types that have no Gemini equivalent
	return []

[greptile-apps]

Broad exception catch silently swallows all backend errors

While adding a try/except here prevents a single malformed message from killing the session (good), catching all Exception types means that serious errors (e.g. auth failures, protocol violations, or bugs in transformation code) will also be silently swallowed, with the loop continue-ing past them. Consider narrowing this to catch only expected transformation errors (like ValueError, KeyError, json.JSONDecodeError) so that unexpected failures still propagate and are visible.

[greptile-apps]

_realtime_health_check not updated for vertex_ai

The _realtime_health_check function (line 271) does not have a vertex_ai branch and will raise ValueError("Unsupported model: ...") if called for a Vertex AI realtime model. If health checks are run for deployed realtime models, this will fail at runtime. Consider adding a vertex_ai case or at least a graceful skip.

[ishaan-jaff]

@greptile-apps review

[greptile-apps]

Additional Comments (1)

litellm/llms/gemini/realtime/transformation.py, line 778
Generated UUIDs not propagated to next call

When generationComplete arrives before any content deltas, current_output_item_id and current_response_id are None. The fix in transform_content_done_event (line 480-483) generates UUIDs locally, but handle_openai_modality_event passes the original None values to return_additional_content_done_events (line 772-774), which still raises ValueError at line 517-520.

The UUIDs generated inside transform_content_done_event are local to that method and don't propagate back. Either:

• Apply the same UUID-generation fallback in return_additional_content_done_events (consistent with what was done for transform_content_done_event and transform_response_done_event)
• Or extract the generated IDs from transformed_content_done_event and pass them through:

            transformed_content_done_event = self.transform_content_done_event(
                current_output_item_id=current_output_item_id,
                current_response_id=current_response_id,
                delta_chunks=current_delta_chunks,
                delta_type=delta_type,
            )
            returned_message = [transformed_content_done_event]

            # Use the IDs from the done event (which may have been generated
            # as UUID fallbacks inside transform_content_done_event).
            resolved_item_id = transformed_content_done_event.get("item_id") or current_output_item_id
            resolved_response_id = transformed_content_done_event.get("response_id") or current_response_id

            additional_items = self.return_additional_content_done_events(
                current_output_item_id=resolved_item_id,
                current_response_id=resolved_response_id,
                delta_done_event=transformed_content_done_event,
                delta_type=delta_type,
            )
            returned_message.extend(additional_items)

[ishaan-jaff]

@greptile-apps review

[greptile-apps]

Additional Comments (2)

litellm/litellm_core_utils/realtime_streaming.py, line 350
Raw OpenAI-format messages sent to non-OpenAI backends

When guardrails are active, this block sends an OpenAI-format session.update JSON directly to self.backend_ws (lines 338–350), and later sends response.create (line 367–369). For Vertex AI (and Gemini), the backend expects its own protocol — not OpenAI Realtime events. These raw messages bypass the provider_config.transform_realtime_request() layer entirely.

For Vertex AI specifically, sending a raw {"type": "session.update", ...} to the WebSocket will either be silently ignored or cause a protocol error. Similarly, response.create is not a valid Vertex AI message type (Vertex AI responds automatically after each turn).

Consider routing these guardrail messages through self.provider_config.transform_realtime_request() before sending, or adding a provider-aware conditional that skips these sends for providers that don't support mid-session reconfiguration.

---

litellm/llms/gemini/realtime/transformation.py, line 317
str.strip() strips characters, not substrings — may corrupt Vertex AI model paths

_model.strip("models/") strips any of the individual characters m, o, d, e, l, s, / from both ends of the string. For Vertex AI, _model is now a fully-qualified path like "projects/my-proj/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-001". It currently works only because the path starts with p and ends with 1 (neither in the strip set), but a model name like "gemini-2.0-flash-live-demo" would have its trailing o stripped.

Consider using removeprefix instead:

            session["model"] = _model.removeprefix(
                "models/"
            )  # keep it consistent with how openai returns the model name

[ishaan-jaff]

@greptile-apps review

[ishaan-jaff]

@greptile-apps review

[greptile-apps]

Additional Comments (2)

litellm/llms/gemini/realtime/transformation.py, line 321
Vertex AI model path leaks into session.created

For Vertex AI, _model will be the fully-qualified resource path (e.g., projects/my-proj/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-001). This doesn't start with "models/", so the startswith("models/") check on line 319 won't strip it. The full Vertex AI resource path will be returned as session["model"] in the session.created event sent to the client, which breaks the OpenAI-compatible contract.

Consider also stripping the Vertex AI resource prefix, e.g.:

        if _model is not None and isinstance(_model, str):
            # Strip "models/" or Vertex AI resource prefix if present
            # to match OpenAI model name format.
            if _model.startswith("models/"):
                _model = _model[len("models/"):]
            elif "/publishers/google/models/" in _model:
                _model = _model.split("/publishers/google/models/")[-1]
            session["model"] = _model

---

litellm/litellm_core_utils/realtime_streaming.py, line 313
Guardrail sends raw OpenAI events to Vertex AI backend

run_realtime_guardrails sends response.cancel (line 298) and response.create (line 300-313) directly via self.backend_ws.send() rather than the new self._send_to_backend() helper. For Vertex AI, these raw OpenAI-format messages will be sent untransformed to the Vertex AI WebSocket, which doesn't understand them. While both event types would be silently dropped by the Gemini transform (returning []), this means the guardrail block+warning flow effectively becomes a no-op for Vertex AI — the model won't speak the warning and won't be cancelled.

Consider using self._send_to_backend() here for consistency, and/or adding a Vertex AI-specific mechanism to handle guardrail-triggered responses.

[ishaan-jaff]

@greptile-apps review

[greptile-apps]

Additional Comments (2)

litellm/llms/vertex_ai/realtime/transformation.py, line 162
Double JSON parse, first without error handling

json.loads(message) on line 155 will raise a raw json.JSONDecodeError if the message is malformed, bypassing the parent class's error handling that wraps it in a ValueError with a helpful message. Additionally, the message is parsed a second time inside super().transform_realtime_request() (line 221 of gemini/realtime/transformation.py), which is wasteful.

Consider guarding the parse or checking the type after the parent has parsed:

        try:
            json_message = json.loads(message)
        except json.JSONDecodeError:
            # Fall through to parent which has proper error handling
            return super().transform_realtime_request(
                message, model, session_configuration_request
            )
        if json_message.get("type") == "session.update":
            # Do not forward as a second setup — Vertex AI rejects it.
            return []

        return super().transform_realtime_request(
            message, model, session_configuration_request
        )

---

litellm/realtime_api/main.py, line 248
access_token may be None at runtime

_ensure_access_token_async can return (None, project_id) if credential resolution fails (e.g. no credentials file, no metadata server). VertexAIRealtimeConfig.__init__ declares access_token: str, so passing None would violate the type contract and result in Authorization: Bearer None being sent to Vertex AI — which will fail with a confusing auth error at WebSocket connect time rather than a clear error here.

Consider adding a guard:

if not access_token:
    raise ValueError(
        "Could not resolve Vertex AI access token. "
        "Set GOOGLE_APPLICATION_CREDENTIALS or run inside a GCP environment."
    )

[greptile-apps]

Additional Comments (3)

litellm/litellm_core_utils/realtime_streaming.py, line 313
Guardrail warning messages silently dropped for Vertex AI

When a guardrail blocks content, _send_to_backend sends both response.cancel and response.create through transform_realtime_request. For Vertex AI (via VertexAIRealtimeConfig → GeminiRealtimeConfig), response.create explicitly returns [] (line 244 of gemini/realtime/transformation.py) and response.cancel falls through to the else branch which also returns []. This means the warning speech message ("Say exactly and only: ...") will be silently dropped — the user will never hear or see the guardrail's block message.

The guardrail correctly returns True (blocked), so the original content won't be forwarded, but the replacement warning is lost. Consider either:

1. Having VertexAIRealtimeConfig.transform_realtime_request translate response.create into an equivalent Gemini realtime_input.text message, or
2. Sending the guardrail warning directly to the client as a synthesized response.text.delta / response.done event without going through the backend at all.

---

litellm/litellm_core_utils/realtime_streaming.py, line 367
Guardrail create_response=false injection has no effect for Vertex AI

When realtime guardrails are registered, this code sends session.update with create_response: false to disable auto-response. However, for Vertex AI, _send_to_backend routes this through VertexAIRealtimeConfig.transform_realtime_request, which returns [] for all session.update messages — so this safety mechanism is silently dropped.

Since Vertex AI has server-side VAD enabled and auto-responds after each turn, guardrails cannot actually prevent the LLM from responding before the guardrail check completes. This makes realtime guardrails ineffective for Vertex AI sessions. Consider documenting this limitation or implementing an alternative mechanism (e.g., client-side buffering).

---

litellm/llms/vertex_ai/realtime/transformation.py, line 159
Double JSON parse on every session.update message

transform_realtime_request parses message with json.loads(message) at line 152, and if the type is NOT session.update, it calls super().transform_realtime_request(message, ...) which parses the same JSON string again at GeminiRealtimeConfig line 221. For session.update this is fine (returns early), but for all other message types the raw JSON is parsed twice.

Consider passing the already-parsed json_message dict into the parent class, or restructuring so parsing happens once. This is a minor performance concern for high-frequency audio messages.

    def transform_realtime_request(
        self,
        message: str,
        model: str,
        session_configuration_request: Optional[str] = None,
    ) -> List[str]:
        """
        Translate OpenAI realtime client messages to Vertex AI format.

        ``session.update`` is intentionally ignored (returns []) because
        Vertex AI only accepts a single ``setup`` message at the start of
        the connection — sending a second one causes a 1007 close error.
        The initial setup (sent automatically before bidirectional_forward)
        already includes AUDIO modality and server VAD, so there is nothing
        more to configure.
        """
        try:
            json_message = json.loads(message)
        except json.JSONDecodeError:
            return super().transform_realtime_request(
                message, model, session_configuration_request
            )
        if json_message.get("type") == "session.update":
            # Do not forward as a second setup — Vertex AI rejects it.
            return []

        return super().transform_realtime_request(
            message, model, session_configuration_request
        )