feat: add Gemini Live API (BidiGenerateContent) realtime support - #4988
Open
Shaik-Sirajuddin wants to merge 4 commits into
Open
feat: add Gemini Live API (BidiGenerateContent) realtime support#4988Shaik-Sirajuddin wants to merge 4 commits into
Shaik-Sirajuddin wants to merge 4 commits into
Conversation
Implements schemas.RealtimeProvider for Gemini, closing maximhq#3736. Current Gemini Live models are audio-output-only (TEXT responseModalities rejected at setup), so T->T/S->T are achieved via the outputAudioTranscription side-channel alongside native T->S/S->S audio support. Also fixes a shared-transport gap where session.update omitting a model would fail providers whose wire protocol only learns the model from the session payload.
Contributor
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds Gemini Live realtime provider support, relay integration, tests, and docs, and updates upstream WebSocket handling to redact dial URLs in error paths. ChangesGemini realtime support
WebSocket URL redaction
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant BifrostRelay
participant GeminiProvider
participant GeminiLive
Client->>BifrostRelay: session.update / input_audio_buffer.append
BifrostRelay->>BifrostRelay: sanitize with connection model
BifrostRelay->>GeminiProvider: ToProviderRealtimeEvent(event)
GeminiProvider->>GeminiLive: setup / clientContent / realtimeInput
GeminiLive-->>GeminiProvider: serverContent / usageMetadata
GeminiProvider->>BifrostRelay: ToBifrostRealtimeEvent(frame)
BifrostRelay-->>Client: forward canonical realtime event
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+24
to
+29
| func (provider *GeminiProvider) RealtimeWebSocketURL(key schemas.Key, model string) string { | ||
| base := provider.networkConfig.BaseURL | ||
| base = strings.Replace(base, "https://", "wss://", 1) | ||
| base = strings.Replace(base, "http://", "ws://", 1) | ||
| base = strings.TrimSuffix(base, "/v1beta") | ||
| return base + "/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=" + url.QueryEscape(key.Value.GetValue()) |
Contributor
There was a problem hiding this comment.
The Gemini API key flows directly into PoolKey.Endpoint as part of the wss://...?key=<API_KEY> URL. Every other realtime provider (OpenAI, ElevenLabs, Azure) sends credentials via request headers, keeping secrets out of the pool key and connection identifiers. With Gemini's URL-based auth the key lives unmasked as a Go map key inside the pool and will appear in plaintext in access logs at any reverse-proxy or load balancer that logs request URLs including query parameters. The PR acknowledges this is required by Gemini's protocol, but no mitigations are applied — e.g. the pool key could use key.ID + a constant Gemini sentinel as the Endpoint value (stripping the query param), while keeping the full URL only in the actual dial call.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
transports/bifrost-http/handlers/webrtc_realtime.go (1)
846-868: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winExtract the duplicated finalize-turn-and-close sequence.
The new
len(providerEvent) == 0guard (872-896) duplicates the exactfinalizeRealtimeTurnHooksOnTransportError→closeWithErrorEventsequence already present in theerr != nilbranch above (846-868), differing only in the error message string. Extracting a small helper avoids having two places to keep in sync if the finalize/close logic changes.♻️ Proposed refactor
+func (r *webrtcRealtimeRelay) finalizeTurnAndClose(status int, code, msg string) { + if finalizeErr := finalizeRealtimeTurnHooksOnTransportError( + r.client, + r.bifrostCtx, + r.session, + r.providerKey, + r.model, + r.key, + status, + code, + msg, + ); finalizeErr != nil { + r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(finalizeErr)) + return + } + r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(newRealtimeWireBifrostError(status, code, msg))) +} + func (r *webrtcRealtimeRelay) handleDownstreamMessage(msg webrtc.DataChannelMessage) { ... if err != nil { if startsTurn { - if finalizeErr := finalizeRealtimeTurnHooksOnTransportError( - r.client, r.bifrostCtx, r.session, r.providerKey, r.model, r.key, - 400, "invalid_request_error", err.Error(), - ); finalizeErr != nil { - r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(finalizeErr)) - return - } - r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(newRealtimeWireBifrostError(400, "invalid_request_error", err.Error()))) - return + r.finalizeTurnAndClose(400, "invalid_request_error", err.Error()) + return } ... } ... if len(providerEvent) == 0 { if startsTurn { - if finalizeErr := finalizeRealtimeTurnHooksOnTransportError( - r.client, r.bifrostCtx, r.session, r.providerKey, r.model, r.key, - 400, "invalid_request_error", "provider dropped a turn-starting event", - ); finalizeErr != nil { - r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(finalizeErr)) - return - } - r.closeWithErrorEvent(newRealtimeTurnErrorEventPayload(newRealtimeWireBifrostError(400, "invalid_request_error", "provider dropped a turn-starting event"))) + r.finalizeTurnAndClose(400, "invalid_request_error", "provider dropped a turn-starting event") } return }Also applies to: 872-896
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@transports/bifrost-http/handlers/webrtc_realtime.go` around lines 846 - 868, The error-handling path in the WebRTC realtime handler duplicates the same finalize-and-close flow in both the existing err != nil branch and the new len(providerEvent) == 0 guard. Extract that repeated finalizeRealtimeTurnHooksOnTransportError → closeWithErrorEvent sequence into a small helper in webrtc_realtime.go, and have both branches call it with the appropriate error message/context so the logic stays in sync.transports/bifrost-http/handlers/wsrealtime.go (1)
379-398: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winGemini realtime needs a setup-first relay
This path only translates an existingsession.updateinto Geminisetup; it never injects one. If the client starts withconversation.item.createorinput_audio_buffer.append, the first upstream message won’t besetup, and Gemini can reject the connection. Add an initial Geminisession.update/setupbefore forwarding any other event, or fail fast when the client doesn’t send one.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@transports/bifrost-http/handlers/wsrealtime.go` around lines 379 - 398, The realtime relay in wsrealtime.go currently only maps an incoming session.update to Gemini setup via provider.ToProviderRealtimeEvent and does not ensure a setup message is sent first. Update the realtime startup flow around startsTurn/startRealtimeTurnHooks so Gemini receives an initial session.update/setup before any conversation.item.create or input_audio_buffer.append events are forwarded, or otherwise reject the connection early if no setup has been provided. Use the existing session, provider, and provider.ToProviderRealtimeEvent path to locate where to inject or validate the first upstream event.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/gemini/realtime.go`:
- Around line 357-369: The RTEventSessionUpdate handling in gemini realtime
forwarding is passing Session.Tools directly into geminiSetup, but Gemini
expects its own tool format. Update the session update path in the realtime
event switch to translate bifrostEvent.Session.Tools through the Gemini tool
mapper before assigning setup.Tools, or clearly change the RealtimeSession.Tools
contract to be Gemini-native; use the geminiSetup and toGeminiModelResourceName
area as the anchor for the fix.
---
Outside diff comments:
In `@transports/bifrost-http/handlers/webrtc_realtime.go`:
- Around line 846-868: The error-handling path in the WebRTC realtime handler
duplicates the same finalize-and-close flow in both the existing err != nil
branch and the new len(providerEvent) == 0 guard. Extract that repeated
finalizeRealtimeTurnHooksOnTransportError → closeWithErrorEvent sequence into a
small helper in webrtc_realtime.go, and have both branches call it with the
appropriate error message/context so the logic stays in sync.
In `@transports/bifrost-http/handlers/wsrealtime.go`:
- Around line 379-398: The realtime relay in wsrealtime.go currently only maps
an incoming session.update to Gemini setup via provider.ToProviderRealtimeEvent
and does not ensure a setup message is sent first. Update the realtime startup
flow around startsTurn/startRealtimeTurnHooks so Gemini receives an initial
session.update/setup before any conversation.item.create or
input_audio_buffer.append events are forwarded, or otherwise reject the
connection early if no setup has been provided. Use the existing session,
provider, and provider.ToProviderRealtimeEvent path to locate where to inject or
validate the first upstream event.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 83573b1a-b72b-49b5-8759-0f0a8ddb7a78
📒 Files selected for processing (9)
core/internal/llmtests/realtime.gocore/providers/gemini/gemini_test.gocore/providers/gemini/realtime.gocore/providers/gemini/realtime_test.godocs/openapi/paths/inference/realtime.yamldocs/providers/supported-providers/gemini.mdxtransports/bifrost-http/handlers/realtime_turn_pipeline.gotransports/bifrost-http/handlers/webrtc_realtime.gotransports/bifrost-http/handlers/wsrealtime.go
- Redact query-string secrets (Gemini's API key) from dial-error messages in the websocket pool/connection layer, since Gemini authenticates via a URL query param instead of headers like every other realtime provider. - Concatenate multiple audio inlineData parts in a single serverContent message instead of keeping only the first. - Document the intentional WS-vs-WebRTC error-handling asymmetry for dropped turn-starting events (pre-existing convention, not something to unify here).
- Convert session.update tools through the existing Gemini tool mapper (convertBifrostToolsToGemini) instead of forwarding the canonical OpenAI-shaped tool array verbatim, which Gemini's wire protocol rejects. - Extract the duplicated finalize-turn-and-close sequence in webrtc_realtime.go into a shared helper. - Document the setup-must-be-first-frame limitation as a deliberately deferred gap (would require a RealtimeProvider interface change shared by all providers) rather than leaving it unexplained.
Contributor
Author
|
Gemini Studio api keys were used to conduct end to end tests , for additive changes |
Follow-up to the Greptile security finding — the earlier fix only redacted dial-error messages, but the raw key still lived in PoolKey.Endpoint itself: the Go map key held in memory for the pool's lifetime and stored on UpstreamConn for diagnostics. Adds PoolKey.DialURL: an optional field carrying the real (possibly credential-bearing) dial target, decoupled from Endpoint, which now stays a sanitized identity value via the new SanitizeEndpointForPoolKey helper. Only strips credential-shaped query param names (key/token/secret/auth), so OpenAI's ?model= and Azure's ?deployment= params — needed for correct per-model pool bucketing — are left untouched. Confirmed via the realtime path's existing Discard-only usage (never Return) that this can't cause a pool-key mismatch for reuse, and all existing pool/websocket/handler tests pass unmodified.
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
transports/bifrost-http/websocket/connection_redact_test.go (1)
57-105: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMissing malformed-URL test case for
SanitizeEndpointForPoolKey.
TestRedactURLForLogcovers the invalid-URL fallback, butTestSanitizeEndpointForPoolKey's table has no equivalent case for a malformedrawURL. Given the fail-open behavior flagged inconnection.go(returns raw input unchanged on parse error), a test case here would have caught that gap and should be added once/if the fail-open path is fixed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@transports/bifrost-http/websocket/connection_redact_test.go` around lines 57 - 105, Add a malformed-URL table case to TestSanitizeEndpointForPoolKey so the fallback behavior is covered alongside the existing secret-preservation cases. Use SanitizeEndpointForPoolKey with an invalid rawURL input and assert the current fail-open behavior (or the intended fixed behavior if connection.go changes) is explicitly tested, matching the pattern used by TestRedactURLForLog.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@transports/bifrost-http/websocket/connection_redact_test.go`:
- Around line 57-105: Add a malformed-URL table case to
TestSanitizeEndpointForPoolKey so the fallback behavior is covered alongside the
existing secret-preservation cases. Use SanitizeEndpointForPoolKey with an
invalid rawURL input and assert the current fail-open behavior (or the intended
fixed behavior if connection.go changes) is explicitly tested, matching the
pattern used by TestRedactURLForLog.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4f2e12ec-aeca-4b62-94ad-ce75b8be6074
📒 Files selected for processing (4)
transports/bifrost-http/handlers/wsrealtime.gotransports/bifrost-http/websocket/connection.gotransports/bifrost-http/websocket/connection_redact_test.gotransports/bifrost-http/websocket/pool.go
akshaydeo
force-pushed
the
dev
branch
2 times, most recently
from
July 14, 2026 17:22
bb4c251 to
5a52603
Compare
Contributor
Author
End-to-end verification (official
|
akshaydeo
force-pushed
the
dev
branch
3 times, most recently
from
August 13, 2026 09:47
244a01d to
ce1b2a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
schemas.RealtimeProvidersupport for Gemini Live (BidiGenerateContent), closing #3736.Details
outputAudioTranscriptionside-channel, since current Gemini Live models only supportAUDIOas a response modality (TEXTis rejected at setup).session.updateomittingmodelnow falls back to the connection's resolved model — needed because Gemini's wire protocol only learns the model from the session payload, unlike OpenAI/Azure which also carry it on the connect URL./v1/realtimewebsocket endpoint already routes to anyRealtimeProviderimplementation.Checklist
go test ./...)