Repository navigation
feat(realtime): add Gemini Live voice support - #3315
Conversation
Serve Google's Gemini Live models over /v1/realtime using Gemini's own BidiGenerateContent protocol, alongside the existing OpenAI realtime path. Shared connect/preflight/account-gate logic is factored out of session.ts so both providers run the same authorization and billing gates. Adds gemini-2.5-flash-native-audio-preview-12-2025 and gemini-3.1-flash-live-preview to the catalogue, a strict usage normalizer for Gemini's usageMetadata blocks, and the playground call UI. Co-Authored-By: Claude <noreply@anthropic.com>
…-realtime-voice # Conflicts: # apps/playground/src/components/playground/realtime-page-client.tsx
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
WalkthroughAdds Gemini realtime model definitions, provider-specific gateway proxying, usage normalization and billing, account gating, a Gemini WebSocket session, and provider-aware playground voice-call support with validation and lifecycle tests. ChangesGemini realtime voice integration
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant VoiceCall as useVoiceCall
participant SessionRoute as realtime session route
participant Gateway as realtime server
participant GeminiSession as GeminiRealtimeProxySession
participant Google as Gemini Live
participant Billing as account and billing services
VoiceCall->>SessionRoute: request realtime session
SessionRoute->>Gateway: open WebSocket with provider and model
Gateway->>GeminiSession: create provider-specific session
GeminiSession->>Google: connect using BidiGenerateContent target
VoiceCall->>GeminiSession: send setup and audio input
GeminiSession->>Google: forward validated Gemini frames
Google-->>GeminiSession: content and usageMetadata frames
GeminiSession->>Billing: normalize usage and persist stage billing
GeminiSession-->>VoiceCall: forward content, transcripts, and terminal events
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf8eca8a39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (this.finalized) { | ||
| return; | ||
| } | ||
| this.finalized = true; |
There was a problem hiding this comment.
When a local shutdown occurs after a usageMetadata snapshot but before its terminal event—for example, the duration timer, ping timeout, backpressure timeout, or server closeAll—setting finalized here causes handleUpstreamClose to return without billing pendingUsage. Google may therefore charge the generation while the gateway records no billing row; locally initiated shutdowns need to drain or explicitly bill any buffered stage before suppressing the close handler.
Useful? React with 👍 / 👎.
| if (turnClosedRef.current || !userTurnIdRef.current) { | ||
| turnCounterRef.current += 1; | ||
| userTurnIdRef.current = `user-${turnCounterRef.current}`; | ||
| assistantTurnIdRef.current = null; | ||
| turnClosedRef.current = false; |
There was a problem hiding this comment.
When Gemini delivers inputTranscription after turnComplete—an ordering the surrounding comments explicitly intend to support—turnClosedRef.current is already true, so this condition allocates a new user ID and clears the previous assistant ID. The late text is consequently saved as a new, incorrectly paired conversation turn instead of being appended to the completed user bubble.
Useful? React with 👍 / 👎.
| const response = await fetch("/api/realtime/session", { | ||
| method: "POST", | ||
| headers: { "Content-Type": "application/json" }, | ||
| body: JSON.stringify({ model: currentModel }), | ||
| }); |
There was a problem hiding this comment.
Replace this new raw request to the Next.js /api/realtime/session route with the generated internal-API client and host the minting operation in apps/api. As written, the Gemini call path has no generated request/response contract, so changes to the session payload or authentication can compile successfully and fail only at runtime, contrary to the repository's explicit frontend and backend-operation boundary.
AGENTS.md reference: AGENTS.md:L212-L213
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
apps/gateway/src/realtime/account-gate.ts (1)
67-80: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winParallelize the independent account lookups.
findApiKeyByToken,findOrganizationById, andfindProjectByIddon't depend on each other but are awaited sequentially. This function runs on everyresponse.createand every completed transcription in a live voice session (seerunGenerationGatesInner/enforceLimitsAfterTranscriptionin session.ts), so serializing three round trips adds avoidable latency to a user-facing low-latency path.⚡ Proposed fix
- let freshKey; - let freshOrg; - let freshProject; try { - freshKey = await findApiKeyByToken(input.gatewayToken); - freshOrg = await findOrganizationById(preflight.project.organizationId); - freshProject = await findProjectById(preflight.project.id); + [freshKey, freshOrg, freshProject] = await Promise.all([ + findApiKeyByToken(input.gatewayToken), + findOrganizationById(preflight.project.organizationId), + findProjectById(preflight.project.id), + ]); } catch (error) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/gateway/src/realtime/account-gate.ts` around lines 67 - 80, Update the account lookup block in the realtime authorization flow to start findApiKeyByToken, findOrganizationById, and findProjectById concurrently and await their results together, while preserving the existing error handling and assignments to freshKey, freshOrg, and freshProject.apps/playground/src/hooks/use-gemini-realtime-call.ts (1)
446-455: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueClear any existing elapsed timer before starting a new one.
A second
setupCompleteframe (or a re-handshake) would overwriteelapsedTimerRef.currentand leak the previous interval untilcleanup()runs.♻️ Proposed guard
if (message.setupComplete !== undefined) { updateStatus("live"); setElapsedSeconds(0); + if (elapsedTimerRef.current) { + clearInterval(elapsedTimerRef.current); + } const startedAt = Date.now();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/playground/src/hooks/use-gemini-realtime-call.ts` around lines 446 - 455, In the setupComplete handling block, clear any existing elapsed timer before assigning a new interval to elapsedTimerRef.current. Preserve the current timer initialization and elapsed-time updates while preventing repeated setupComplete frames or re-handshakes from leaving the prior interval active.apps/playground/src/app/api/realtime/session/route.ts (1)
166-166: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider sharing the Gemini provider id constant.
"google-ai-studio"is hardcoded here and again asGEMINI_PROVIDER_IDinapps/playground/src/hooks/use-voice-call.ts(Line 17). A single exported constant keeps the mint-side and client-side protocol decisions from drifting.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/playground/src/app/api/realtime/session/route.ts` at line 166, Replace the hardcoded provider ID in the usesNativeTranscription logic with the shared GEMINI_PROVIDER_ID constant already used by use-voice-call.ts, importing or exporting it as needed so mint-side and client-side checks use one source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/gateway/src/realtime/gemini-session.ts`:
- Around line 1171-1180: Update the drain-timeout callback around
this.drainTimer to bill any buffered usage before shutting down: invoke
billStage("cancelled", { requireUsage: false }) using the same handling as the
upstream-close path, then retain the existing shutdown behavior and diagnostics
logging.
---
Nitpick comments:
In `@apps/gateway/src/realtime/account-gate.ts`:
- Around line 67-80: Update the account lookup block in the realtime
authorization flow to start findApiKeyByToken, findOrganizationById, and
findProjectById concurrently and await their results together, while preserving
the existing error handling and assignments to freshKey, freshOrg, and
freshProject.
In `@apps/playground/src/app/api/realtime/session/route.ts`:
- Line 166: Replace the hardcoded provider ID in the usesNativeTranscription
logic with the shared GEMINI_PROVIDER_ID constant already used by
use-voice-call.ts, importing or exporting it as needed so mint-side and
client-side checks use one source of truth.
In `@apps/playground/src/hooks/use-gemini-realtime-call.ts`:
- Around line 446-455: In the setupComplete handling block, clear any existing
elapsed timer before assigning a new interval to elapsedTimerRef.current.
Preserve the current timer initialization and elapsed-time updates while
preventing repeated setupComplete frames or re-handshakes from leaving the prior
interval active.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 79772690-d20f-44a8-aa52-60bc7474fd1b
📒 Files selected for processing (21)
apps/gateway/src/realtime/account-gate.tsapps/gateway/src/realtime/catalog.spec.tsapps/gateway/src/realtime/connect-upstream.spec.tsapps/gateway/src/realtime/connect-upstream.tsapps/gateway/src/realtime/gemini-pricing.spec.tsapps/gateway/src/realtime/gemini-pricing.tsapps/gateway/src/realtime/gemini-session.spec.tsapps/gateway/src/realtime/gemini-session.tsapps/gateway/src/realtime/preflight.spec.tsapps/gateway/src/realtime/preflight.tsapps/gateway/src/realtime/server.spec.tsapps/gateway/src/realtime/server.tsapps/gateway/src/realtime/session.tsapps/playground/src/app/api/realtime/session/route.tsapps/playground/src/components/playground/realtime-page-client.tsxapps/playground/src/hooks/use-gemini-realtime-call.tsapps/playground/src/hooks/use-voice-call.tsapps/playground/src/lib/realtime-model-value.spec.tsapps/playground/src/lib/realtime-model-value.tspackages/models/src/models/google.tspackages/models/src/realtime-models.spec.ts
…ealtime-voice # Conflicts: # apps/playground/src/components/playground/realtime-page-client.tsx
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
2 participants
Adds Gemini Live as a second realtime provider alongside OpenAI, proxying Google's native
BidiGenerateContentprotocol end to end rather than translating it: a siblingGeminiRealtimeProxySessionhandles setup validation, media/hosted-tool rejection, per-generation-stage billing fromusageMetadata, and post-billing account/credit/rate-limit gates, whileconnectUpstreamand the account gate are extracted into shared modules and the OpenAI path is left unchanged (session.spec.tsis untouched as the regression proof). Two models are added to the catalogue —gemini-2.5-flash-native-audio-preview-12-2025andgemini-3.1-flash-live-preview— with prices taken from Google's published pricing page, and the playground gains a Gemini call hook plus a provider-selecting facade so the browser speaks whichever protocol serves the selected model.Usage normalization was derived from live traces rather than assumption: Gemini excludes
thoughtsTokenCountfromtotalTokenCount, and 3.1 leaves ~18 prompt tokens per turn unattributed to any modality, so the normalizer treats the reported total as an upper bound and bills unattributed input at the text rate — still failing closed on over-attribution, unpriceable modalities, or a block whose details explain less than half the total. Verified withpnpm build,pnpm lint, and 305 targeted unit tests, plus live calls through the local gateway against both models where a billed turn matched the persistedbilling_costexactly; no database migration is required.One assumption remains unvalidated: a
toolCallarriving without bufferedusageMetadatafails the session closed, which no exercised path covers yet, so it is worth a look during review.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests