feat: add Azure realtime provider and nested model normalization

[danpiths]

Summary

Adds Azure as a realtime provider (WebSocket, WebRTC, client secrets) and
introduces nested model prefix stripping so Bifrost-style provider/model
strings in session configs (e.g. openai/whisper-1 in
input_audio_transcription.model) are normalized to bare model names before
forwarding upstream.

Changes

• core/providers/azure/realtime.go (new): Full Azure realtime
  implementation covering RealtimeProvider (WebSocket),
  RealtimeWebRTCProvider (SDP exchange), and RealtimeSessionProvider (client
  secrets only — legacy /sessions returns a clear error). Reuses OpenAI event
  converters since Azure uses the same wire protocol. Key Azure-specific
  behavior:
  • URLs use /openai/v1/realtime prefix with preview api-version query param
  • Auth uses api-key header for API keys, Authorization: Bearer for
    ephemeral tokens (ek_*)
  • Model value maps to the Azure deployment name (resolved via key aliases
    upstream)
• core/providers/openai/realtime.go: Exported StripNestedModelPrefixes
  and ExtractNestedVoice for reuse by Azure and the transport handlers. Added
  StripNestedModelPrefixes calls in normalizeRealtimeClientSecretsRequest
  and normalizeRealtimeSessionsRequest to strip provider prefixes from nested
  model fields in both old format (input_audio_transcription.model) and new
  format (audio.input.transcription.model)
• core/providers/openai/realtime_test.go: Updated test expectations for
  exported helpers

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

# Provider tests
go test ./core/providers/azure/ -count=1 -v
go test ./core/providers/openai/ -count=1 -v

# Full build
make build LOCAL=1

Manual verification:

1. Configure an Azure key with a realtime deployment alias
2. Connect via WebSocket to /v1/realtime?model=<azure-deployment>
3. Connect via WebRTC with ephemeral token from /v1/realtime/client_secrets
4. Send a session config with openai/whisper-1 as transcription model — verify
   it's normalized to whisper-1 upstream
5. POST to /v1/realtime/sessions routed to Azure — verify clear error message
   about using /client_secrets instead

Screenshots/Recordings

N/A

Breaking changes

• Yes
• No

Related issues

N/A

Security considerations

Ephemeral token detection uses ek_ prefix check to switch from api-key to
Authorization: Bearer header. This is consistent with how OpenAI ephemeral
tokens work and doesn't expose any additional auth surface.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[danpiths]

• feat: improve realtime log detail UI with voice, transport, and audio tokens #3336
• feat: enrich realtime routing, logging, cost, and session tracking #3335
• feat: add Azure realtime provider and nested model normalization #3334 👈 (View in Graphite)
• fix: wrap Makefile subshell cd commands in parentheses #3333
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8c0a379a-34e3-4c36-ad1a-38eb2922d672

📥 Commits

Reviewing files that changed from the base of the PR and between 34a893c and de6e290.

📒 Files selected for processing (7)

• core/internal/llmtests/realtime.go
• core/providers/azure/realtime.go
• core/providers/elevenlabs/realtime.go
• core/providers/openai/realtime.go
• core/providers/openai/realtime_test.go
• core/schemas/realtime.go
• transports/bifrost-http/handlers/wsrealtime.go

🚧 Files skipped from review as they are similar to previous changes (2)

• core/providers/openai/realtime_test.go
• core/providers/azure/realtime.go

---

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Azure realtime: WebSocket and WebRTC access, plus realtime client-secret provisioning and session proxying.
• Improvements
  • Context-aware realtime headers and improved upstream error handling for more reliable connections.
  • Normalizes nested model fields and extracts audio voice info for more accurate realtime routing.
• Refactor
  • Shared realtime utilities exported and standardized across providers.
• Tests
  • Unit tests updated to use the exported realtime utilities.

Walkthrough

Exports OpenAI realtime normalization helpers; updates RealtimeHeaders signature across providers; implements AzureProvider realtime (WebSocket URL/headers, WebRTC SDP exchange, client_secrets proxy), delegating event conversion and extraction to OpenAI helpers.

Changes

Azure Realtime and OpenAI Utility Export

Layer / File(s)	Summary
OpenAI Realtime Utilities Export core/providers/openai/realtime.go, core/providers/openai/realtime_test.go	NormalizeRealtimeClientSecretRequest, StripNestedModelPrefixes, and ExtractNestedVoice are exported; tests updated to call the exported normalizer.
OpenAI CreateRealtimeClientSecret integration core/providers/openai/realtime.go	CreateRealtimeClientSecret now calls exported normalizer, computes upstreamURL locally, and strips nested model prefixes during both session and non-session normalization flows.
RealtimeHeaders interface & implementations core/schemas/realtime.go, core/providers/openai/realtime.go, core/providers/elevenlabs/realtime.go	RealtimeHeaders signature updated to accept *schemas.BifrostContext and return (map[string]string, *schemas.BifrostError); implementations updated accordingly.
Azure realtime wiring and delegation core/providers/azure/realtime.go	Adds Azure realtime provider foundation, WebSocket URL rewrite (ws/wss) with model and api-version, header/auth selection for ephemeral (ek_) vs static keys, and delegates realtime event conversion/usage/output extraction to the OpenAI event helper; exposes data channel label and WebSocket subprotocol.
Azure WebRTC SDP, client-secrets, and helpers core/providers/azure/realtime.go	Implements ExchangeRealtimeWebRTCSDP (multipart SDP POST with conditional session, auth/header merging, and upstream status handling), CreateRealtimeClientSecret proxy for /v1/realtime/client_secrets (rejects legacy /sessions), and Azure helpers for shaping errors and selecting/parsing api-version and upstream error bodies.
Handler & test updates transports/bifrost-http/handlers/wsrealtime.go, core/internal/llmtests/realtime.go	runRealtimeSession and internal realtime tests now call RealtimeHeaders(bifrostCtx, key), handle returned errors, and convert header maps for upstream connection acquisition.

Sequence Diagram

sequenceDiagram
  participant Client
  participant AzureProvider
  participant AzureUpstream
  Client->>AzureProvider: POST /v1/realtime/client_secrets or SDP multipart (sdp + optional session)
  AzureProvider->>AzureUpstream: POST /openai/v1/realtime?model&api-version (JSON or multipart + auth/extra headers)
  AzureUpstream-->>AzureProvider: response + headers (2xx or error)
  AzureProvider-->>Client: passthrough body (2xx) or shaped BifrostError (non-2xx)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐰 I dug a tunnel through the cloud,
WSS and SDP I now avowed,
OpenAI guides the event parade,
Azure keys and headers laid —
Streams awake, the realtime crowd.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 46.15% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main changes: adding Azure realtime provider support and implementing nested model prefix normalization for session configs.
Description check	✅ Passed	The description comprehensively covers all required template sections including summary, detailed changes, type of change, affected areas, testing instructions, and security considerations.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/05-08-feat_add_azure_realtime_provider_and_nested_model_normalization

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Confidence Score: 4/5

Safe to merge with one open issue in the WebRTC session path that was flagged in a prior review and is unaddressed here.

The core normalization logic is correct for both the WebSocket and client-secrets flows. The previously flagged omission — resolveRealtimeSDPTarget in webrtc_realtime.go normalizing the top-level model but not calling StripNestedModelPrefixes on the remaining session fields — is confirmed still present and means a WebRTC session sent with input_audio_transcription.model of openai/whisper-1 will reach Azure with the bare openai/ prefix intact and be rejected. The rest of the new Azure provider and the normalization additions look correct and well-structured.

transports/bifrost-http/handlers/webrtc_realtime.go — resolveRealtimeSDPTarget normalizes the top-level model but leaves nested model fields in the forwarded session without stripping provider prefixes.

Important Files Changed

Filename	Overview
core/providers/azure/realtime.go	New Azure realtime provider implementing WebSocket, WebRTC SDP exchange, and client secrets. Correctly delegates to OpenAI wire-protocol helpers via a zero-value struct and handles ephemeral-token vs. standard API-key auth. RealtimeHeaders now calls getAzureAuthHeaders for Azure AD cases (fixing the previous auth gap). Previously-flagged issues around WebRTC upstream error passthrough and the WebRTC session path missing StripNestedModelPrefixes remain open.
core/providers/openai/realtime.go	Exports NormalizeRealtimeClientSecretRequest, StripNestedModelPrefixes, and ExtractNestedVoice for reuse. Correctly calls StripNestedModelPrefixes in both normalizeRealtimeClientSecretsRequest (on session) and normalizeRealtimeSessionsRequest (on root). ExtractNestedVoice is exported but has no callers in the current codebase.
core/providers/openai/realtime_test.go	Updates test call sites from the now-exported NormalizeRealtimeClientSecretRequest; no logic changes, tests continue to cover the same normalization scenarios.
core/schemas/realtime.go	RealtimeHeaders interface updated to accept a BifrostContext and return a BifrostError, enabling providers to perform async auth (e.g., Azure AD token fetch). Change is additive and all callers updated.
transports/bifrost-http/handlers/wsrealtime.go	Correctly separates header construction from pool.Get call and propagates the new BifrostError return; auth failures are now surfaced to the client instead of silently failing.
core/providers/elevenlabs/realtime.go	Mechanical update to match the new RealtimeHeaders signature; returns the same headers with a nil error.
core/internal/llmtests/realtime.go	Test helper updated to call the new RealtimeHeaders signature and fatally fail when header construction returns an error, rather than silently proceeding with missing auth.

_{Reviews (7): Last reviewed commit: "feat: add Azure realtime provider and ne..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/providers/azure/realtime.go`:
- Around line 253-254: The client_secrets failure path currently returns
parseAzureRealtimeError(resp) which cannot access ctx/sendBackRawResponse so
RawResponse isn’t attached; change the error-return path in realtime.go (the
spot calling parseAzureRealtimeError(resp) around the client_secrets branch and
the similar block at lines ~343-366) to use the same helper that preserves raw
payloads (e.g., call realtimeWebRTCUpstreamError(resp, ctx) or extend
parseAzureRealtimeError to accept ctx/sendBackRawResponse) so that when
sendBackRawResponse is enabled the returned error includes RawResponse; ensure
the modified call site passes the request context (ctx) and any flags required
to attach resp.Body to the returned error object.
- Around line 337-339: The current code returns
key.AzureKeyConfig.APIVersion.GetValue() without checking for an empty string,
which yields "api-version=" and breaks realtime calls; update the logic in the
block that references key.AzureKeyConfig and APIVersion.GetValue() to treat an
empty resolved value as unset and instead return AzureAPIVersionPreview as the
fallback (i.e., get the value from APIVersion.GetValue(), and if it's empty,
return AzureAPIVersionPreview).
- Around line 197-208: The BifrostError returned for the legacy /sessions
endpoint uses a hardcoded provider key (schemas.Azure) which can mislabel errors
when an Azure provider alias is configured; update the ExtraFields.Provider to
use provider.GetProviderKey() instead so the error metadata reflects the
configured provider key (change the provider field in the BifrostError
construction in the branch where endpointType ==
schemas.RealtimeSessionEndpointSessions to call provider.GetProviderKey()).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d2eff3dd-c4f1-42ce-8637-0f63a47ae4c1

📥 Commits

Reviewing files that changed from the base of the PR and between ddc3e56 and a1a3812.

📒 Files selected for processing (3)

• core/providers/azure/realtime.go
• core/providers/openai/realtime.go
• core/providers/openai/realtime_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

• core/providers/openai/realtime_test.go
• core/providers/openai/realtime.go

[akshaydeo]

Merge activity

• May 14, 1:58 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 14, 2:00 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.

The base branch was changed.