feat(sarvam): add realtime speech to text on /v1/realtime

[pythongiant]

TLDR

Problem this solves:

• Sarvam realtime speech to text was unreachable through the proxy
• Its WebSocket protocol is not OpenAI shaped, so no client worked
• Realtime transcription sessions had no Sarvam pricing to bill against

How it solves it:

• Translates Sarvam's realtime socket into OpenAI realtime events
• Routes it as a normal model_list deployment on /v1/realtime
• Bills each session from the audio duration Sarvam reports

User Flow

Before: a developer pointing an OpenAI realtime client at a Sarvam speech to text deployment gets the socket closed before any audio is transcribed

1. The proxy admin adds a sarvam-stt deployment for sarvam/saaras:v3-realtime and restarts the proxy
2. The developer opens a WebSocket to ws://localhost:4000/v1/realtime?model=sarvam-stt&intent=transcription
3. The proxy closes it with code 1011 and the reason Unsupported model: saaras:v3-realtime
4. No audio is ever sent and nothing is logged for the session

After: the same client streams audio and receives transcripts, and the session is billed

1. The proxy admin adds the same sarvam-stt deployment and restarts the proxy
2. The developer opens the same WebSocket and sends a session.update naming the transcription model and language
3. The proxy replies with session.created describing a transcription session at 16 kHz, then session.updated once the language they asked for takes effect
4. They stream pcm16 audio as input_audio_buffer.append frames and see input_audio_buffer.speech_started, a run of transcription delta events, then input_audio_buffer.speech_stopped
5. A conversation.item.input_audio_transcription.completed event carries the full transcript
6. The session is logged at non-zero spend for the seconds of audio Sarvam reports

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Shared setup, used for both runs.

sarvam_config.yaml:

model_list:
  - model_name: sarvam-stt
    litellm_params:
      model: sarvam/saaras:v3-realtime
      api_key: os.environ/SARVAM_API_KEY

general_settings:
  master_key: sk-sarvam-dev

Proxy, in one shell:

LITELLM_LOCAL_MODEL_COST_MAP=True python litellm/proxy/proxy_cli.py --config sarvam_config.yaml --port 4000

realtime_client.py, a plain OpenAI realtime transcription client. WebSockets cannot be driven with curl, so this is the smallest client that sends the same frames an OpenAI SDK would. It streams the repo's tests/audio_tests/gettysburg.wav resampled to 16 kHz:

import asyncio, base64, json, wave
import numpy as np
from websockets.asyncio.client import connect

SESSION_UPDATE = json.dumps({"type": "session.update", "session": {
    "type": "transcription",
    "audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000},
                        "transcription": {"model": "sarvam-stt", "language": "en"},
                        "turn_detection": {"type": "server_vad"}}}}})

def pcm_16k(path):
    w = wave.open(path)
    raw = np.frombuffer(w.readframes(w.getnframes()), dtype=np.int16)
    idx = np.arange(0, len(raw) / w.getframerate() * 16000) * w.getframerate() / 16000
    return np.interp(idx, np.arange(len(raw)), raw).astype(np.int16).tobytes()

async def main():
    audio = pcm_16k("tests/audio_tests/gettysburg.wav") + b"\x00\x00" * 16000
    url = "ws://localhost:4000/v1/realtime?model=sarvam-stt&intent=transcription"
    async with connect(url, additional_headers={"Authorization": "Bearer sk-sarvam-dev"}) as ws:
        await ws.send(SESSION_UPDATE)

        async def send_audio():
            for start in range(0, len(audio), 3200):
                await ws.send(json.dumps({"type": "input_audio_buffer.append",
                                          "audio": base64.b64encode(audio[start:start + 3200]).decode()}))
                await asyncio.sleep(0.1)
            await ws.send(json.dumps({"type": "input_audio_buffer.end"}))

        asyncio.create_task(send_audio())
        deltas = 0
        while True:
            event = json.loads(await ws.recv())
            if event["type"] == "conversation.item.input_audio_transcription.delta":
                deltas += 1
                continue
            if event["type"] == "conversation.item.input_audio_transcription.completed":
                print(f"{deltas} x conversation.item.input_audio_transcription.delta")
                print(f"completed: {event['transcript']}")
                return
            print(json.dumps(event)[:220])

asyncio.run(asyncio.wait_for(main(), 120))

Both runs hit the live Sarvam API with a real subscription key and bill real money.

Before (071cb49)

1. Run the client against the proxy:

python realtime_client.py

2. The socket is closed before any audio is sent:

websockets.exceptions.ConnectionClosedError: received 1011 (internal error) Unsupported model: saaras:v3-realtime; then sent 1011 (internal error) Unsupported model: saaras:v3-realtime

After (9334169)

1. Run the same client against the proxy:

python realtime_client.py

2. The session is created on Sarvam's defaults, the client's session.update comes back as session.updated, audio streams, and the transcript arrives:

{"type": "session.created", "event_id": "event_a6074fd0aae44cd79b49acb0448dfa15", "session": {"id": "20260922_9890ed08-de65-4c9a-a857-8a8756061018", "object": "realtime.transcription_session", "type": "transcription", "audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000}, "transcription": {"model": "saaras:v3-realtime"}, "turn_detection": {"type": "server_vad"}}}}}

{"type": "session.updated", "event_id": "event_552980f7b62d49538c3b4748596f7d34", "session": {"id": "20260922_9890ed08-de65-4c9a-a857-8a8756061018", "object": "realtime.transcription_session", "type": "transcription", "audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000}, "transcription": {"model": "saaras:v3-realtime", "language": "en-IN"}, "turn_detection": {"type": "server_vad"}}}}}

{"type":"input_audio_buffer.speech_started","event_id":"event_4cf3270937ab4d23b3dfafc9919d4b22","item_id":"item_3cd60d9ef009464f90d8877a7734c874"}

{"type":"input_audio_buffer.speech_stopped","event_id":"event_6120b7707dce49e4903f22e4ecb3d413","item_id":"item_3cd60d9ef009464f90d8877a7734c874"}

71 x conversation.item.input_audio_transcription.delta
completed: Four score and seven years ago, our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation or any nation so conceived and so dedicated can long endure.

3. The session is logged with the audio duration Sarvam bills for:

[SPEND] model=saaras:v3-realtime cost_usd=0.0 usage=[{'type': 'duration', 'seconds': 18.2}] billed_seconds=18.2 billed_hours=0.005056

Type

🆕 New Feature

Caveats (if any)

Sample rate is fixed when the socket opens, because Sarvam only accepts 16000 or 8000 and only as a connect time query parameter. The default is 16 kHz and a deployment can pick 8 kHz with api_base: wss://api.sarvam.ai?sample_rate=8000. Cleartext ws:// and http:// api_base values are refused unless they point at a loopback address, since the subscription key travels as a connection header. A client session that declares a different rate, including the 24 kHz that OpenAI realtime clients commonly default to, is refused rather than transcribed as noise. The proxy does not resample.

Deltas are append only. Sarvam revises partial transcripts mid utterance, and OpenAI realtime has no way to retract a delta, so a revision that drops words emits nothing and the authoritative text arrives in the completed event. The transcript a client accumulates from deltas can therefore be shorter than the completed transcript.

Sessions log a duration, not a cost. Sarvam prices speech to text in INR per hour and this registry is USD only, so the entries carry no rate and sessions cost $0 out of the box. Each session logs the audio seconds Sarvam bills for, so an operator can apply their own conversion downstream, or register a per-second USD rate for that model to have LiteLLM cost it. Deployment level input_cost_per_second will not do it: the realtime cost path resolves pricing by model name and never sees the router deployment id.

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Confidence Score: 3/5

This PR is not safe to merge until session creation ordering, partial endpointing updates, and the explicit model-registry requirement are addressed.

Findings

1.  Session reports stale settings ▶
2.  Partial updates re-enable VAD ▶
3.  Model support is hardcoded ▶
4.   Cleartext sockets expose credentials ▶
5.  IPv6 brackets are dropped ▶

Summary

This PR adds a native Sarvam realtime transcription adapter, provider registration, model metadata, development configuration, and unit coverage.

• Translates OpenAI transcription-session and PCM audio events into Sarvam WebSocket frames
• Maps Sarvam speech, transcript, error, and duration events back into shared realtime forms
• Registers Sarvam realtime models for routing and duration-based accounting
• The session update lifecycle has ordering and partial-update defects that must be fixed before merge

_{Reviews (1) · Last reviewed commit: "chore(sarvam): drop the USD conversion f..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing pythongiant:litellm_sarvam_realtime_stt (9334169) with main (071cb49)}

[codecov]

Codecov Report

❌ Patch coverage is 63.63636% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...m/llms/base_llm/realtime/transcription_protocol.py	33.33%	2 Missing ⚠️
litellm/utils.py	33.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[pythongiant]

All five review findings fixed in 9334169: registry-driven model support, session.updated on config.updated, endpointing preserved across partial updates, loopback-only cleartext, IPv6 brackets. CodeQL fall-through returns made explicit.