fix(router): resolve realtime session model to routed deployment by devin-ai-integration[bot] · Pull Request #36811 · BerriAI/litellm

devin-ai-integration · 2026-08-13T19:53:55Z

TLDR

Problem this solves:

Realtime client secrets fail for aliased model groups
The pre-routing group name still rides inside session
The minted key names one model, /v1/realtime/calls sends another, OpenAI rejects the call

How it solves it:

Router rewrites session.model to the routed deployment
Sessions without a model stay untouched
Nested transcription models are preserved

User Flow

Before: a developer whose voice app mints ephemeral realtime keys through the gateway cannot connect a WebRTC call on any model group whose name differs from the underlying model

Their proxy admin serves a group named openai/openai/gpt-realtime-2.1-mini backed by openai/gpt-realtime-2.1-mini
The app server sends POST https://litellm-domain/v1/realtime/client_secrets with {"session": {"type": "realtime", "model": "openai/openai/gpt-realtime-2.1-mini"}}
Back comes HTTP 200 with an ephemeral value, but session.model reads openai/gpt-realtime-2.1-mini, the group name minus one prefix rather than the deployment's own model
The browser sends POST https://litellm-domain/v1/realtime/calls with that ephemeral key as the bearer and its SDP offer as the body
Back comes HTTP 400 with Model "gpt-realtime-2.1-mini" does not match the realtime token model., so the call never connects
A shorter alias such as my-realtime fails one step earlier: step 2 returns HTTP 400 LLM Provider NOT provided ... You passed model=my-realtime

After: the same two requests connect the call

Their proxy admin serves a group named openai/openai/gpt-realtime-2.1-mini backed by openai/gpt-realtime-2.1-mini
The app server sends POST https://litellm-domain/v1/realtime/client_secrets with {"session": {"type": "realtime", "model": "openai/openai/gpt-realtime-2.1-mini"}}
Back comes HTTP 200 with an ephemeral value and session.model reading gpt-realtime-2.1-mini, the deployment's own model
The browser sends POST https://litellm-domain/v1/realtime/calls with that ephemeral key as the bearer and its SDP offer as the body
Back comes HTTP 201 with the SDP answer, so the WebRTC call connects
A shorter alias such as my-realtime mints and connects the same way

Relevant issues

Fixes #36742

Linear ticket

Resolves LIT-6677

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy against real OpenAI; every 200 and 201 below mints or connects a real realtime session and costs real money. Both legs run the same topology: two proxy instances per leg, each booted with --num_workers 2, the key minted on instance A and the call placed through instance B, so the ephemeral token has to survive a hop between pods that share nothing but the config

Config served on both legs:

model_list:
  - model_name: openai/openai/gpt-realtime-2.1-mini
    litellm_params:
      model: openai/gpt-realtime-2.1-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: my-realtime
    litellm_params:
      model: openai/gpt-realtime-2.1-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-realtime-2.1-mini
    litellm_params:
      model: openai/gpt-realtime-2.1-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-lit6677

Launcher, run once per instance on its own free random port:

export PYTHONPATH=<worktree>
python litellm/proxy/proxy_cli.py --config lit6677_config.yaml --port <port> --num_workers 2 --detailed_debug

offer.sdp is a real audio sendrecv offer generated with aiortc's RTCPeerConnection. ws_probe.py opens /v1/realtime?model=... with the websockets library, sends nothing, and prints the first server event's type and session.model. EPHEMERAL is the value returned by the mint step right above each call

Before (`ba2e5d2`)

Provider-prefixed alias `openai/openai/gpt-realtime-2.1-mini`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:36020/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "openai/openai/gpt-realtime-2.1-mini"}}'

{"value":"GmF80SoVWvRf...","expires_at":1788376690,"session":{"type":"realtime","model":"openai/gpt-realtime-2.1-mini","id":"sess_EJkba852LiCRS3xzm6qnF"}}
HTTP 200

Place the WebRTC call on instance B with that key

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:24545/v1/realtime/calls -H "Authorization: Bearer $EPHEMERAL" -H 'Content-Type: application/sdp' --data-binary @offer.sdp

{"error":{"message":"{\n \"error\": {\n \"message\": \"Model \\\"gpt-realtime-2.1-mini\\\" does not match the realtime token model.\",\n \"type\": \"invalid_request_error\",\n \"code\": \"invalid_model\",\n \"param\": \"\"\n }\n}. Received 
HTTP 400

Plain alias `my-realtime`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:36020/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "my-realtime"}}'

{"error": {"message": "litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=my-realtime\n Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.
HTTP 400

Non-aliased group `gpt-realtime-2.1-mini`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:36020/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "gpt-realtime-2.1-mini"}}'

{"value":"bj17_3bZPsAM...","expires_at":1788376697,"session":{"type":"realtime","model":"gpt-realtime-2.1-mini","id":"sess_EJkbhlXwebhYPtrtgKl57"}}
HTTP 200

Place the WebRTC call on instance B with that key

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:24545/v1/realtime/calls -H "Authorization: Bearer $EPHEMERAL" -H 'Content-Type: application/sdp' --data-binary @offer.sdp

v=0
o=- 868949830145856169 1788376099 IN IP4 0.0.0.0
HTTP 201

Websocket `/v1/realtime` on both instances

Open the websocket and read the first event

python ws_probe.py ws://localhost:36020/v1/realtime?model=openai/openai/gpt-realtime-2.1-mini sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:36020/v1/realtime?model=my-realtime sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:24545/v1/realtime?model=openai/openai/gpt-realtime-2.1-mini sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:24545/v1/realtime?model=my-realtime sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

After (`ec1fba5`)

Provider-prefixed alias `openai/openai/gpt-realtime-2.1-mini`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:48795/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "openai/openai/gpt-realtime-2.1-mini"}}'

{"value":"1NBLYGJ7Qu-8...","expires_at":1788380462,"session":{"type":"realtime","model":"gpt-realtime-2.1-mini","id":"sess_EJlaQCkPqiz7kbka2Fxw2"}}
HTTP 200

Place the WebRTC call on instance B with that key

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:38771/v1/realtime/calls -H "Authorization: Bearer $EPHEMERAL" -H 'Content-Type: application/sdp' --data-binary @offer.sdp

v=0
o=- 1310369432372948089 1788379862 IN IP4 0.0.0.0
HTTP 201

Plain alias `my-realtime`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:48795/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "my-realtime"}}'

{"value":"6dFQA5i44lKR...","expires_at":1788380463,"session":{"type":"realtime","model":"gpt-realtime-2.1-mini","id":"sess_EJlaRA6a2eiv1uJepgYOG"}}
HTTP 200

Place the WebRTC call on instance B with that key

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:38771/v1/realtime/calls -H "Authorization: Bearer $EPHEMERAL" -H 'Content-Type: application/sdp' --data-binary @offer.sdp

v=0
o=- 151078748963338984 1788379864 IN IP4 0.0.0.0
HTTP 201

Non-aliased group `gpt-realtime-2.1-mini`

Mint the ephemeral key on instance A

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:48795/v1/realtime/client_secrets -H 'Authorization: Bearer sk-lit6677' -H 'Content-Type: application/json' -d '{"session": {"type": "realtime", "model": "gpt-realtime-2.1-mini"}}'

{"value":"6JANahu_OEMj...","expires_at":1788380464,"session":{"type":"realtime","model":"gpt-realtime-2.1-mini","id":"sess_EJlaSPZ7Y0In0bWS6kED6"}}
HTTP 200

Place the WebRTC call on instance B with that key

curl -s -w '\nHTTP %{http_code}\n' -X POST http://localhost:38771/v1/realtime/calls -H "Authorization: Bearer $EPHEMERAL" -H 'Content-Type: application/sdp' --data-binary @offer.sdp

v=0
o=- 632503616080809326 1788379865 IN IP4 0.0.0.0
HTTP 201

Websocket `/v1/realtime` on both instances

Open the websocket and read the first event

python ws_probe.py ws://localhost:48795/v1/realtime?model=openai/openai/gpt-realtime-2.1-mini sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:48795/v1/realtime?model=my-realtime sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:38771/v1/realtime?model=openai/openai/gpt-realtime-2.1-mini sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Open the websocket and read the first event

python ws_probe.py ws://localhost:38771/v1/realtime?model=my-realtime sk-lit6677

{"type": "session.created", "session.model": "gpt-realtime-2.1-mini", "error": null}

Observations from the run:

Websocket route already resolved both aliases before this PR; left alone
Non-aliased group gpt-realtime-2.1-mini worked on both legs; left alone
Key minted on instance A was accepted on instance B on both legs

Type

🐛 Bug Fix

Caveats (if any)

Low

/openai/v1/realtime/client_secrets passes through untouched; it rejects a top-level model before and after
A group mixing different realtime models can mint on one deployment and call on another; pre-existing, unchanged here
Any future router function that passes a session kwarg carrying a model key gets the same rewrite; today only the two realtime entry points pass one
CircleCI logging_testing is red on test_bedrock_kb_request_body_has_transformed_filters at the merge base too (staging PR fix(vector_stores): s3 vectors search router bypass + rag query config drop + ui error swallow #34788 added a router kwarg the test fake does not accept); it is not a required check and is unrelated to this change

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/17272f72fa994ee0903d2de98ccb5874
Open in Devin Desktop: https://app.devin.ai/desktop/session/17272f72fa994ee0903d2de98ccb5874?variant=devin

ec1fba5 passes /live-pr-risk

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-08-13T19:53:59Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps · 2026-08-13T19:58:24Z

Greptile Summary

The PR synchronizes a realtime session model with the deployment selected by the router while preserving sessions without a model and nested transcription configuration

Adds a validated helper for rewriting session.model
Applies the rewrite before invoking the routed function
Adds focused tests for model replacement and session preservation

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/router.py	Validates session mappings and replaces an existing session model with the selected deployment model
tests/test_litellm/test_router.py	Adds regression coverage for model resolution, nested transcription preservation, and sessions without models

_{Reviews (2): Last reviewed commit: "chore: merge litellm_internal_staging in..." | Re-trigger Greptile}

greptile-apps · 2026-08-13T19:58:27Z

+    """
+    captured: dict = {}
+
+    async def capture_kwargs(**kwargs):


Untyped mutable test capture

Both new tests mutate coarse dict capture state through untyped callbacks, weakening type discipline and making request assertions less explicit

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

codecov · 2026-08-13T20:07:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

codspeed · 2026-08-13T20:40:00Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_realtime_router_session_model (ec1fba5) with litellm_internal_staging (7114302)}

…ssion_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

CLAassistant · 2026-09-02T06:37:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…ssion_model Carries a mutable-ok suppression on the router session rewrite for the tightened LIT002 budget, since the realtime callees deep-copy and JSON-dump the session, and captures the realtime session kwargs through an async mock in the router tests instead of an untyped dict.

mateo-berri · 2026-09-02T20:43:04Z

@greptileai

mateo-berri · 2026-09-02T20:43:05Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ec1fba5. Configure here.}

mateo-berri

LGTM

fix(router): resolve realtime session model to routed deployment

00a2059

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri self-assigned this Aug 13, 2026

test(router): cover realtime session model resolver directly

e5b5462

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Aug 13, 2026

View reviewed changes

chore: merge litellm_internal_staging into litellm_realtime_router_se…

2d643a8

…ssion_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri added the run-ci label Sep 2, 2026

cursor Bot reviewed Sep 2, 2026

View reviewed changes

mateo-berri approved these changes Sep 2, 2026

View reviewed changes

mateo-berri merged commit 719b671 into litellm_internal_staging Sep 2, 2026
124 of 125 checks passed

mateo-berri deleted the litellm_realtime_router_session_model branch September 2, 2026 21:16

mateo-berri mentioned this pull request Sep 2, 2026

fix(router): resolve session.model to the picked deployment in _ageneric_api_call_with_fallbacks_helper #39107

Closed

5 tasks

Navigation Menu

Uh oh!

fix(router): resolve realtime session model to routed deployment - #36811

fix(router): resolve realtime session model to routed deployment#36811
mateo-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_realtime_router_session_model

devin-ai-integration Bot commented Aug 13, 2026 •

edited by mateo-berri

Loading

Uh oh!

devin-ai-integration Bot commented Aug 13, 2026

Uh oh!

greptile-apps Bot commented Aug 13, 2026 •

edited

Loading

Important Files Changed

Uh oh!

greptile-apps Bot Aug 13, 2026

Uh oh!

codecov Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Sep 2, 2026

Uh oh!

mateo-berri commented Sep 2, 2026

Uh oh!

mateo-berri commented Sep 2, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

mateo-berri left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

devin-ai-integration Bot commented Aug 13, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (ba2e5d2)

Provider-prefixed alias openai/openai/gpt-realtime-2.1-mini

Plain alias my-realtime

Non-aliased group gpt-realtime-2.1-mini

Websocket /v1/realtime on both instances

After (ec1fba5)

Provider-prefixed alias openai/openai/gpt-realtime-2.1-mini

Plain alias my-realtime

Non-aliased group gpt-realtime-2.1-mini

Websocket /v1/realtime on both instances

Type

Caveats (if any)

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Aug 13, 2026

🤖 Devin AI Engineer

Uh oh!

greptile-apps Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

CLAassistant commented Sep 2, 2026

Uh oh!

mateo-berri commented Sep 2, 2026

Uh oh!

mateo-berri commented Sep 2, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration Bot commented Aug 13, 2026 •

edited by mateo-berri

Loading

Before (`ba2e5d2`)

Provider-prefixed alias `openai/openai/gpt-realtime-2.1-mini`

Plain alias `my-realtime`

Non-aliased group `gpt-realtime-2.1-mini`

Websocket `/v1/realtime` on both instances

After (`ec1fba5`)

Provider-prefixed alias `openai/openai/gpt-realtime-2.1-mini`

Plain alias `my-realtime`

Non-aliased group `gpt-realtime-2.1-mini`

Websocket `/v1/realtime` on both instances

greptile-apps Bot commented Aug 13, 2026 •

edited

Loading

codecov Bot commented Aug 13, 2026 •

edited

Loading

codspeed Bot commented Aug 13, 2026 •

edited

Loading