fix(proxy): register WebSocket passthrough for OpenAI prefixes by LHMQ878 · Pull Request #36151 · BerriAI/litellm

LHMQ878 · 2026-08-07T02:47:48Z

TLDR

Problem this solves:

WebSocket upgrades under /openai and /openai_passthrough always returned 403
client.realtime.connect() and client.responses.connect() could not pass through the proxy

How it solves it:

registers catch-all WebSocket routes for both OpenAI prefixes
injects the provider key and forwards the client query string
decodes upstream frames as utf-8 so real sessions survive
negotiates the client's WebSocket subprotocol like the native realtime route
refuses model-restricted keys at connect, keeping model ACLs intact

User Flow

Before: you cannot open any OpenAI WebSocket session through the proxy's OpenAI passthrough prefixes

You point the OpenAI SDK at the proxy with your LiteLLM key, base_url http://<proxy>/openai/v1 or http://<proxy>/openai_passthrough/v1
You call client.realtime.connect(model="gpt-realtime-2.1-mini") against /openai_passthrough/v1/realtime and the upgrade is rejected with HTTP 403
You call client.responses.connect() against /openai/v1/responses or /openai_passthrough/v1/responses and that upgrade is also rejected with HTTP 403

After: the same SDK calls open live WebSocket sessions through both prefixes

You point the OpenAI SDK at the proxy exactly as before
client.realtime.connect(model="gpt-realtime-2.1-mini") upgrades, session.created arrives, and your realtime conversation streams back with token usage
client.responses.connect() upgrades on either prefix and response.create returns real model output over the socket
A browser client that authenticates via Sec-WebSocket-Protocol values (openai-insecure-api-key.<key>, openai-beta.realtime-v1) gets its first offered subprotocol echoed back in the handshake, so spec-compliant browsers keep the connection open instead of aborting it
If your key is restricted to specific models, both connects are refused at the handshake with HTTP 403, so a restricted key cannot reach arbitrary models over these WebSocket routes; the native /openai/v1/realtime route keeps serving that key's allowed models

Relevant issues

Fixes #36088

Linear ticket

Resolves LIT-5396

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy booted from the repo with a one-model config (gpt-realtime-2.1-mini + master key), real OpenAI API calls costing real $. Clients are the plain OpenAI Python SDK (2.33.0) pointed at the proxy, plus a raw websockets client that mimics a browser's subprotocol handshake

realtime_qa.py (OpenAI SDK realtime client used below)

import asyncio, os, sys
from openai import AsyncOpenAI

PORT, PREFIX = sys.argv[1], sys.argv[2]
MODEL = sys.argv[3] if len(sys.argv) > 3 else "gpt-realtime-2.1-mini"

async def main() -> None:
    client = AsyncOpenAI(api_key=os.environ["LITELLM_PROXY_KEY"], base_url=f"http://127.0.0.1:{PORT}/{PREFIX}/v1")
    async with client.realtime.connect(model=MODEL) as conn:
        await conn.session.update(session={"type": "realtime", "output_modalities": ["text"]})
        await conn.conversation.item.create(item={"type": "message", "role": "user",
            "content": [{"type": "input_text", "text": "Reply with exactly: WS PASSTHROUGH OK"}]})
        await conn.response.create()
        async for event in conn:
            print("event:", event.type)
            if event.type == "response.output_text.done":
                print("  model said:", event.text)
            if event.type == "response.done":
                print("  status:", event.response.status)
                print("  usage:", event.response.usage)
                break

asyncio.run(main())

responses_qa.py (OpenAI SDK responses.connect client used below)

import asyncio, os, sys
from openai import AsyncOpenAI

PORT, PREFIX = sys.argv[1], sys.argv[2]
MODEL = sys.argv[3] if len(sys.argv) > 3 else "gpt-5.6"

async def main() -> None:
    client = AsyncOpenAI(api_key=os.environ["LITELLM_PROXY_KEY"], base_url=f"http://127.0.0.1:{PORT}/{PREFIX}/v1")
    async with client.responses.connect() as conn:
        await conn.send({"type": "response.create", "model": MODEL, "input": "Reply with exactly: WS RESPONSES OK"})
        while True:
            event = await conn.recv()
            print("event:", event.type)
            if event.type == "response.output_text.done":
                print("  model said:", event.text)
            if event.type in ("response.completed", "response.failed", "response.incomplete"):
                print("  status:", event.response.status)
                print("  model:", event.response.model)
                print("  usage:", event.response.usage)
                break

asyncio.run(main())

subproto_qa.py (browser-style client: auth via Sec-WebSocket-Protocol, no Authorization header)

import asyncio, json, sys
import websockets

PORT, PREFIX, KEY = sys.argv[1], sys.argv[2], sys.argv[3]
MODEL = sys.argv[4] if len(sys.argv) > 4 else "gpt-realtime-2.1-mini"

async def main() -> None:
    url = f"ws://localhost:{PORT}/{PREFIX}/v1/realtime?model={MODEL}"
    offered = ["realtime", f"openai-insecure-api-key.{KEY}", "openai-beta.realtime-v1"]
    async with websockets.connect(url, subprotocols=offered) as ws:
        print("SELECTED_SUBPROTOCOL:", ws.subprotocol)
        first = json.loads(await ws.recv())
        print("FIRST_EVENT:", first.get("type"))
        await ws.send(json.dumps({"type": "response.create",
            "response": {"instructions": "Say hola in one word", "output_modalities": ["text"]}}))
        async for raw in ws:
            event = json.loads(raw)
            if event.get("type") == "response.done":
                response = event.get("response", {})
                print("RESPONSE_DONE status:", response.get("status"),
                      "total_tokens:", response.get("usage", {}).get("total_tokens"))
                break
    print("BROWSER SUBPROTO OK")

asyncio.run(main())

Before, at merge base `4e5495e` (proxy on port 28517)

$ python realtime_qa.py 28517 openai_passthrough
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

$ python responses_qa.py 28517 openai
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

$ python responses_qa.py 28517 openai_passthrough
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

Control on the same proxy: the native realtime bridge works, so only the passthrough surfaces were broken

$ python realtime_qa.py 28517 openai
event: response.done
  status: completed
  usage: ... input_tokens=132 ... output_tokens=23, total_tokens=155

After, at `5965648` (proxy on port 29226)

$ python realtime_qa.py 29226 openai_passthrough
event: response.output_text.done
  model said: WS PASSTHROUGH OK
event: response.done
  status: completed
  usage: ... input_tokens=132 ... output_tokens=24, total_tokens=156

$ python responses_qa.py 29226 openai
event: response.output_text.done
  model said: WS RESPONSES OK
event: response.completed
  status: completed
  model: gpt-5.6-sol
  usage: ... input_tokens=15 ... output_tokens=9, total_tokens=24

$ python responses_qa.py 29226 openai_passthrough
event: response.output_text.done
  model said: WS RESPONSES OK
event: response.completed
  status: completed
  model: gpt-5.6-sol
  usage: ... total_tokens=24

Subprotocol negotiation for browser clients

At a258b2b (before this fix) the route accepted the socket without selecting any subprotocol even when the client offered some; RFC 6455 requires such clients (all browsers) to fail the connection. After the fix the first offered subprotocol is echoed, mirroring the native /openai/v1/realtime route, and the whole session runs authenticated purely by the openai-insecure-api-key. subprotocol

$ python subproto_qa.py 22160 openai_passthrough $LITELLM_KEY   # at a258b2b130
SELECTED_SUBPROTOCOL: None
FIRST_EVENT: session.created

$ python subproto_qa.py 29226 openai_passthrough $LITELLM_KEY   # at 5965648547
SELECTED_SUBPROTOCOL: realtime
FIRST_EVENT: session.created
RESPONSE_DONE status: completed total_tokens: 61
BROWSER SUBPROTO OK

$ python subproto_qa.py 29226 openai $LITELLM_KEY               # at 5965648547
SELECTED_SUBPROTOCOL: realtime
FIRST_EVENT: session.created
RESPONSE_DONE status: completed total_tokens: 52
BROWSER SUBPROTO OK

Model ACL enforcement, at `5965648` (same proxy, DB-backed)

A key restricted to one model is refused at the handshake on both WebSocket surfaces, in both auth styles, and still works on the native realtime route with its allowed model

$ curl -s http://127.0.0.1:29226/key/generate -H "Authorization: Bearer $MASTER" \
    -d '{"models": ["gpt-realtime-2.1-mini"]}'   # -> sk-MS_WO...

$ LITELLM_PROXY_KEY=sk-MS_WO... python realtime_qa.py 29226 openai_passthrough
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

$ python subproto_qa.py 29226 openai_passthrough sk-MS_WO...
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

$ LITELLM_PROXY_KEY=sk-MS_WO... python realtime_qa.py 29226 openai
event: response.done
  status: completed
  usage: ... input_tokens=132 ... output_tokens=28, total_tokens=160

Missing provider credential, at `5965648` (proxy on port 22245, no OPENAI_API_KEY configured)

The handshake is refused cleanly with no server-side traceback, instead of the previous close-then-raise that left an ASGI ValueError in the logs

$ python subproto_qa.py 22245 openai_passthrough $LITELLM_KEY
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403

$ grep -c "Traceback" qa_proxy_nokey2.log; grep -c "ValueError" qa_proxy_nokey2.log
0
0

Caveats:

native /openai/v1/realtime route is untouched; it shadows the catch-all
model-restricted keys are refused on these WebSocket routes
first offered subprotocol is echoed, same as the native route
/vertex_ai/live model-ACL parity is a separate follow-up

Type

🐛 Bug Fix

Changes

Registers a catch-all @router.websocket route for the /openai and /openai_passthrough prefixes in llm_passthrough_endpoints.py. The handler authenticates through user_api_key_auth_websocket, injects the configured OpenAI key upstream, preserves the client's query string, and hands off to the shared websocket_passthrough_request forwarder

The shared forwarder previously decoded the upstream's first frame as ascii, which crashed on any non-ascii byte in OpenAI's session.created payload; it now decodes utf-8. Keys restricted to specific models are refused at connect with close code 1008 so these routes cannot be used to sidestep model ACLs, matching the enforcement the HTTP passthrough applies to request bodies. The logged endpoint now reflects the actual request path instead of hardcoding /openai/

The route accepts the socket itself, echoing the client's first offered Sec-WebSocket-Protocol exactly like the native /openai/v1/realtime route, so browser clients that carry auth in subprotocols survive the handshake. The URL join helper moved from a private static method to a module-level _join_url_paths so the route does not reach into another class's private API. When no OpenAI credential is configured the route closes with 1011 and returns instead of raising after the close, matching the Vertex live path

Unit tests cover route registration, query and auth forwarding, subprotocol selection, the restricted-key refusal, the unrestricted-key allow list, the credential-missing clean close, and a regression test that fails on the old ascii decode

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

create_websocket_passthrough_route existed but /openai and /openai_passthrough only registered HTTP methods, so WS upgrades were rejected at routing. Add catch-all websocket routes mirroring the HTTP passthrough target construction. Fixes BerriAI#36088

CLAassistant · 2026-08-07T02:47:55Z

All committers have signed the CLA.

greptile-apps · 2026-08-07T02:50:20Z

Greptile Summary

The PR adds authenticated OpenAI WebSocket passthrough routes and updates the shared forwarder to handle UTF-8 setup frames.

Registers catch-all WebSocket routes for /openai and /openai_passthrough.
Preserves client query parameters while injecting configured provider credentials.
Adds route, authorization, subprotocol, credential, and UTF-8 regression coverage.
Regenerates the dashboard HTTP schema for the new routes.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py	Adds authenticated OpenAI WebSocket route registration, upstream URL construction, credential injection, and subprotocol negotiation.
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py	Changes initial upstream WebSocket frame handling from ASCII to UTF-8.
tests/test_litellm/proxy/test_openai_ws_passthrough_routes.py	Adds focused coverage for route registration, query forwarding, authentication isolation, subprotocol selection, ACL handling, and missing credentials.
tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py	Adds regression coverage for forwarding non-ASCII setup frames.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Regenerates schema declarations for the two WebSocket endpoints.

_{Reviews (6): Last reviewed commit: "fix(proxy): close websocket cleanly when..." | Re-trigger Greptile}

veria-ai · 2026-08-07T02:52:17Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

codecov · 2026-08-07T02:55:27Z

Codecov Report

❌ Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...y/pass_through_endpoints/pass_through_endpoints.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Forward realtime model query string, keep OPENAI_API_KEY (forward_headers=False), satisfy ruff strict gates, sync dashboard OpenAPI types, and cover the behavior in tests.

LHMQ878 · 2026-08-07T03:09:57Z

Addressed the review/CI feedback in the latest commit:

Forward WebSocket query string (realtime model, etc.)
forward_headers=False so proxy auth cannot replace OPENAI_API_KEY
Ruff strict-gate cleanups (Annotated Depends, -> None, ValueError)
Synced schema.d.ts for the new WS routes
Added a unit test covering query forwarding + provider credential headers

CLA: please re-check if still pending — I may need to re-sign in the browser for this PR.

…route

codspeed-hq · 2026-08-07T04:29:31Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing LHMQ878:fix/36088-openai-ws-passthrough (5965648) with litellm_internal_staging (973329e)}

- decode upstream first frame as utf-8 instead of ascii - reject model-restricted keys at connect to match HTTP model enforcement - log the actual request path for /openai_passthrough traffic

mateo-berri · 2026-08-16T20:57:04Z

@greptileai

mateo-berri · 2026-08-16T20:57:06Z

bugbot run

cursor · 2026-08-16T20:57:19Z

PR Summary

^{Cursor Bugbot is generating a summary for commit a258b2b. Configure here.}

…itellm_pr36151_ws_passthrough # Conflicts: # litellm/proxy/pass_through_endpoints/pass_through_endpoints.py # tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py

mateo-berri · 2026-08-16T21:25:35Z

@greptileai

mateo-berri · 2026-08-16T21:25:36Z

bugbot run

mateo-berri · 2026-08-16T21:31:01Z

@greptileai

mateo-berri · 2026-08-16T21:31:02Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4ba9d6b. Configure here.}

mateo-berri · 2026-08-16T21:49:39Z

@greptileai

mateo-berri · 2026-08-16T21:49:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5965648. Configure here.}

greptile-apps · 2026-08-16T21:52:04Z

Want your agent to iterate on Greptile's feedback? Try greploops.

mateo-berri

LGTM. Thanks for the contribution!

LHMQ878 · 2026-08-17T05:39:52Z

Thanks for picking up the remaining review items and landing them on this branch.

The remaining merge block I can see is the CLA check on my commit. I'll sign that so it doesn't hold the rest of the work.

greptile-apps Bot reviewed Aug 7, 2026

View reviewed changes

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py Outdated

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py Outdated

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py Outdated

veria-ai Bot reviewed Aug 7, 2026

View reviewed changes

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py Outdated

fix(proxy): preserve OpenAI WS query params and provider auth

55b5297

Forward realtime model query string, keep OPENAI_API_KEY (forward_headers=False), satisfy ruff strict gates, sync dashboard OpenAPI types, and cover the behavior in tests.

LHMQ878 added 3 commits August 7, 2026 11:21

fix(proxy): satisfy type-discipline Final/mutable rules on OpenAI WS …

ae4ee36

…route

fix(proxy): place mutable-ok on OpenAI WS headers dict literal

7a9c38e

style(proxy): ruff-format OpenAI websocket passthrough route

02d4f8d

fix(proxy): harden OpenAI websocket passthrough

a258b2b

- decode upstream first frame as utf-8 instead of ascii - reject model-restricted keys at connect to match HTTP model enforcement - log the actual request path for /openai_passthrough traffic

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py

mateo-berri closed this Aug 16, 2026

mateo-berri reopened this Aug 16, 2026

veria-ai Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py

fix(proxy): negotiate client subprotocol on OpenAI websocket passthrough

862f33b

mateo-berri mentioned this pull request Aug 16, 2026

fix(proxy): register WebSocket passthrough for OpenAI prefixes (CI mirror of #36151) #37120

Closed

Merge remote-tracking branch 'origin/litellm_internal_staging' into l…

81aefe4

…itellm_pr36151_ws_passthrough # Conflicts: # litellm/proxy/pass_through_endpoints/pass_through_endpoints.py # tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py

fix(proxy): expose url join helper at module level for websocket route

4ba9d6b

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py Outdated

cursor Bot reviewed Aug 16, 2026

View reviewed changes

fix(proxy): close websocket cleanly when OpenAI credentials are missing

5965648

cursor Bot reviewed Aug 16, 2026

View reviewed changes

mateo-berri approved these changes Aug 16, 2026

View reviewed changes

mateo-berri enabled auto-merge August 16, 2026 22:43

github-actions Bot mentioned this pull request Aug 16, 2026

🦞 OpenClaw 生态日报 2026-08-17 junlinzhao327-oss/big_model_radar#371

Open

mateo-berri merged commit cfb2eba into BerriAI:litellm_internal_staging Aug 17, 2026
141 checks passed

Uh oh!

Uh oh!

Conversation

LHMQ878 commented Aug 7, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before, at merge base 4e5495e (proxy on port 28517)

After, at 5965648 (proxy on port 29226)

Subprotocol negotiation for browser clients

Model ACL enforcement, at 5965648 (same proxy, DB-backed)

Missing provider credential, at 5965648 (proxy on port 22245, no OPENAI_API_KEY configured)

Type

Changes

Final Attestation

Uh oh!

CLAassistant commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

veria-ai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

codecov Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

LHMQ878 commented Aug 7, 2026

Uh oh!

codspeed-hq Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

cursor Bot commented Aug 16, 2026

PR Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

mateo-berri commented Aug 16, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Aug 16, 2026

Uh oh!

LHMQ878 commented Aug 7, 2026 •

edited by mateo-berri

Loading

Before, at merge base `4e5495e` (proxy on port 28517)

After, at `5965648` (proxy on port 29226)

Model ACL enforcement, at `5965648` (same proxy, DB-backed)

Missing provider credential, at `5965648` (proxy on port 22245, no OPENAI_API_KEY configured)

CLAassistant commented Aug 7, 2026 •

edited

Loading

greptile-apps Bot commented Aug 7, 2026 •

edited

Loading

veria-ai Bot commented Aug 7, 2026 •

edited

Loading

codecov Bot commented Aug 7, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 7, 2026 •

edited

Loading