fix(realtime): azure Foundry realtime endpoints + self-provision e2e realtime suite by mubashir1osmani · Pull Request #32558 · BerriAI/litellm

mubashir1osmani · 2026-07-09T00:22:19Z

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Verified against a live proxy on localhost:4000 driving the real provider realtime APIs (OpenAI, Azure AI Foundry, Gemini AI Studio, Vertex AI), no mocks, real spend

Before the fix (branch at a239ab1), the Azure realtime deployment pointed at an Azure AI Foundry base that carries a project path. The handler appended the realtime path to it verbatim, producing an invalid URL, so the websocket handshake hung and the client never got a session:

constructed: wss://litellm-e2e-suite-resource.services.ai.azure.com/api/projects/<project>/openai/v1/realtime?model=gpt-realtime
Error in AzureOpenAIRealtime.async_realtime -> websockets handshake TimeoutError
client: TimeoutError: no 'session.created' within 20s; got []

After the fix (23e608a), api_base is normalized to scheme+host, the URL is correct, and the session establishes against real Azure:

constructed: wss://litellm-e2e-suite-resource.services.ai.azure.com/openai/v1/realtime?model=gpt-realtime
client: [azure-realtime] OK -> session.created

Full realtime raw-websocket suite, all four providers (text conversation and tool-call round-trip), every request reaching the real provider:

tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_text_conversation[openai] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_text_conversation[azure] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_text_conversation[gemini] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_text_conversation[vertex_ai] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_tool_call_round_trip[openai] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_tool_call_round_trip[azure] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_tool_call_round_trip[gemini] PASSED
tests/e2e/llm_translation/realtime/test_realtime_e2e.py::test_tool_call_round_trip[vertex_ai] PASSED
8 passed

Gemini and Vertex need LITELLM_GEMINI_LIVE_DEFER_SETUP=true on the gateway so the deferred-setup path emits session.updated and the full Live to GA flow normalizes; Azure Foundry needs AZURE_API_BASE/AZURE_API_KEY set to the Foundry resource

Type

🐛 Bug Fix
✅ Test

Changes

litellm/llms/azure/realtime/handler.py: the realtime URL builder appended the realtime path onto api_base as given, so an Azure AI Foundry base like https://<resource>.services.ai.azure.com/api/projects/<project> yielded a URL with the project path in front of /openai/v1/realtime and the handshake hung. _construct_url now reduces api_base to scheme and host before appending the realtime path, which leaves a plain Azure OpenAI base unchanged and makes Foundry bases work

The e2e realtime suite now provisions each provider's deployment through /model/new at session start instead of depending on a statically configured gateway model_list, so it is self-contained. The provider configs were corrected to the models that are actually served (Azure gpt-realtime, OpenAI gpt-realtime-2) and stopped passing os.environ/* references for api_base and vertex_project, which the realtime path does not unwrap; the gateway resolves those from its own environment by name. models.py gains a realtime_protocol field on the deployment body

This branch also carries the earlier rust OCR fix: litellm/ocr/main.py routed Azure Document Intelligence OCR through the generic azure_ai base because the doc-intelligence branch in _rust_bridge_api_base sat after an early return that always matched, so it now resolves AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT first, and the rust OCR e2e cases were pointed at the correct regions and env resolution

…endpoints The azure realtime handler appended the realtime path to api_base verbatim, so a Foundry base carrying a project path (.../api/projects/<name>) produced an invalid realtime URL and the websocket handshake hung. Normalize api_base to scheme and host before building the realtime path so both Azure OpenAI and Foundry bases connect Point the e2e realtime azure deployment at the GA gpt-realtime model and stop passing the os.environ refs the realtime path never unwraps, resolving them from the gateway env by name instead. Drop the local docker-compose scaffolding from the tree

greptile-apps · 2026-07-09T00:26:44Z

Greptile Summary

This PR fixes the Azure AI Foundry realtime URL construction (normalizing api_base to scheme+host before appending the realtime path) and reorganizes the e2e realtime suite to self-provision deployments via /model/new instead of depending on a statically configured gateway model_list. It also corrects the OCR _prepare_ocr_request logic so that dynamic_api_base no longer overwrites api_base for Azure Document Intelligence models, allowing _rust_bridge_api_base to correctly resolve AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT.

Realtime e2e suite is moved to tests/e2e/llm_translation/realtime/, gains a realtime_models session fixture that provisions each provider's deployment at start and deletes them on teardown, and drops the skip_if_unconfigured pattern in favour of hard-fails per the suite's contract.
OCR fix (litellm/ocr/main.py): adds _is_doc_intelligence guard to prevent dynamic_api_base from routing doc-intelligence calls to the wrong Azure endpoint.
models.py gains realtime_protocol on LiteLLMParamsBody so the Azure GA protocol flag can be passed when registering a deployment.

Confidence Score: 4/5

Safe to merge with one fix: the provisioning fixture can leak partially-created proxy deployments if an early provider raises before the try block is entered.

The realtime_models session fixture in conftest.py builds the records tuple before the try/finally block. If any client.provision() call raises after earlier providers have already been registered, those models are left on the proxy with no cleanup path. All other changes — the OCR guard, the suite reorganization, the model provisioning design — are correct and the WAV fixture path resolves correctly after the rename.

tests/e2e/llm_translation/realtime/conftest.py — provisioning loop sits outside the try/finally guard.

Important Files Changed

Filename	Overview
litellm/ocr/main.py	Adds `_is_doc_intelligence` guard so that `dynamic_api_base` no longer overwrites a doc-intelligence endpoint with a generic Azure AI base; `_rust_bridge_api_base` then correctly falls through to `AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT` when `api_base` is None.
tests/e2e/llm_translation/realtime/conftest.py	New fixture provisions realtime deployments via /model/new; partial provisioning failure before the `try` block leaves already-created models undeleted on the proxy.
tests/e2e/llm_translation/realtime/realtime_client.py	Replaces static `skip_if_unconfigured` + `configured_models()` pattern with self-provisioning via `/model/new`; `RealtimeProvider` gains `litellm_params` for deployment creation and `PROVIDERS` is updated with concrete model specs.
tests/e2e/llm_translation/realtime/test_realtime_e2e.py	Tests updated to use provisioned model names from `realtime_models` fixture instead of static aliases; otherwise unchanged.
tests/e2e/llm_translation/realtime/test_realtime_pipecat_audio_e2e.py	WAV fixture skip converted to assert (hard-fail); fixture path is correctly relative to the new location; tests updated to use provisioned model names.
tests/e2e/llm_translation/test_ocr_rust_e2e.py	Removes `vertex_project`/`vertex_credentials` os.environ refs from `VertexOcr.litellm_params` (the OCR path doesn't unwrap them); removes the vertex-deepseek test case; updates Azure AI model to `mistral-document-ai-2512`.
tests/e2e/models.py	Adds `realtime_protocol: str
tests/e2e/e2e_gateway.py	Adds `list_files` and `list_fine_tuning_jobs` gateway helpers; unrelated to the realtime/OCR fix but straightforward.

_{Reviews (2): Last reviewed commit: "fix(ocr): route azure_ai doc-intelligenc..." | Re-trigger Greptile}

codecov · 2026-07-09T00:28:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-09T00:43:20Z

Merging this PR will not alter performance

✅ 30 untouched benchmarks

_{Comparing mubashir1osmani:litellm_fix_failing (07218cb) with litellm_internal_staging (9d74548)}

ishaan-berri

small changes requested

…iscovery suite The discovery endpoints suite calls client.gateway.list_files and list_fine_tuning_jobs, which did not exist on Gateway, so both tests errored with AttributeError before reaching the proxy. Add the two GET wrappers using the existing FileListResponse / FineTuningJobsResponse models

The azure realtime handshake failure was a config issue, not a litellm bug: the realtime base was set to the Azure AI Foundry project endpoint (.../api/projects/<p>), but the OpenAI-compatible realtime route lives at the resource root. litellm correctly appends the realtime path to whatever base it is given, so pointing the realtime deployment at the resource root is the fix and no core change is needed

…source get_llm_provider inherits AZURE_AI_API_BASE into api_base for every azure_ai/* OCR model, but Azure Document Intelligence is a separate resource reached via AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, so doc-intelligence requests went to the wrong host. Stop inheriting the azure_ai base for doc-intelligence models so api_base stays unset and both the rust bridge and the python get_complete_url fall back to the document-intelligence endpoint. This drops the earlier _rust_bridge_api_base reorder, which only covered the rust path and let the env silently override an explicit api_base

mubashir1osmani · 2026-07-09T04:09:58Z

@greptile-apps

mubashir1osmani · 2026-07-09T04:12:47Z

closing in favor of #32577 to run CI

mubashir1osmani added 3 commits July 8, 2026 16:00

fix: rust ocr tests finally pass

0b1d328

fix: move realtime dir

a239ab1

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread litellm/ocr/main.py Outdated

ishaan-berri requested changes Jul 9, 2026

View reviewed changes

Comment thread litellm/ocr/main.py

Comment thread litellm/llms/azure/realtime/handler.py Outdated

mubashir1osmani added 3 commits July 8, 2026 18:24

mubashir1osmani mentioned this pull request Jul 9, 2026

test: litellm fix failing tests #32577

Merged

4 tasks

mubashir1osmani closed this Jul 9, 2026

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread tests/e2e/llm_translation/realtime/conftest.py

Navigation Menu

Uh oh!

Uh oh!

fix(realtime): azure Foundry realtime endpoints + self-provision e2e realtime suite - #32558

fix(realtime): azure Foundry realtime endpoints + self-provision e2e realtime suite#32558
mubashir1osmani wants to merge 6 commits into
BerriAI:litellm_internal_stagingfrom
mubashir1osmani:litellm_fix_failing

mubashir1osmani commented Jul 9, 2026

Uh oh!

greptile-apps Bot commented Jul 9, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

codecov Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

ishaan-berri left a comment

Uh oh!

Uh oh!

Uh oh!

mubashir1osmani commented Jul 9, 2026

Uh oh!

mubashir1osmani commented Jul 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mubashir1osmani commented Jul 9, 2026

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Changes

Uh oh!

greptile-apps Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

Uh oh!

codecov Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

ishaan-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mubashir1osmani commented Jul 9, 2026

Uh oh!

mubashir1osmani commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Jul 9, 2026 •

edited

Loading

codecov Bot commented Jul 9, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 9, 2026 •

edited

Loading

mubashir1osmani commented Jul 9, 2026 •

edited

Loading