test: litellm fix failing tests by mubashir1osmani · Pull Request #32577 · BerriAI/litellm

mubashir1osmani · 2026-07-09T04:11:57Z

Relevant issues

supersedes #32558

built a patch with this PR on top of 1.92.0.rc and ran OCR tests

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

…endpoints The azure realtime handler appended the realtime path to api_base verbatim, so a Foundry base carrying a project path (.../api/projects/<name>) produced an invalid realtime URL and the websocket handshake hung. Normalize api_base to scheme and host before building the realtime path so both Azure OpenAI and Foundry bases connect Point the e2e realtime azure deployment at the GA gpt-realtime model and stop passing the os.environ refs the realtime path never unwraps, resolving them from the gateway env by name instead. Drop the local docker-compose scaffolding from the tree

…iscovery suite The discovery endpoints suite calls client.gateway.list_files and list_fine_tuning_jobs, which did not exist on Gateway, so both tests errored with AttributeError before reaching the proxy. Add the two GET wrappers using the existing FileListResponse / FineTuningJobsResponse models

The azure realtime handshake failure was a config issue, not a litellm bug: the realtime base was set to the Azure AI Foundry project endpoint (.../api/projects/<p>), but the OpenAI-compatible realtime route lives at the resource root. litellm correctly appends the realtime path to whatever base it is given, so pointing the realtime deployment at the resource root is the fix and no core change is needed

…source get_llm_provider inherits AZURE_AI_API_BASE into api_base for every azure_ai/* OCR model, but Azure Document Intelligence is a separate resource reached via AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, so doc-intelligence requests went to the wrong host. Stop inheriting the azure_ai base for doc-intelligence models so api_base stays unset and both the rust bridge and the python get_complete_url fall back to the document-intelligence endpoint. This drops the earlier _rust_bridge_api_base reorder, which only covered the rust path and let the env silently override an explicit api_base

greptile-apps · 2026-07-09T04:16:18Z

Greptile Summary

This PR fixes two OCR routing bugs (Azure Document Intelligence api_base hijacking, Vertex OCR credential literal-string injection) and refactors the realtime e2e suite to self-provision models via the gateway API rather than depending on a static model list.

OCR api_base fix: caller_supplied_api_base is captured before get_llm_provider runs; the AZURE_AI_API_BASE env fallback is suppressed only for Document Intelligence models when the caller did not supply an explicit api_base. The is_azure_document_intelligence_model helper in common_utils.py is now the single source of truth for that routing decision. Three mock regression tests in tests/test_litellm/ocr/ pin all three cases.
Realtime suite reorganised from tests/e2e/realtime/ into tests/e2e/llm_translation/realtime/; the configured_models/skip_if_unconfigured pattern is replaced by a session-scoped realtime_models fixture that provisions each provider through /model/new and hard-fails (never skips) on a missing provider. xai is commented out with a TODO, mirroring the existing Bedrock treatment.

Confidence Score: 4/5

The OCR production changes are well-guarded by dedicated regression tests; the realtime reorganisation touches no production code. The silent vertex-deepseek test removal (unaddressed from a prior review) and the unexplained azure-ai model bump leave a coverage gap worth resolving before merge.

The two production file changes are correct and covered by the new mock test suite. The remaining concern is test_ocr_rust_e2e.py, which still silently removes the vertex-deepseek OCR case with no comment — a regression in that provider path would now go undetected.

tests/e2e/llm_translation/test_ocr_rust_e2e.py — the vertex-deepseek removal and the unexplained azure-ai model version bump both deserve clarification before merge.

Important Files Changed

Filename	Overview
litellm/llms/azure_ai/ocr/common_utils.py	Added `is_azure_document_intelligence_model` helper as the single source of truth for sub-route detection; `get_azure_ai_ocr_config` now delegates to it instead of repeating the string check.
litellm/ocr/main.py	Captures `caller_supplied_api_base` before `get_llm_provider` runs and uses it to suppress the AZURE_AI_API_BASE env fallback only for Document Intelligence models where the caller did not provide an explicit api_base. Explicit api_base now always wins.
tests/test_litellm/ocr/test_ocr_azure_document_intelligence_api_base.py	New mock-only regression suite pinning three key api_base resolution cases: env fallback suppressed for doc-intelligence, explicit api_base honoured, non-doc-intelligence still uses AZURE_AI_API_BASE. No real network calls; correct location for tests/test_litellm/.
tests/e2e/llm_translation/test_ocr_rust_e2e.py	Removes vertex-deepseek test case without explanation (flagged in prior review, still unaddressed); bumps azure-ai model from 2505 to 2512 without comment; correctly strips erroneous os.environ/* vertex_project/credentials refs that would have been passed literally to the URL.
tests/e2e/llm_translation/realtime/realtime_client.py	Refactored from static model-list lookup to provision-based deployment: provision() registers each provider through /model/new with marker-unique names; skip_if_unconfigured replaced by realtime_model() which asserts instead of skipping. xai properly commented out with TODO.
tests/e2e/llm_translation/realtime/conftest.py	New session-scoped realtime_models fixture that provisions all providers at session start and cleans up via try/finally.
tests/e2e/e2e_gateway.py	Added list_files and list_fine_tuning_jobs gateway client methods using existing transport helpers and typed response models.
tests/e2e/models.py	Added realtime_protocol: str

_{Reviews (2): Last reviewed commit: "refactor(ocr): consolidate azure doc-int..." | Re-trigger Greptile}

…licit api_base Extract is_azure_document_intelligence_model as the single source of truth for the azure_ai doc-intelligence sub-route so the check is no longer duplicated across _prepare_ocr_request and _rust_bridge_api_base, and gate the dynamic_api_base suppression on the caller not supplying an api_base so an explicit endpoint is always honoured. Restore xai to the realtime PROVIDERS as a documented disabled entry instead of dropping it silently, and add a regression test pinning doc-intelligence api_base resolution. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

CLAassistant · 2026-07-09T04:28:07Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mubashir1osmani
❌ Mubashir Osmani

Mubashir Osmani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

mubashir1osmani · 2026-07-09T04:29:51Z

@greptile-apps

codecov · 2026-07-09T04:39:32Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-09T04:49:43Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_fix_failing (4808489) with litellm_internal_staging (1d9a86e)}

deepseek-reasoner honors thinking={"type":"disabled"} on the live API, but map_openai_params dropped both that and reasoning_effort="none" without emitting any disable signal, so reasoning stayed on. Map reasoning_effort "none" to disabled and pass an explicit thinking disabled through. The two e2e disable tests now pass, so their xfail(strict) markers are removed (LIT-3686)

The discovery suite lists a provider's fine-tuning jobs, which 500s with "fine_tuning_config is not set" until finetune_settings is present. It is a startup-only config with no management endpoint, so add an openai entry to the e2e gateway config

…e bypass) service_tier="flex" returned a transient 429 when OpenAI flex capacity was unavailable; switch to "priority", which this account echoes back reliably and is still distinct from litellm's default injection (service_tier="auto" reports "default"). The bedrock prompt-cache test saw cache_read=0 because the proxy response cache served the first response for the identical second request; send cache={"no-cache": true} so the second call reaches Bedrock and reads the provider prompt cache

On a red e2e run, hand the failing tests and their tracebacks to Devin through the proxy's own MCP gateway (POST /mcp-rest/tools/call) so it files a Linear ticket per failure and opens fix PRs. Wired as a best-effort pytest_sessionfinish step, opt-in via E2E_DEVIN_REMEDIATION=1 (DEVIN_DRY_RUN=1 to preview), so local runs are unaffected. The runner image already ships tests/e2e, so no image or chart change is needed; the proxy holds DEVIN_API_KEY and injects it upstream

devin-ai-integration · 2026-07-09T19:15:34Z

The failing CodSpeed check is test_completion_multi_turn (-25%), which benchmarks litellm.completion(model="gpt-4o", mock_response=...). Nothing in this PR touches that hot path: the OCR change is a separate endpoint, and the deepseek map_openai_params change only runs for deepseek models. In the same run the other two completion benchmarks improved (+42% and +31%), and the previous commit's CodSpeed report flagged "Different runtime environments detected", so this up/down swing is runner variance rather than a real regression. Acknowledging on CodSpeed / re-running rather than a code change

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mubashir1osmani added 6 commits July 8, 2026 16:00

fix: rust ocr tests finally pass

0b1d328

fix: move realtime dir

a239ab1

mubashir1osmani mentioned this pull request Jul 9, 2026

fix(realtime): azure Foundry realtime endpoints + self-provision e2e realtime suite #32558

Closed

4 tasks

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread litellm/ocr/main.py Outdated

Comment thread litellm/ocr/main.py Outdated

Comment thread tests/e2e/llm_translation/realtime/realtime_client.py

mubashir1osmani enabled auto-merge (squash) July 9, 2026 05:01

mubashir1osmani added 4 commits July 8, 2026 22:47

yassin-berriai mentioned this pull request Jul 9, 2026

refactor(otel): move litellm error detail keys under the litellm.* namespace #32591

Merged

4 tasks

Mubashir Osmani and others added 2 commits July 9, 2026 19:43

ci: re-trigger benchmark run (codspeed variance)

f3f8e41

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

ci: re-trigger flaky google_generate_content check

4808489

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yuneng-berri approved these changes Jul 9, 2026

View reviewed changes

mubashir1osmani merged commit 8519d7f into litellm_internal_staging Jul 9, 2026
125 checks passed

mubashir1osmani deleted the litellm_fix_failing branch July 9, 2026 20:54

Navigation Menu

Uh oh!

Uh oh!

test: litellm fix failing tests - #32577

test: litellm fix failing tests#32577
mubashir1osmani merged 13 commits into
litellm_internal_stagingfrom
litellm_fix_failing

mubashir1osmani commented Jul 9, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 9, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CLAassistant commented Jul 9, 2026

Uh oh!

mubashir1osmani commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026

Uh oh!

codspeed-hq Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mubashir1osmani commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Changes

Uh oh!

greptile-apps Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CLAassistant commented Jul 9, 2026

Uh oh!

mubashir1osmani commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

devin-ai-integration Bot commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mubashir1osmani commented Jul 9, 2026 •

edited

Loading

greptile-apps Bot commented Jul 9, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 9, 2026 •

edited

Loading