fix(azure_ai): add passthrough config so router-model relays reach the deployment's own endpoint - #39863
Conversation
…e deployment's own endpoint Every /azure_ai/<router model>/<native path> relay failed with HTTP 500 because azure_ai had no passthrough config. The new AzureAIPassthroughConfig strips the router-model prefix from the relayed path, forwards to the deployment's api_base with its own credential (api-key on Foundry and Azure OpenAI hosts, Bearer elsewhere, Entra as the fallback), and delegates chat/completions cost logging to the Azure passthrough config. The router's provider inference now receives the deployment's api_base so an OpenAI-family model on a Foundry resource stays azure_ai instead of flipping to azure through the AZURE_AI_API_BASE env var.
Greptile SummaryThis PR adds Azure AI passthrough routing so router-model relays use each deployment’s endpoint and credentials.
Confidence Score: 5/5The PR appears safe to merge, with no outstanding blocking or non-blocking findings. The latest changes address remote high-detail image accounting without fetching untrusted remote resources, and no new correctness, security, or repository-rule failures were established. The previous thread was manually resolved without an explanatory reply and is therefore not outstanding.
|
| Filename | Overview |
|---|---|
| litellm/llms/azure_ai/passthrough/transformation.py | Implements deployment-aware Azure AI passthrough URL construction, authentication, and logging integration. |
| litellm/llms/azure/passthrough/transformation.py | Corrects Azure relay path handling and adds streaming chat response reconstruction. |
| litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py | Preserves upstream HTTP responses and accepts charset-qualified JSON request bodies. |
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/openai_passthrough_logging_handler.py | Reconstructs missing streaming prompt usage while avoiding remote high-detail image downloads. |
| litellm/litellm_core_utils/token_counter.py | Adds a bounded high-detail image token estimate used when exact remote dimensions are unavailable. |
| litellm/router_utils/common_utils.py | Preserves explicit provider prefixes for generic router calls. |
| litellm/router.py | Uses segment-aware endpoint replacement and centralized generic-call provider resolution. |
Reviews (7): Last reviewed commit: "fix(passthrough): charge remote high-det..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…ure_ai_passthrough_config
…ure_ai_passthrough_config
Generic passthrough calls inferred the provider from the bare model name, so an azure_ai/gpt-* deployment on an Azure OpenAI host flipped to azure and get_llm_provider re-prefixed the deployment name into azure_ai/gpt-5.4-mini, a 404 DeploymentNotFound. provider_for_generic_call takes the declared custom_llm_provider first, then the model's own prefix, and only infers for unprefixed models
…router relays Streaming chat relays on Azure and azure_ai deployments rebuild the response from the SSE chunks through the OpenAI passthrough assembler, so the spend log carries usage. The router relays keep the JSON body when the Content-Type carries a charset, return the upstream status and body instead of a 500 when the deployment rejects the call, and fall back to the caller's api-version when the deployment sets none. Lint budgets ratcheted to the measured totals
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewThis pull request adds passthrough configuration for Azure AI router-model relays so requests reach the deployment’s own endpoint, including handling for non-chat responses. Two issues have been addressed, but non-chat parse or OCR relays can still omit provider usage data from budget accounting. An authenticated caller could repeatedly consume billable Azure AI operations without debiting key or team budgets, creating a direct but financially bounded abuse path. Open issues (1)
Fixed/addressed: 2 · PR risk: 6/10 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…ead of a default-image flag
…itellm_lit_7022_azure_ai_passthrough_config
| ) | ||
| if chat_result is not None: | ||
| return chat_result | ||
| return StandardPassThroughResponseObject(response=relayed_body(httpx_response)) |
There was a problem hiding this comment.
Medium: Non-chat relays bypass budget accounting
This wrapper discards provider billing fields such as processed pages by placing the response inside a generic object with no recognized usage. An authenticated caller can repeatedly invoke billable parse or OCR routes while response_cost remains zero or unset, so key and team budgets are not debited. Transform supported responses into their typed LiteLLM response with usage information, or explicitly calculate and attach the provider-reported cost before dispatching success callbacks.
1 participant
TLDR
Problem this solves:
/azure_ai/<router model>/<native path>relays failed on every call with HTTP 500How it solves it:
AzureAIPassthroughConfig: strips the router-model prefix, forwards to the deployment'sapi_baseapi-keyon Foundry/Azure OpenAI hosts, Bearer elsewhere, Entra as fallbackazure_ai/gpt-*deployments keep working on Foundry and Azure OpenAI hosts alikeapi-versionapi_baseends in/models, the deployment'sapi_versionfills in when the caller sends none, and anazure_ai/gpt-*deployment on a<resource>.openai.azure.comhost relays to/openai/deployments/...gptno longer corrupts agpt-5.4-minideployment segmentUser Flow
Before: a developer relaying Cohere Parse through the gateway's Azure AI passthrough gets a 500 on every call, so no document ever gets parsed
model_name: Cohere-parse-v5withmodel: azure_ai/Cohere-parse-v5, the resource'sapi_baseandapi_key, and restarts the proxy"output_format": "markdown"{"error":{"message":"Internal server error","type":"internal_server_error"}}and nothing reaches Azure"model": "foundry-gpt-5.4-mini"returns 200After: the same relay reaches the deployment's own Foundry endpoint with its own credential and returns the parsed document
"stream": truestreams SSEdata:chunks, and the same path under /azure/ works tooinput_cost_per_token/output_cost_per_token(Cohere Parse has no built-in cost entry yet, LIT-6992), and GET https://litellm-domain/key/info shows the totalContent-Type: application/json; charset=utf-8request is relayed with its bodyapi_baseends in/models, one pointing at<resource>.openai.azure.comwithapi_versionset, and one whose router name is a prefix of the deployment name (gptforgpt-5.4-mini) all relay with HTTP 200 on the same routes"model": "foundry-gpt-5.4-mini"still returns 200Relevant issues
Linear ticket
Resolves LIT-7022
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both legs ran the same proxy config with
--num_workers 2(two uvicorn workers, one process) against real Azure resources: a Cohere Parse v5 Foundry deployment and a gpt-5.4-mini deployment reached both through its Foundry host and through the same resource's<resource>.openai.azure.comhost. Before ran at 2151dcb, the litellm_internal_staging tip when the run started (the merge base, aea5358, has no Azure AI passthrough config either), on port 59723 without a database, After at this PR's tip (4706ace, which merges litellm_internal_staging at aea5358) on port 46237 with a local Postgres behindDATABASE_URLso spend logs could be read back. The parse payload (lit7022_parse_req.json) is a one-page PNG as a data URI{"model": "Cohere-parse-v5", "document": {"type": "image_url", "image_url": "data:image/png;base64,..."}, "output_format": "markdown"}Before (2151dcb)
1. Cohere Parse relay (Foundry-native path)
2. Chat relay via the model-inference path
3. Chat relay via the Azure OpenAI deployments path
4. Streaming chat relay
5. Same router model through /azure/ (shares the route)
6. Regression: unified /v1/chat/completions on the same azure_ai deployment
7. Relay to an Azure OpenAI host (api_base ends in .openai.azure.com, api_version on the deployment)
8. Deployment whose api_base already ends in /models
9. Router model name that is a prefix of the deployment name (gpt vs gpt-5.4-mini)
10. Deployment api_version fills in when the caller sends none
After (4706ace)
1. Cohere Parse relay (Foundry-native path)
2. Chat relay via the model-inference path
3. Chat relay via the Azure OpenAI deployments path
4. Streaming chat relay
5. Same router model through /azure/ (shares the route)
6. Regression: unified /v1/chat/completions on the same azure_ai deployment
7. Relay to an Azure OpenAI host (api_base ends in .openai.azure.com, api_version on the deployment)
8. Deployment whose api_base already ends in /models
9. Router model name that is a prefix of the deployment name (gpt vs gpt-5.4-mini)
10. Deployment api_version fills in when the caller sends none
After, spend logging with a generated virtual key (4706ace)
S1. Generate a virtual key limited to the deployment
S2. Non-streaming chat relay with that key
S3. Streaming chat relay with that key
S3b. Non-chat relay (Cohere Parse with custom pricing on the deployment) with that key
S4. Spend logs for the key (after the proxy's batch write)
S5. Key info shows the accumulated spend
After, relay edge cases (4706ace)
C. Bad native path returns the deployment's own error
D. Content-Type with a charset still carries the body
E. Caller's api-version reaches an Azure OpenAI deployments path
F. Wrong virtual key
At the Before commit none of the relays reached Azure (HTTP 500 on every case above), so there was nothing to bill; the spend section therefore has no Before leg
Observations from the run:
stream_options.include_usage, so the streaming spend row counts prompt tokens from the relayed messages and completion tokens from the assembled textimage_urlparts in relayed messages are charged at the high-detail upper bound (1445 tokens, the largest image high-res mode keeps) during the log flush; data URLs and low/auto detail images are counted exactly and nothing is downloaded, so a slow image host cannot stall the event loopCost tracking failed ... Add custom pricingpolicy and writes no row, like any unmapped model/azure_ai/providers/...paths still 500 at base; out of scope, left alonechat,models,deployments,parse) is rewritten wherever that word appears in the relayed path, because the Azure OpenAI deployments pattern needs every occurrence rewritten; pick router model names that are not path wordsType
🐛 Bug Fix
Caveats (if any)
Caveats (if any)
Low
_should_use_api_key_headerdecision and Azure's documented auth (api-key on Foundry and Azure OpenAI hosts, Bearer for Entra tokens and non-Azure hosts); the chat path now shares the helper so the two cannot driftchat,models,deployments,parse) is rewritten wherever that word appears in the relayed path, because the Azure OpenAI deployments pattern (/azure_ai/<group>/openai/deployments/<group>/...) needs every occurrence rewritten; the whole-segment rewrite fixes the substring case (gptvsgpt-5.4-mini) only, so name router models distinctly from path words/azure_ai/...paths still fall back to the globalAZURE_API_BASE, and Cohere Parse has no built-in cost entry yet (LIT-6992), so an unpriced parse deployment writes no spend row until it carries custom pricingerror.status_code, whichhttpx.HTTPStatusErrorlacks; every provider's router relay shares that shapeFinal Attestation