fix(azure/passthrough): populate standard_logging_object via logging hook - #25679
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fixes a real observability gap: Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| litellm/llms/azure/passthrough/transformation.py | Adds logging_non_streaming_response for chat/completions, mirroring Bedrock's pattern using OpenAIGPTConfig; two style-level findings: inline imports and a placeholder message logged as the user input. |
| tests/test_litellm/llms/azure/passthrough/test_azure_passthrough_transformation.py | New unit tests with no real network calls; covers the happy path (chat/completions → ModelResponse) and the pass-through path (other endpoints → None). Clean and well-scoped. |
Sequence Diagram
sequenceDiagram
participant Client
participant Router as allm_passthrough_route
participant Logging as LiteLLMLogging
participant AzurePT as AzurePassthroughConfig
participant OpenAI as OpenAIGPTConfig
Client->>Router: POST /azure/…/chat/completions
Router->>Logging: success_handler(httpx_response)
Logging->>Logging: normalize_logging_result(result)
Logging->>AzurePT: logging_non_streaming_response(model, httpx_response, endpoint)
alt endpoint contains "chat/completions"
AzurePT->>OpenAI: transform_response(model, httpx_response, …)
OpenAI-->>AzurePT: ModelResponse (with usage)
AzurePT-->>Logging: ModelResponse
Logging->>Logging: build standard_logging_object
Logging->>Logging: calculate response_cost
Logging-->>Router: standard_logging_object populated
else other endpoint
AzurePT-->>Logging: None
Logging-->>Router: "standard_logging_object = None (unchanged)"
end
Reviews (1): Last reviewed commit: "fix(azure/passthrough): populate standar..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| from litellm import encoding | ||
| from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig | ||
| from litellm.types.utils import ModelResponse |
There was a problem hiding this comment.
from litellm import encoding, from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig, and from litellm.types.utils import ModelResponse are placed inside the method body. CLAUDE.md says to avoid imports within methods and only allow it to break circular imports. If circular-import avoidance is genuinely needed here (as it is in the Bedrock counterpart), adding a brief inline comment explaining why would make the intent clear. Consider moving any imports that are not circular-import-driven to the module level.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| litellm_model_response: ModelResponse = openai_chat_config.transform_response( | ||
| model=model, | ||
| messages=[{"role": "user", "content": "no-message-pass-through-endpoint"}], | ||
| raw_response=httpx_response, | ||
| model_response=ModelResponse(), | ||
| logging_obj=logging_obj, | ||
| optional_params={}, | ||
| litellm_params={}, | ||
| api_key="", | ||
| request_data=request_data, | ||
| encoding=encoding, | ||
| ) | ||
|
|
There was a problem hiding this comment.
messages=[{"role": "user", "content": "no-message-pass-through-endpoint"}] is forwarded to logging_obj.post_call(input=messages, ...). For Azure chat/completions passthrough the real messages are already in request_data["messages"], so every Datadog/cost trace will show the placeholder string as the request input instead of the actual conversation. The actual content is still reachable via additional_args, but tools that surface the primary input field will silently show wrong data. Using request_data.get("messages", [{"role": "user", "content": "no-message-pass-through-endpoint"}]) here would match reality without breaking anything:
| litellm_model_response: ModelResponse = openai_chat_config.transform_response( | |
| model=model, | |
| messages=[{"role": "user", "content": "no-message-pass-through-endpoint"}], | |
| raw_response=httpx_response, | |
| model_response=ModelResponse(), | |
| logging_obj=logging_obj, | |
| optional_params={}, | |
| litellm_params={}, | |
| api_key="", | |
| request_data=request_data, | |
| encoding=encoding, | |
| ) | |
| litellm_model_response: ModelResponse = openai_chat_config.transform_response( | |
| model=model, | |
| messages=request_data.get( | |
| "messages", | |
| [{"role": "user", "content": "no-message-pass-through-endpoint"}], | |
| ), | |
| raw_response=httpx_response, | |
| model_response=ModelResponse(), | |
| logging_obj=logging_obj, | |
| optional_params={}, | |
| litellm_params={}, | |
| api_key="", | |
| request_data=request_data, | |
| encoding=encoding, | |
| ) |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| import httpx | ||
| from httpx import Response | ||
|
|
||
| from litellm.litellm_core_utils.litellm_logging import Logging |
| if TYPE_CHECKING: | ||
| from httpx import URL | ||
|
|
||
| from litellm.types.utils import CostResponseTypes |
| endpoint: str, | ||
| ) -> Optional["CostResponseTypes"]: | ||
| from litellm import encoding | ||
| from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig |
| ) -> Optional["CostResponseTypes"]: | ||
| from litellm import encoding | ||
| from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig | ||
| from litellm.types.utils import ModelResponse |
15ef3fa
into
BerriAI:litellm_ishaan_april14
…through-standard-logging-object fix(azure/passthrough): populate standard_logging_object via logging hook
3 participants
Relevant issues
No existing GitHub issue — bug reported directly by a customer using Azure passthrough (
use_in_pass_through: true) with the Datadog callback enabled.Pre-Submission checklist
tests/test_litellm/— new filetests/test_litellm/llms/azure/passthrough/test_azure_passthrough_transformation.pywith two unit tests (populatedModelResponseonchat/completions,Nonefall-through on other endpoints).make test-unit— the new tests pass locally viauv run pytest tests/test_litellm/llms/azure/passthrough/test_azure_passthrough_transformation.py -v(2 passed). Fullmake test-unitwas not runnable on my machine due to apyproject.tomlparse error in current main (exclude-newer = "3 days") unrelated to this change; relying on CI for the full sweep.chat/completions)./openai/responsesand/openai/messagesstill returnNonefrom the hook, unchanged from today.Screenshots / Proof of Fix
Minimal reproduction: real Azure
gpt-4.1-minideployment withuse_in_pass_through: trueandcallbacks: [\"datadog\"]. RequestingPOST /azure/openai/deployments/gpt-4.1-mini/chat/completions?api-version=2025-01-01-preview.Before fix
Request returns 200 to the client, but three callbacks raise on missing `standard_logging_object` on every request — a silent observability + cost-tracking drop:
```
litellm.router.Router::deployment_callback_on_success(): Exception occured - standard_logging_object is None
Error in tracking cost callback - Cost tracking failed for model=gpt-4.1-mini.
Debug info - standard_logging_object not found
Datadog Layer Error - standard_logging_object not found in kwargs
Traceback (most recent call last):
File ".../integrations/datadog/datadog.py", line 480, in create_datadog_logging_payload
raise ValueError("standard_logging_object not found in kwargs")
ValueError: standard_logging_object not found in kwargs
```
After fix
Same request, same config — all three errors gone. Cost is calculated, and Datadog builds + queues a payload with real usage:
```
response_cost: 1.92e-05
Datadog: Logger - Logging payload = {"id": "chatcmpl-DURD...", "call_type": "allm_passthrough_route", "custom_llm_provider": "azure", "usage_object": {"completion_tokens": 10, "prompt_tokens": 8, "total_tokens": 18, ...}, "response_cost": 1.92e-05, ...}
Datadog, event added to queue. Will flush in 5 seconds...
```
The subsequent `403 Forbidden` on the Datadog intake flush is expected — the repro uses `DD_API_KEY=fake`. It confirms the payload reached the shipping layer, which was never possible before the fix.
Type
🐛 Bug Fix
Changes
Root cause. Requests routed through `Router.allm_passthrough_route` on Azure with `use_in_pass_through: true` never populate `kwargs["standard_logging_object"]`. `_success_handler_helper_fn` in `litellm_logging.py` delegates to the provider's `logging_non_streaming_response` hook to build it; `AzurePassthroughConfig` inherits the base-class no-op which returns `None`, so no standard logging object ever gets attached. Every callback that hard-requires it then raises:
Fix. Implement `AzurePassthroughConfig.logging_non_streaming_response` for the `chat/completions` endpoint, mirroring the existing Bedrock pattern in `litellm/llms/bedrock/passthrough/transformation.py:121-162`. When the hook returns a `ModelResponse`, the existing consumer branch in `_success_handler_helper_fn` builds `standard_logging_object` with real usage data and the triple-symptom disappears on one code path.
One deliberate difference from Bedrock: the method parses the raw response via `OpenAIGPTConfig().transform_response` rather than `ProviderConfigManager.get_provider_chat_config(AZURE, model)`. The latter returns `AzureOpenAIConfig`, whose `transform_response` raises `NotImplementedError` because Azure is normally routed through the OpenAI SDK rather than the BaseConfig path. Azure's `chat/completions` body is OpenAI-format JSON, so `OpenAIGPTConfig` parses it cleanly — including for o-series and GPT-5 model names.
Scope:
Files changed: