Litellm dev 04 02 2026 p1

[krrish-berri-2]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 3, 2026 4:25am

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_dev_04_02_2026_p1 (9db062c) with main (1e5b79d)}

[greptile-apps]

Greptile Summary

This PR fixes web search cost tracking for Anthropic Claude models — both direct API and via Vertex AI — by ensuring the server_tool_use.web_search_requests field is recognised inside the ModelResponse branch and by routing Vertex AI Claude models to the Anthropic cost calculator instead of the Gemini one. The remaining changes are cosmetic: replacing a real internal Azure endpoint hostname (my-endpoint-sweden-berri992.openai.azure.com) with a generic placeholder across docs and tests.

Key changes:

• tool_call_cost_tracking.py: adds a server_tool_use.web_search_requests guard inside the isinstance(response_object, ModelResponse) branch so Claude's usage data triggers _handle_web_search_cost()
• llms/__init__.py: in the vertex_ai branch of get_cost_for_web_search_request, detects Claude models via model_info[\"key\"] and delegates to get_cost_for_anthropic_web_search instead of the Gemini calculator
• tests/llm_translation/test_azure_openai.py + three doc files: sanitise the real Azure hostname to a generic .invalid placeholder

Issues found:

• The new ModelResponse + Anthropic/Vertex AI Claude paths have no unit tests; the existing test_get_cost_for_anthropic_web_search exercises the response_object=None fallback, not the new branch — this is a gap given the PR checklist's hard requirement of ≥1 test for new functionality
• The server_tool_use.web_search_requests is not None guard returns True for web_search_requests == 0, causing a spurious "web search detected" signal (cost remains 0 but the detection is semantically wrong)

Confidence Score: 4/5

Safe to merge after adding unit tests for the new ModelResponse/Vertex AI Claude paths; no security or data-integrity risks.

The core logic change is small and targeted. The main blocker is the absence of tests for the two new branches (ModelResponse + server_tool_use, and vertex_ai/claude routing), which the CLAUDE.md rules treat as a hard requirement. The is not None vs > 0 guard is a minor semantic issue with no monetary side effect. Score 4 rather than 5 because the P1 testing gap should be addressed before merge.

litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py and litellm/llms/__init__.py both need corresponding unit tests in tests/test_litellm/

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py	Adds server_tool_use.web_search_requests detection inside the ModelResponse branch; logic is correct but guard should be > 0 to avoid false positives when web_search_requests == 0
litellm/llms/init.py	Routes Vertex AI Claude models to the Anthropic web-search cost calculator via model_info["key"]; logic is sound but uses an inline import and has no dedicated unit test for the new routing branch
tests/llm_translation/test_azure_openai.py	Replaces real Azure endpoint hostnames with generic fake-azure-endpoint.invalid placeholder — good privacy improvement, no logic changes
docs/my-website/docs/providers/azure/azure_responses.md	Sanitizes real Azure endpoint URL in documentation examples — purely cosmetic improvement
docs/my-website/docs/proxy/team_model_add.md	Sanitizes real Azure endpoint URL in documentation examples — purely cosmetic improvement
docs/my-website/docs/realtime.md	Sanitizes real Azure endpoint URL in commented-out documentation example — purely cosmetic improvement

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[response_object_includes_web_search_call] --> B{ModelResponse?}
    B -- Yes --> C{has url_citation annotations?}
    C -- Yes --> D[return True]
    C -- No --> E{usage != None?}
    E -- Yes --> F{prompt_tokens_details.web_search_requests != None? Vertex AI Gemini}
    F -- Yes --> D
    F -- No --> G{server_tool_use.web_search_requests != None? NEW: Anthropic / Vertex AI Claude}
    G -- Yes --> D
    G -- No --> H[return False]
    E -- No --> H
    B -- No: ResponsesAPIResponse --> I[check output_type == web_search_call]
    B -- No: other --> J{usage != None?}
    J -- Yes --> K{server_tool_use or prompt_tokens_details?}
    K -- Yes --> D
    K -- No --> H
    J -- No --> H
    D --> L[get_cost_for_web_search_request]
    L --> M{custom_llm_provider?}
    M -- anthropic --> N[get_cost_for_anthropic_web_search]
    M -- vertex_ai + claude in key --> N
    M -- vertex_ai + not claude --> O[cost_per_web_search_request_vertex_ai Gemini calculator]
    M -- gemini --> P[cost_per_web_search_request Gemini calculator]

Loading

Comments Outside Diff (1)

1. tests/test_litellm/litellm_core_utils/llm_cost_calc/test_tool_call_cost_tracking.py, line 122-139 (link)

   New ModelResponse branch for Anthropic/Vertex AI Claude has no test coverage

   This PR adds two new code paths to response_object_includes_web_search_call:

   1. isinstance(response_object, ModelResponse) → new server_tool_use check (tool_call_cost_tracking.py lines 337–345)
   2. custom_llm_provider.startswith("vertex_ai") + "claude" in model_key → route to Anthropic calculator (llms/__init__.py lines 40–47)

   The existing test_get_cost_for_anthropic_web_search passes response_object=None, so it exercises the elif usage is not None fallback path that already existed — not the new isinstance(response_object, ModelResponse) branch. The Vertex AI / Claude routing path also has no corresponding test.

   Per CLAUDE.md: "Always add tests when adding new entity types or features — if the existing test file covers other entity types, add corresponding tests for the new one."

   A minimal test to cover both gaps would look like:

   def test_response_object_includes_web_search_call_anthropic_model_response():
       from litellm.types.utils import Choices, Message, ServerToolUse, Usage
   
       response = ModelResponse(
           id="test-id",
           choices=[
               Choices(finish_reason="stop", index=0, message=Message(content="Hi", role="assistant"))
           ],
           created=1234567890,
           model="claude-3-5-sonnet-20241022",
           object="chat.completion",
       )
       usage = Usage(server_tool_use=ServerToolUse(web_search_requests=1))
       assert StandardBuiltInToolCostTracking.response_object_includes_web_search_call(
           response, usage
       ) is True
   
   
   def test_get_cost_for_vertex_ai_claude_web_search():
       from litellm.types.utils import ServerToolUse, Usage
   
       usage = Usage(server_tool_use=ServerToolUse(web_search_requests=1))
       cost = StandardBuiltInToolCostTracking.get_cost_for_built_in_tools(
           model="claude-3-5-sonnet-v2@20241022",
           usage=usage,
           response_object=None,
           standard_built_in_tools_params=None,
           custom_llm_provider="vertex_ai",
       )
       assert cost > 0.0

_{Reviews (1): Last reviewed commit: "fix: Anthropic web search cost not track..." | Re-trigger Greptile}

[greptile-apps]

web_search_requests == 0 triggers web-search cost path unnecessarily

The guard usage.server_tool_use.web_search_requests is not None returns True even when web_search_requests == 0 (zero actual searches). This causes response_object_includes_web_search_call to report True for a response that contained no web searches at all.

While downstream get_cost_for_anthropic_web_search would compute 0 * rate = 0.0, returning True here is semantically incorrect and mirrors a pre-existing gap also present in the Gemini branch. The parallel Gemini check (prompt_tokens_details.web_search_requests is not None) has the same issue, so at minimum be consistent — but consider tightening both to > 0:

Suggested change

	# Anthropic Claude (direct API and Vertex AI) uses server_tool_use.web_search_requests.
	# Without this check, Claude ModelResponse always falls through to return False
	# and _handle_web_search_cost() is never called.
	if (
	hasattr(usage, "server_tool_use")
	and usage.server_tool_use is not None
	and usage.server_tool_use.web_search_requests is not None
	):
	return True
	if (
	hasattr(usage, "server_tool_use")
	and usage.server_tool_use is not None
	and usage.server_tool_use.web_search_requests is not None
	and usage.server_tool_use.web_search_requests > 0
	):
	return True

Evidence: test_anthropic_chat_transformation.py line 1208 explicitly asserts usage.server_tool_use.web_search_requests == 0 as a valid server-returned value.

[greptile-apps]

Inline import inside function body

The new import at line 42 (from .anthropic.cost_calculation import get_cost_for_anthropic_web_search) is placed inside the function body. CLAUDE.md requires module-level imports unless inline placement is strictly required to avoid a circular import.

All other branches in this file already follow the same inline-import pattern, so this is consistent — but if refactoring ever brings the imports to module-level, this new one should move too. Consider a top-of-file import with a conditional guard, or document the circular-import reason in a comment, so future contributors know the inline pattern is intentional.

Context Used: CLAUDE.md (source)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}