Litellm oss staging 03 10 2026

[RheagalFire]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[greptile-apps]

Greptile Summary

This is a large staging-to-main sync commit (88 files, ~11 755 net line changes) bundling several independent features and fixes: Responses API support for JSON-declared OpenAI-like providers, a new Gemini 2.0+ schema builder for tool parameters, PANW Prisma AIRS MCP guardrail expansion, a lookup-table refactor of map_finish_reason, credential endpoint consolidation, and various provider-specific fixes.

Key issues found:

• Fireworks AI model-listing URL is broken — removing the /v1-stripping step causes the default api_base (https://api.fireworks.ai/inference/v1) to produce …/v1/v1/accounts/… in get_models(), resulting in 404 errors for all model-listing calls.
• Proxy startup crash — /credentials/by_model/{model_id} route registers credential_name: str = Path(...), but {credential_name} is not a path variable in that route template. FastAPI will raise FastAPIError: Path parameter 'credential_name' is not found in route on startup, preventing the proxy from booting.
• Async in-memory cache TTL regression — default_in_memory_ttl is no longer propagated to async_set_cache / async_set_cache_pipeline, while the synchronous set_cache still applies it. Async cache writes (the common path) will now store items without expiry, causing unbounded memory growth and stale data for users who set default_in_memory_ttl.
• LiteLLM-internal extra_body keys forwarded to Vertex AI — removing _LITELLM_INTERNAL_EXTRA_BODY_KEYS means cache and tags from extra_body are now merged into the Vertex AI request body, triggering 400 validation errors.
• Non-retryable errors now exhaust all retries — the early-exit guard for non-retryable errors (e.g. HTTP 400 context-window-exceeded) was removed from the router retry loop, and its test file was deleted.
• Anthropic multi-tool-call regression — the consecutive function_call merging logic (required to keep all tool_use blocks in a single assistant message for Anthropic) was deleted from the Responses → Chat Completion transformation path.
• Unused get_args import in litellm/litellm_core_utils/core_helpers.py.

Confidence Score: 2/5

• This PR contains multiple regressions that will break runtime behaviour for existing users — including a proxy startup crash and a broken Fireworks model-listing URL — and should not be merged without addressing the identified issues.
• Three of the issues are straight-up bugs that will be immediately visible in production: the FastAPI Path mismatch crashes the proxy on startup, the Fireworks URL produces a 404 for get_models, and the async cache no longer respects default_in_memory_ttl. Two additional regressions (Vertex AI internal key leakage and Anthropic consecutive tool-call merging) affect correctness for specific providers. The removal of non-retryable error short-circuit logic further degrades performance. Score 2 reflects critical startup-blocking and runtime bugs.
• litellm/proxy/credential_endpoints/endpoints.py (startup crash), litellm/llms/fireworks_ai/chat/transformation.py (URL bug), litellm/caching/dual_cache.py (TTL regression), litellm/llms/vertex_ai/gemini/transformation.py (internal key leak), litellm/responses/litellm_completion_transformation/transformation.py (Anthropic tool-call regression)

Important Files Changed

Filename	Overview
litellm/llms/fireworks_ai/chat/transformation.py	URL construction bug: removing the /v1 stripping causes a doubled path segment (/v1/v1/) in the Fireworks model-listing URL
litellm/proxy/credential_endpoints/endpoints.py	Route consolidation bug: /credentials/by_model/{model_id} route declares credential_name: str = Path(...) but {credential_name} is not in that route template, causing a FastAPI startup error
litellm/caching/dual_cache.py	Async TTL regression: default_in_memory_ttl is no longer propagated to async_set_cache and async_set_cache_pipeline, while sync set_cache still applies it — inconsistent TTL behavior
litellm/llms/vertex_ai/gemini/transformation.py	Removed _LITELLM_INTERNAL_EXTRA_BODY_KEYS filter — cache and tags from extra_body can now be forwarded to Vertex AI API, causing unrecognized field errors
litellm/litellm_core_utils/core_helpers.py	Refactored map_finish_reason to use a lookup table; get_args is imported but never used; MALFORMED_FUNCTION_CALL and FINISH_REASON_UNSPECIFIED now map to stop instead of their previous custom values
litellm/llms/bedrock/chat/converse_transformation.py	Removed completion_tokens_details (reasoning token breakdown) from Bedrock usage response; removed output_config (snake_case) cleanup from inference params
litellm/utils.py	Provider lookup priority reversed: Python classes now take precedence over JSON providers; get_provider_responses_api_config extended to support JSON providers for Responses API
litellm/router.py	Removed early-exit logic for non-retryable errors in retry loop — errors like 400 context-window-exceeded will now exhaust all retries instead of failing fast
litellm/llms/openai_like/dynamic_config.py	Added create_responses_config_class for dynamically generating Responses API configs from JSON provider declarations; classes are cached per slug
litellm/responses/litellm_completion_transformation/transformation.py	Removed consecutive function_call merging logic for multi-turn tool calls; this was required by Anthropic to avoid "tool_use ids found without tool_result blocks" rejection

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Incoming Request] --> B{Provider Lookup\nProviderConfigManager}
    B -->|Python class exists| C[Python Config Class\nhas priority]
    B -->|No Python class| D[JSON Provider Registry\nopenai_like]
    
    C --> E{Endpoint Type}
    D --> E

    E -->|Chat Completion| F[chat/transformation.py]
    E -->|Responses API| G[responses/transformation.py]
    
    F --> H{Provider-specific transform}
    H -->|Vertex AI| I[_map_function\nGemini 2.0+ → _build_vertex_schema_for_gemini_2\nGemini 1.5 → _build_vertex_schema]
    H -->|OpenAI GPT-5| J[gpt_5_transformation\nnormalize reasoning_effort]
    H -->|Fireworks AI| K[FireworksAIConfig\nBUG: double /v1 in get_models URL]

    G -->|Perplexity| L[PerplexityResponsesConfig]
    G -->|JSON providers| M[OpenAILikeResponsesConfig\ncreate_responses_config_class]

    N[Response] --> O{finish_reason mapping}
    O --> P[_FINISH_REASON_MAP lookup table\ncore_helpers.py]
    P -->|unmapped| Q[default: stop + warning log]

    R[DualCache.async_set_cache] --> S{TTL source}
    S -->|explicit kwarg| T[Use provided TTL]
    S -->|no kwarg| U[BUG: default_in_memory_ttl\nno longer applied async]

    style K fill:#ff9999
    style U fill:#ff9999

Loading

Comments Outside Diff (7)

1. litellm/llms/fireworks_ai/chat/transformation.py, line 429-431 (link)

   Double /v1 segment in Fireworks model-listing URL

   The default api_base returned by _get_openai_compatible_provider_info is "https://api.fireworks.ai/inference/v1". The old code explicitly stripped the trailing /v1 before appending /v1/accounts/.... Now that stripping is gone, the constructed URL becomes:

   https://api.fireworks.ai/inference/v1/v1/accounts/{account_id}/models
   

   instead of the correct:

   https://api.fireworks.ai/inference/v1/accounts/{account_id}/models
   

   This will result in a 404 / error for all calls to get_models() when using the default API base.

2. litellm/caching/dual_cache.py, line 343-356 (link)

   default_in_memory_ttl no longer applied in async cache writes

   The synchronous set_cache (line ~99) still applies default_in_memory_ttl when no explicit ttl is provided:

   if "ttl" not in kwargs and self.default_in_memory_ttl is not None:
       kwargs["ttl"] = self.default_in_memory_ttl

   But this PR removed the same guard from both async_set_cache and async_set_cache_pipeline. All async cache writes (the hot path for LiteLLM's async operations) will now ignore default_in_memory_ttl, causing items to be stored without TTL in the in-memory cache. This means stale entries will never expire, leading to unbounded memory growth and stale data being returned for users who configured default_in_memory_ttl.

3. litellm/llms/vertex_ai/gemini/transformation.py, line 586-595 (link)

   LiteLLM-internal extra_body keys now forwarded to Vertex AI

   The removed _LITELLM_INTERNAL_EXTRA_BODY_KEYS = frozenset({"cache", "tags"}) filter was intentionally preventing the cache and tags keys (consumed internally by LiteLLM) from being forwarded to the Vertex AI API. Without the filter, any caller who passes extra_body={"cache": {...}} or extra_body={"tags": [...]} will have those keys merged into the outgoing Vertex AI request body. The Gemini API does not recognise these fields and will return a 400 validation error.

   These keys should either be filtered out here, or stripped upstream before _pop_and_merge_extra_body is invoked.

4. litellm/litellm_core_utils/core_helpers.py, line 3 (link)

   Unused import: get_args

   get_args is imported from typing but is never referenced anywhere in this file after the refactor.

5. litellm/router.py, line 5505-5519 (link)

   Non-retryable errors now exhaust all retries before raising

   The removed block short-circuited the retry loop for errors that should_retry_this_error identifies as non-retryable (e.g. HTTP 400 context-window-exceeded, 401 auth errors). Without it, these errors will now go through all num_retries iterations — burning time and model credits — before finally surfacing to the caller.

   The associated test file tests/test_litellm/test_router_retry_non_retryable_errors.py was deleted in this same PR (251 lines), removing coverage for this behaviour. Please confirm this removal is intentional and that the equivalent protection exists elsewhere in the retry path.

6. litellm/responses/litellm_completion_transformation/transformation.py, line 380-415 (link)

   Removal of consecutive function_call merging breaks Anthropic multi-tool calls

   The deleted block merged back-to-back assistant messages that each contained a tool_use block into a single assistant message before forwarding to the chat-completion layer. This transformation was explicitly documented as required for Anthropic, which rejects requests where tool_use IDs appear in separate assistant messages:

   "tool_use ids were found without tool_result blocks immediately after"

   Without this merge, multi-turn Responses-API calls that produce multiple sequential tool calls in the same assistant turn will generate separate assistant messages. When those messages are forwarded to Anthropic via litellm.completion, the API will reject the request.

   Please add a test that exercises the multi-tool-call path against an Anthropic provider to confirm this regression is not present.

7. litellm/proxy/credential_endpoints/endpoints.py, line 145-157 (link)

   Path(...) parameter name mismatch causes FastAPI startup error

   The route template /credentials/by_model/{model_id} contains the path variable {model_id}, but the function declares credential_name: str = Path(...) — a required path parameter whose name is credential_name. FastAPI validates at startup that every Path(...) parameter name appears in the route template, and raises:

   FastAPIError: Path parameter 'credential_name' is not found in route '/credentials/by_model/{model_id}'
   

   For the /credentials/by_name/{credential_name:path} route the binding is correct, but the second route registration will crash the proxy on boot.

   The fix is to either keep the two endpoints as separate functions (one for each route) or make model_id an explicit Path(...) parameter for the /credentials/by_model/{model_id} route and use a query param or different mechanism to pass credential_name.

_{Last reviewed commit: f243e56}

[greptile-apps]

Standalone if breaks the elif chain

The fallback check at line 110 is a standalone if, not an elif. While this is currently correct because every preceding branch unconditionally returns, it visually disconnects the guard from the if/elif chain above and becomes a latent bug the moment any future branch is added without a return (e.g. a branch that only does a side-effect and falls through).

Using elif here makes the intent explicit and prevents the guard from being accidentally "re-entered" from an earlier branch:

Suggested change

	if finish_reason not in _VALID_OPENAI_FINISH_REASONS:
	verbose_logger.warning(
	"litellm.map_finish_reason: unknown finish_reason %r from provider; "
	"mapping to 'finish_reason_unspecified' to avoid ValidationError.",
	finish_reason,
	)
	return "finish_reason_unspecified"
	elif finish_reason not in _VALID_OPENAI_FINISH_REASONS:
	verbose_logger.warning(
	"litellm.map_finish_reason: unknown finish_reason %r from provider; "
	"mapping to 'finish_reason_unspecified' to avoid ValidationError.",
	finish_reason,
	)
	return "finish_reason_unspecified"

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Error		Mar 11, 2026 1:22pm