fix: restore commits from revert of PR #21601

[Chesars]

Summary

PR #23313 reverted all ~50 commits from PR #21601 to clean up the staging diff. However, only the model_cost aliases feature was re-applied (#23314). The remaining fixes from other contributors were lost when staging was merged to main (#23276).

This PR reverts the revert (#23313) to restore all lost commits, including:

• DualCache async TTL fix (fix(caching): inject default_in_memory_ttl in DualCache async_set_cache and async_set_cache_pipeline #22241)
• Router retry loop break on non-retryable errors (fix(router): break retry loop on non-retryable errors #21370)
• Bedrock converse completion_tokens_details + output_config stripping (fix(bedrock): populate completion_tokens_details in Responses API #23243, fix(bedrock): strip output_config from Converse requests; fix spend tracking redaction test #23240)
• Fireworks AI duplicate /v1 fix (fix(fireworks): strip duplicate /v1 from models endpoint URL #23113)
• Duration parser month overflow fix (fix: handle month overflow in duration_in_seconds for multi-month durations #23099)
• Redact messages standard_logging_object helper (fix(logging): preserve ModelResponse choices format in redacted standard_logging_object + add Charity Engine provider endpoint #23235)
• ResponseApplyPatchToolCall + streaming usage preservation (fix handling of ResponseApplyPatchToolCall in completion bridge #20913, Streaming responses from Responses API bridge drops prompt_tokens_details (cached_tokens always None) #22192)
• Sagemaker embedding role assumption
• VirtualKeysTable refetch button
• MCP server OpenAPI improvements
• Various test additions

Conflicts with #23314 (aliases) and #23151 (gpt_5_transformation) were resolved keeping the current (newer) versions.

Test plan

• Verify make test-unit passes
• Verify no duplicate code from already-applied PRs (fix(openai): preserve reasoning_effort summary field for Responses API #23151, feat: add model_cost aliases expansion support #23314)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Mar 12, 2026 4:44pm

[greptile-apps]

Greptile Summary

This PR restores ~50 commits that were lost when PR #23313 reverted PR #21601 to clean a staging diff — the aliases feature (PR #23314) and GPT-5 transformation (PR #23151) that had already been re-applied are intentionally excluded here. The changes land a broad set of independent bug fixes and small features across the caching, routing, provider transformation, proxy, and UI layers.

Key changes:

• DualCache async TTL — default_in_memory_ttl was not propagated to the async set_cache / set_cache_pipeline methods; now fixed.
• Router retry loop — Non-retryable errors (e.g. 400 ContextWindowExceeded) surfaced during a retry attempt are now raised immediately instead of being swallowed and replaced by the original error; original_exception is also updated on every iteration so the last error is always re-raised when retries are exhausted.
• Lowest-latency strategy — Average latency was always divided by len(item_latency) even in streaming mode, where item_ttft_latency should be used; this could produce a ZeroDivisionError or wrong averages.
• Duration parser — Month overflow (+2m from November, etc.) was only handled for December; now uses modular arithmetic for full correctness.
• Bedrock converse — Adds completion_tokens_details (with estimated reasoning_tokens) to usage output.
• Fireworks AI — Removes duplicate /v1 prefix when api_base already contains it.
• Sagemaker embeddings — Uses _load_credentials (supporting aws_role_name / aws_session_name) instead of manual boto3 client construction.
• _redact_standard_logging_object — New helper to redact the standard_logging_object field in model_call_details is added but never called from perform_redaction, leaving the redaction incomplete and causing the updated e2e test assertions to fail.
• Credential endpoints — Splits a previously-broken single handler (mismatched Path parameter) into two clean routes: /credentials/by_name/{credential_name:path} and /credentials/by_model/{model_id}.
• MCP tool filtering — Case-insensitive comparison for OpenAPI operationId camelCase names.
• VirtualKeysTable — Adds a manual "Fetch / Fetching" refresh button; stale data remains visible during background refetches.

Confidence Score: 3/5

• Mostly safe to merge, but the redact-messages helper is dead code and will cause the updated e2e assertions to fail.
• The vast majority of fixes are correct and well-tested with mocked unit tests. The one concrete logic gap is that _redact_standard_logging_object is defined but never wired into perform_redaction, meaning standard_logging_object content is not redacted when turn_off_message_logging is enabled, and the matching e2e test assertions were updated to expect the new behaviour that is not yet implemented. This is a functional regression for users relying on message-logging redaction.
• litellm/litellm_core_utils/redact_messages.py — _redact_standard_logging_object must be called from perform_redaction to complete the redaction fix.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/redact_messages.py	Adds _redact_standard_logging_object helper to redact the standard_logging_object field, but the function is never called from perform_redaction — the new redaction logic is dead code and the updated e2e test assertions will fail.
litellm/caching/dual_cache.py	Correctly propagates default_in_memory_ttl to the async set_cache and set_cache_pipeline methods, which previously only applied it to the sync counterparts.
litellm/litellm_core_utils/duration_parser.py	Correctly fixes month-overflow for multi-month duration strings (e.g. +2m in November), replacing the December-only special case with proper modular arithmetic.
litellm/router.py	Adds non-retryable-error early-exit check inside the async retry loop and tracks the latest exception as original_exception. Logic is sound but uses a bare except Exception that could mask unexpected internal errors.
litellm/router_strategy/lowest_latency.py	Fixes a latency calculation bug: the average is now computed from item_ttft_latency when streaming (rather than always dividing by len(item_latency)), preventing a potential ZeroDivisionError and incorrect averages.
litellm/llms/bedrock/chat/converse_transformation.py	Adds completion_tokens_details (with reasoning_tokens estimated via token_counter) to the Bedrock usage response, improving parity with the OpenAI response schema.
litellm/llms/fireworks_ai/chat/transformation.py	Strips a trailing /v1 from api_base before appending /v1/accounts/{account_id}/models, preventing a duplicate /v1/v1/… URL when the base already contains the version prefix.
litellm/llms/sagemaker/completion/handler.py	Replaces the manual boto3 client creation in the embedding path with _load_credentials + boto3.Session, enabling IAM role assumption (aws_role_name / aws_session_name) to match the completion handler's behaviour.
litellm/proxy/credential_endpoints/endpoints.py	Splits the previously-broken single get_credential handler (which had a credential_name Path parameter mismatched against the /by_model/{model_id} route) into two clean, correctly-routed handlers.
litellm/responses/litellm_completion_transformation/transformation.py	Adds convert_apply_patch_tool_call_to_chat_completion_tool_call for ResponseApplyPatchToolCall serialisation, and merges consecutive assistant messages to satisfy Anthropic's "tool_use must precede tool_result" requirement.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[async_function_with_retries] --> B[Initial attempt: make_call]
    B -->|success| C[Return response]
    B -->|exception e| D[original_exception = e]
    D --> E[log_retry / get healthy deployments]
    E --> F{_retry_policy_applies?}
    F -->|Yes| I[Sleep, increment attempt]
    F -->|No| G[should_retry_this_error]
    G -->|returns normally - retryable| I
    G -->|raises - non-retryable| H[raise e immediately]
    I --> J{attempts < num_retries?}
    J -->|Yes| B
    J -->|No| K[raise original_exception\nlast error seen]

Loading

_{Last reviewed commit: 4e6e1d8}

[greptile-apps]

_redact_standard_logging_object is defined but never called

The helper function _redact_standard_logging_object is added to handle redacting the standard_logging_object inside model_call_details, but it is never invoked anywhere in the codebase (confirmed via global grep).

As a result, model_call_details["standard_logging_object"]["response"] and model_call_details["standard_logging_object"]["messages"] are not redacted when turn_off_message_logging is enabled, even though the e2e tests in tests/logging_callback_tests/test_logging_redaction_e2e_test.py were updated to assert the new finer-grained redaction behavior (e.g. response["choices"][0]["message"]["content"] == "redacted-by-litellm").

The function needs to be called from perform_redaction:

def perform_redaction(model_call_details: dict, result):
    # Redact model_call_details
    model_call_details["messages"] = [
        {"role": "user", "content": "redacted-by-litellm"}
    ]
    model_call_details["prompt"] = ""
    model_call_details["input"] = ""

    # Redact standard_logging_object if present
    _redact_standard_logging_object(model_call_details)
    ...

Without this call, the updated e2e test assertions will fail and users who rely on turn_off_message_logging will continue to have unredacted content leaked via the standard_logging_object.

[greptile-apps]

Bare except Exception silently masks unexpected errors from should_retry_this_error

The pattern used to detect non-retryable errors is:

try:
    self.should_retry_this_error(error=e, ...)
except Exception:
    raise e

should_retry_this_error always raises error (i.e., e) itself when the error is non-retryable, so except Exception: raise e is equivalent to except Exception: raise. However, if should_retry_this_error ever raises an unexpected internal exception (e.g., a TypeError from a None comparison or a missing attribute), that bug will be silently swallowed and e will be raised instead, making debugging very difficult.

Consider catching only the specific LiteLLM exception types that should_retry_this_error intentionally raises:

try:
    self.should_retry_this_error(error=e, ...)
except litellm.LITELLM_EXCEPTION_TYPES:  # or tuple of specific types
    raise e

Or simpler, since the function always re-raises the same error object:

try:
    self.should_retry_this_error(error=e, ...)
except Exception:
    raise  # re-raise the actual exception from should_retry_this_error