fix(proxy): trigger gateway fallbacks on local rate limit errors

[devin-ai-integration]

Relevant issues

Fixes #8822

Linear ticket

Resolves LIT-3890

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Config used (test_fallback_config.yaml):

model_list:
  - model_name: anthropic-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: anthropic-sonnet-4-5
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks: [{"anthropic-haiku-4-5": ["anthropic-sonnet-4-5"]}]

general_settings:
  master_key: sk-1234

litellm_settings:
  drop_params: True
  telemetry: False
  callbacks: ["test_rate_limit_hook.proxy_handler_instance"]

A custom callback (test_rate_limit_hook.py) that raises ProxyRateLimitError for anthropic-haiku-4-5 on every call, simulating what parallel_request_limiter does when model_tpm_limit is exceeded:

from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError

class LocalRateLimitSimulator(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        model = data.get("model", "")
        if model == "anthropic-haiku-4-5":
            raise ProxyRateLimitError(
                detail={"error": f"Simulated local rate limit: model={model} exceeded TPM limit"},
                model=model,
            )
        return data

Started the proxy:

python litellm/proxy/proxy_cli.py --config test_fallback_config.yaml --detailed_debug --port 4000

Test 1: Rate-limited model falls back successfully

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model": "anthropic-haiku-4-5", "messages": [{"role": "user", "content": "Say hello in exactly 3 words"}], "max_tokens": 20}'

{
    "id": "chatcmpl-e989d131-abcb-4fb0-9501-9c62da9bc34b",
    "model": "anthropic-haiku-4-5",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hello to you.",
                "role": "assistant"
            }
        }
    ],
    "usage": {
        "completion_tokens": 7,
        "prompt_tokens": 15,
        "total_tokens": 22
    }
}

Proxy log confirms the fallback:

05:08:37 - LiteLLM Proxy:INFO: common_request_processing.py:1226 - Local rate limit hit for model=anthropic-haiku-4-5, attempting fallbacks: ['anthropic-sonnet-4-5']

Test 2: Non-rate-limited model works directly (no fallback)

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model": "anthropic-sonnet-4-5", "messages": [{"role": "user", "content": "Say hi in 2 words"}], "max_tokens": 10}'

{
    "id": "chatcmpl-db87e126-7cab-421e-ba29-67c089669c8d",
    "model": "anthropic-sonnet-4-5",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hi there!",
                "role": "assistant"
            }
        }
    ]
}

Test 3: disable_fallbacks=true returns 429 (no fallback)

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model": "anthropic-haiku-4-5", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 10, "disable_fallbacks": true}'

{
    "error": {
        "message": "Simulated local rate limit: model=anthropic-haiku-4-5 exceeded TPM limit (call #2)",
        "type": "throttling_error",
        "code": "429"
    }
}

Type

Bug Fix

Changes

ProxyRateLimitError raised by pre-call hooks (parallel_request_limiter, dynamic_rate_limiter_v3, etc.) happens in proxy_logging_obj.pre_call_hook() which runs before route_request(). Since the router's async_function_with_fallbacks() never sees the error, configured fallback models are never tried

This PR adds _pre_call_with_fallbacks() in CommonRequestProcessing that wraps common_processing_pre_call_logic(). When a ProxyRateLimitError is caught, it resolves fallback models from key-level router_settings or router-level fallbacks (using the same get_fallback_model_group() the router uses), then retries pre-call logic with each fallback. If all fallbacks also fail, the original error is re-raised

The disable_fallbacks request flag is respected; when set, the original error propagates immediately

Five regression tests cover: fallback triggered on rate limit, no fallbacks configured, all fallbacks also rate-limited, key-level router_settings precedence, and the disable_fallbacks flag

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yuneng-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

@greptileai

[greptile-apps]

Greptile Summary

This PR fixes a gap where ProxyRateLimitError raised by pre-call hooks (e.g., parallel_request_limiter) during proxy_logging_obj.pre_call_hook() was never seen by the router's async_function_with_fallbacks, so configured fallback models were silently skipped and a 429 was returned to the client. The fix wraps common_processing_pre_call_logic in a new _pre_call_with_fallbacks method that catches ProxyRateLimitError from the primary model and retries with each configured fallback.

• _pre_call_with_fallbacks resolves fallbacks from key-level router_settings (taking precedence) or router-level fallbacks via the existing get_fallback_model_group helper, and retries pre-call logic for each fallback model in order.
• The disable_fallbacks request flag is respected; model-state mutation is guarded with except BaseException to restore self.data["model"] on any non-rate-limit exception during fallback attempts.
• Five unit tests cover: successful fallback, no-fallbacks configured, all fallbacks exhausted, key-level precedence, and disable_fallbacks; an integration test drives the real parallel_request_limiter end-to-end using a frozen clock and in-memory cache.

Confidence Score: 5/5

Safe to merge — the change is additive, backward-compatible, and well-tested.

The new _pre_call_with_fallbacks wrapper is inserted at a single call-site and degrades gracefully (re-raises the original error) when no router, no fallbacks, or disable_fallbacks is set. Previous review feedback on model-state mutation and the unused proxy_config parameter in _resolve_fallback_models has been addressed. Tests cover all edge cases, and the integration test's hardcoded cache-key format matches the actual parallel_request_limiter implementation.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Adds _pre_call_with_fallbacks and _resolve_fallback_models; replaces direct common_processing_pre_call_logic call with the new wrapper at the single call-site in common_proxy_request_processing. Control flow, exception handling (BaseException guard for model-state restore), and backward-compatibility (no-op when no router or no fallbacks) all look correct.
tests/test_litellm/proxy/test_common_request_processing.py	Adds six tests covering the new fallback path. The integration test (test_real_parallel_request_limiter_model_tpm_limit_triggers_fallback) uses real limiter code with an in-memory DualCache (no network calls), and its hardcoded cache-key format matches the actual parallel_request_limiter key format ({api_key}::{model}::{precise_minute}::request_count).

_{Reviews (6): Last reviewed commit: "test(proxy): cover per-key per-model TPM..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 95.12195% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	95.12%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes a gap where ProxyRateLimitError raised inside pre_call_hook was never seen by the router's fallback handler, so locally rate-limited requests always returned 429 instead of trying configured fallback models. The fix wraps common_processing_pre_call_logic in a new _pre_call_with_fallbacks method that catches ProxyRateLimitError, resolves fallbacks from key-level or router-level settings, and retries each fallback in order.

• _pre_call_with_fallbacks intercepts ProxyRateLimitError and retries with each fallback model, with disable_fallbacks respected and an exhaustion path that re-raises the original error.
• _resolve_fallback_models resolves the fallback list from key-level router_settings (taking precedence) or from llm_router.fallbacks, using the existing get_fallback_model_group helper.
• Five unit tests are added covering the main success/failure/precedence/flag scenarios, all fully mocked.

Confidence Score: 3/5

The new fallback wrapper has two correctness gaps that affect production behaviour: model state is left dirty on certain exception paths, and the primary model's logging object is silently dropped rather than closed with a failure callback.

The core fallback loop mutates self.data["model"] before calling common_processing_pre_call_logic for each fallback, but only restores it after the loop exhausts all fallbacks via ProxyRateLimitError. Any other exception from a fallback escapes with the model name stuck at the last-tried fallback. Separately, every call to common_processing_pre_call_logic creates a fresh LiteLLMLoggingObj; when the primary model's attempt is caught and swallowed, its logging object receives no callback, so rate-limit events on the primary are invisible to every configured handler.

litellm/proxy/common_request_processing.py — specifically the fallback loop in _pre_call_with_fallbacks and the exception handling around the primary model's logging object lifecycle.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Adds _pre_call_with_fallbacks and _resolve_fallback_models to catch ProxyRateLimitError before routing and retry with configured fallback models; two bugs: (1) self.data["model"] is not restored when a fallback raises a non-rate-limit exception, and (2) the primary model's LiteLLMLoggingObj is abandoned without firing any failure callback.
tests/test_litellm/proxy/test_common_request_processing.py	Adds 5 focused unit tests for the new fallback logic; all tests use mocks and make no real network calls; missing a test case for the non-ProxyRateLimitError exception path in the fallback loop (the unguarded state mutation bug).

_{Reviews (2): Last reviewed commit: "fix(proxy): trigger gateway fallbacks on..." | Re-trigger Greptile}

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai review

[yuneng-berri]

@greptile