Skip to content

fix(router): resolve retry_policy by exception hierarchy, add ServiceUnavailableErrorRetries and DefaultRetries - #35853

Merged
ryan-crabbe-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_retry_policy_503
Sep 5, 2026
Merged

fix(router): resolve retry_policy by exception hierarchy, add ServiceUnavailableErrorRetries and DefaultRetries#35853
ryan-crabbe-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_retry_policy_503

Conversation

@shivamrawat1

@shivamrawat1 shivamrawat1 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • No retry_policy key exists for 503 ServiceUnavailableError
  • InternalServerErrorRetries is accepted but silently ignored
  • 502, connection and other errors have no key at all
  • A downed model always gets retried, users cannot turn it off

How it solves it:

  • Adds ServiceUnavailableErrorRetries and a DefaultRetries catch-all to RetryPolicy
  • Resolves the retry count by walking the exception's class hierarchy, most specific first
  • Every RetryPolicy field is wired through one mapping, so none can go dead again
  • Adds both new fields to the Admin UI retry settings tab

User Flow

Before: a proxy admin whose provider is returning 503s wants those requests to fail fast, but every request is retried anyway

  1. They add model_group_retry_policy: {gpt-5.6: {InternalServerErrorRetries: 0}} under router_settings and restart the proxy
  2. They send POST http://localhost:4000/v1/chat/completions with "model": "gpt-5.6"
  3. About four seconds later they get HTTP 503 litellm.ServiceUnavailableError, and the provider's access log shows 3 requests for that one call
  4. No retry_policy key names 503, and the 500 key they set is ignored, so a 500 from the provider also produces 3 requests

After: the same admin names the error they see, or all errors, and each failed request makes exactly one upstream attempt

  1. They add ServiceUnavailableErrorRetries: 0 to the same model group, or DefaultRetries: 0 to cover every error type, and restart the proxy
  2. They send the same POST http://localhost:4000/v1/chat/completions with "model": "gpt-5.6"
  3. The HTTP 503 comes back in well under a second, and the provider's access log shows exactly 1 request
  4. Groups with no policy keep retrying twice by default, so nothing changes for admins who never set one

Relevant issues

Linear ticket

Resolves LIT-5202

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

A provider outage cannot be summoned on demand, so an openai deployment points at a local upstream on 127.0.0.1:19503 that answers every POST with a chosen 5xx status and counts the attempts it receives. GET /status/503 picks the status and zeroes the counter, GET /reset zeroes it, GET / reports {"attempts": n}. Proxy config:

model_list:
  - model_name: gpt-5.6-no-retry-503
    litellm_params:
      model: openai/gpt-5.6
      api_key: sk-fake
      api_base: http://127.0.0.1:19503/v1
  - model_name: gpt-5.6-no-retry-500
    litellm_params:
      model: openai/gpt-5.6
      api_key: sk-fake
      api_base: http://127.0.0.1:19503/v1
  - model_name: gpt-5.6-no-retry-any
    litellm_params:
      model: openai/gpt-5.6
      api_key: sk-fake
      api_base: http://127.0.0.1:19503/v1
  - model_name: gpt-5.6-default
    litellm_params:
      model: openai/gpt-5.6
      api_key: sk-fake
      api_base: http://127.0.0.1:19503/v1

router_settings:
  num_retries: 2
  model_group_retry_policy:
    gpt-5.6-no-retry-503:
      ServiceUnavailableErrorRetries: 0
    gpt-5.6-no-retry-500:
      InternalServerErrorRetries: 0
    gpt-5.6-no-retry-any:
      DefaultRetries: 0

general_settings:
  master_key: sk-1234

Each case below runs this loop, with $PATH_ and $BODY set for the endpoint named in the case heading (/v1/messages adds "max_tokens": 16, /v1/responses sends "input": "hi"):

for status in 503 500 502; do
  curl -s "http://127.0.0.1:19503/status/$status" >/dev/null
  for group in gpt-5.6-no-retry-503 gpt-5.6-no-retry-500 gpt-5.6-no-retry-any gpt-5.6-default; do
    curl -s http://127.0.0.1:19503/reset >/dev/null
    code=$(curl -s -o resp.json -w '%{http_code}' "http://127.0.0.1:$PORT$PATH_" \
      -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
      -d "{\"model\": \"$group\", \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}")
    err=$(python3 -c 'import json,sys; print(json.load(open("resp.json"))["error"]["message"].split(":")[0])')
    attempts=$(curl -s http://127.0.0.1:19503/ | python3 -c 'import json,sys; print(json.load(sys.stdin)["attempts"])')
    printf 'upstream=%s group=%-22s http=%s error=%-34s upstream_attempts=%s\n' "$status" "$group" "$code" "$err" "$attempts"
  done
done

Before (d23bec8)

Every request makes 3 upstream attempts no matter which key is set. The 503 and catch-all keys do not exist yet and are ignored, and the 500 key exists but is never consulted

/v1/chat/completions

  1. Run the loop with PORT=14503 PATH_=/v1/chat/completions
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

/v1/messages

  1. Run the loop with PORT=14503 PATH_=/v1/messages
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

/v1/responses

  1. Run the loop with PORT=14503 PATH_=/v1/responses
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

After (541ab50)

The group whose key matches the error makes exactly 1 attempt. A key for a different error does not leak (the 503 key still lets 500 and 502 retry), DefaultRetries: 0 stops retries on every status, and the group with no policy still makes 3 attempts

/v1/chat/completions

  1. Run the loop with PORT=14505 PATH_=/v1/chat/completions
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=1
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

/v1/messages

  1. Run the loop with PORT=14505 PATH_=/v1/messages
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=1
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

/v1/responses

  1. Run the loop with PORT=14505 PATH_=/v1/responses
  2. Observed:
upstream=503 group=gpt-5.6-no-retry-503   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-no-retry-500   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=503 group=gpt-5.6-no-retry-any   http=503 error=litellm.ServiceUnavailableError    upstream_attempts=1
upstream=503 group=gpt-5.6-default        http=503 error=litellm.ServiceUnavailableError    upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-503   http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=500 group=gpt-5.6-no-retry-500   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-no-retry-any   http=500 error=litellm.InternalServerError        upstream_attempts=1
upstream=500 group=gpt-5.6-default        http=500 error=litellm.InternalServerError        upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-503   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-500   http=502 error=litellm.BadGatewayError            upstream_attempts=3
upstream=502 group=gpt-5.6-no-retry-any   http=502 error=litellm.BadGatewayError            upstream_attempts=1
upstream=502 group=gpt-5.6-default        http=502 error=litellm.BadGatewayError            upstream_attempts=3

Type

🐛 Bug Fix

Changes

get_num_retries_from_retry_policy used to be a hand-ordered chain of isinstance checks, which is how InternalServerErrorRetries sat unused since May 2024 and why 503, 502, connection and not-found errors had no key at all. It now holds one mapping from exception class to RetryPolicy field and walks the exception's class hierarchy (its MRO) from the most specific class outward, returning the first field that is set. Subclasses inherit their parent's field when they have none of their own, so ContentPolicyViolationError falls back to BadRequestErrorRetries and MidStreamFallbackError to ServiceUnavailableErrorRetries. When no class in the chain has a set field, the new DefaultRetries applies, so any error can be governed without adding a field per class

RetryPolicy gains ServiceUnavailableErrorRetries and DefaultRetries. The Admin UI retry settings tab shows them as "ServiceUnavailableError (503)" and "All other errors", and schema.d.ts carries the new fields

Tests: tests/test_litellm/router_utils/test_get_retry_from_policy.py is parametrized over every RetryPolicy field, so adding a field without wiring it fails the suite. It also pins that a field does not leak to unrelated errors, that subclasses prefer their own field and fall back to the parent's, that DefaultRetries covers 502 and 404, and that a specific field beats DefaultRetries. tests/test_litellm/test_router.py points a deployment at a respx-faked upstream that answers 503, 500 or 502 and counts the HTTP requests the router makes, so the responses travel through the real OpenAI SDK and exception mapping. It covers the 503, 500 and DefaultRetries paths and pins that a 503 key does not govern a 502. The amplification test in test_router_per_deployment_num_retries.py now expects 3 upstream requests instead of 6, because InternalServerErrorRetries=2 finally overrides the per-deployment num_retries=5 the way the docs always said it would

Caveats (if any)

Medium

  • Configs that already set InternalServerErrorRetries now honor it on 500s; it was silently ignored before
  • A per-deployment num_retries loses to a matching retry_policy field for that error, same as it always did for 429 and 400

Low

  • MidStreamFallbackError inherits ServiceUnavailableErrorRetries because it subclasses that error
  • Docs for ServiceUnavailableErrorRetries and DefaultRetries live in the docs repo and need a follow-up PR
  • schema.d.ts was updated by hand for the two new fields only; the schema sync CI check verifies it

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@greptile-apps

greptile-apps Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR resolves router retry counts through exception inheritance and adds configurable retry handling for service-unavailable and otherwise-unmapped errors.

  • Adds ServiceUnavailableErrorRetries and DefaultRetries policy fields.
  • Exposes both settings in the dashboard and generated schema.
  • Adds unit and routed-request coverage for precedence, inheritance, defaults, and upstream attempt counts.

Confidence Score: 5/5

The PR appears safe to merge, with only the previously reported non-blocking generated-comment issue still outstanding.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/router_utils/get_retry_from_policy.py Replaces ordered exception checks with an MRO-based mapping and a configurable default fallback.
litellm/types/router.py Extends RetryPolicy with service-unavailable and catch-all retry counts.
tests/test_litellm/router_utils/test_get_retry_from_policy.py Covers every specific policy field, hierarchy precedence, defaults, unrelated errors, and model-group policy selection.
tests/test_litellm/test_router.py Verifies configured policies control actual upstream attempt counts for mapped and default error paths.
tests/test_litellm/test_router_per_deployment_num_retries.py Updates the amplification expectation to reflect the now-effective internal-server retry policy.
ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelRetrySettingsTab.tsx Adds dashboard choices for service-unavailable and catch-all retry policies.
ui/litellm-dashboard/src/lib/http/schema.d.ts Adds generated declaration fields for the two new retry-policy settings.

Reviews (3): Last reviewed commit: "test(router): fake the upstream with res..." | Re-trigger Greptile

Comment on lines 30949 to +30950
RateLimitErrorRetries?: number | null;
/** Serviceunavailableerrorretries */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 New generated documentation comment

The generated declaration adds a documentation comment for ServiceUnavailableErrorRetries, contrary to the repository convention against adding comments unless explicitly requested; remove it or adjust generation so the checked-in declaration follows that convention.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_retry_policy_503 (541ab50) with litellm_internal_staging (d23bec8)

Open in CodSpeed

…itellm_retry_policy_503

# Conflicts:
#	litellm/types/router.py
#	tests/test_litellm/test_router.py
…ltRetries

Replace the hand-ordered isinstance ladder in get_num_retries_from_retry_policy
with a class-to-field mapping walked along the exception's MRO, most specific
class first. A RetryPolicy field can no longer go silently dead the way
InternalServerErrorRetries did, and subclasses such as
ContentPolicyViolationError or MidStreamFallbackError pick up their parent's
field when they have none of their own.

Add a DefaultRetries catch-all so errors without a dedicated field
(BadGatewayError, APIConnectionError, NotFoundError, ...) can be governed by the
policy too. Specific fields still win over DefaultRetries.

Wiring the previously dead InternalServerErrorRetries changes one test
expectation: a policy of 2 now overrides a per-deployment num_retries of 5, so
the amplification test sees 3 upstream requests instead of 6.

Expose DefaultRetries as "All other errors" in the Admin UI retry settings tab
and ratchet the lint budgets down by the violations this branch fixed.
@ryan-crabbe-berri
ryan-crabbe-berri requested a review from a team September 4, 2026 23:11
@ryan-crabbe-berri ryan-crabbe-berri changed the title fix(router): honor ServiceUnavailableErrorRetries and InternalServerErrorRetries in retry policy fix(router): resolve retry_policy by exception hierarchy, add ServiceUnavailableErrorRetries and DefaultRetries Sep 4, 2026
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor

@greptileai re review

…t test

The test-quality gate rejects patching litellm.acompletion, and faking the
HTTP boundary is the stronger test anyway: the 503, 500 and 502 responses now
travel through the real OpenAI SDK and exception mapping before the router
decides how many times to retry. Adds a case showing that a 503 key does not
govern a 502.
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor

@greptileai re review

@ryan-crabbe-berri
ryan-crabbe-berri merged commit be76dfa into litellm_internal_staging Sep 5, 2026
183 of 184 checks passed
@ryan-crabbe-berri
ryan-crabbe-berri deleted the litellm_retry_policy_503 branch September 5, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants