fix(router): honor request-level num_retries over a deployment's litellm_params value by yassin-berriai · Pull Request #35483 · BerriAI/litellm

yassin-berriai · 2026-08-01T19:37:10Z

TLDR

Problem this solves:

model_list num_retries outranked the header and the body
Documented precedence was inverted for any deployment that sets it
x-litellm-num-retries: 0 could not disable retries

How it solves it:

Adopt the deployment value only when the request carried none
Stop pre-filling the router default, which erased that distinction
Precedence is now header > body > model_list > litellm_settings

Relevant issues

Docs companion, which writes the precedence down and explains the max_retries answer: BerriAI/litellm-docs#734. It carries no CI dependency on this PR and can merge in either order

Linear ticket

Resolves LIT-4772

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The observable is the number of requests one proxy call sends upstream, which no real provider emits deterministically, so the deployment points at a tiny counting upstream that 500s every POST /v1/chat/completions and serves its tally on GET /count. One proxy call, then read the tally; 1 + effective num_retries is what the customer counts as "number of queries"

Both runs use the same config, the same proxy launcher and the same driver script, on the same machine. Before was captured at 1e7b39d15c (the base branch tip this PR was cut from, i.e. litellm/router.py in its pre-fix state) and after at f95038231f, which the current head eaad030fe6 carries forward with an identical tree (git diff f95038231f eaad030fe6 is empty; only the commit message changed)

config.yaml, counting upstream, and the driver (click to expand)

model_list:
  - model_name: retry-test
    litellm_params:
      model: openai/gpt-fake
      api_base: http://127.0.0.1:8772/v1
      api_key: sk-fake
      num_retries: 2

  - model_name: retry-test-nomodel
    litellm_params:
      model: openai/gpt-fake
      api_base: http://127.0.0.1:8772/v1
      api_key: sk-fake

litellm_settings:
  num_retries: 1

router_settings:
  disable_cooldowns: true

general_settings:
  master_key: sk-1234

# mock_upstream.py - counts every upstream chat completion and always fails it
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

count = 0


class Handler(BaseHTTPRequestHandler):
    def _send(self, code, payload):
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(body)))
        self.send_header("retry-after", "0")
        self.end_headers()
        self.wfile.write(body)

    def do_POST(self):
        global count
        if self.path.endswith("/reset"):
            count = 0
            self._send(200, {"count": count})
            return
        self.rfile.read(int(self.headers.get("content-length", 0) or 0))
        count += 1
        self._send(500, {"error": {"message": "upstream boom", "type": "server_error"}})

    def do_GET(self):
        self._send(200, {"count": count})

    def log_message(self, *args):
        pass


ThreadingHTTPServer(("127.0.0.1", 8772), Handler).serve_forever()

python mock_upstream.py &
litellm --config config.yaml --port 4772 &

run() {  # run <model> <header num_retries> <body num_retries>
  curl -s -X POST http://127.0.0.1:8772/reset >/dev/null
  hdr=(); [ "$2" != "-" ] && hdr=(-H "x-litellm-num-retries: $2")
  body="{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]"
  [ "$3" != "-" ] && body="$body,\"num_retries\":$3"
  body="$body}"
  curl -s -o /dev/null http://127.0.0.1:4772/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    "${hdr[@]}" -d "$body"
  n=$(curl -s http://127.0.0.1:8772/count | python3 -c 'import sys,json;print(json.load(sys.stdin)["count"])')
  printf '| %-18s | %-6s | %-4s | %s |\n' "$1" "$2" "$3" "$n"
}

run retry-test-nomodel -  -
run retry-test-nomodel 3  -
run retry-test-nomodel -  5
run retry-test         -  -
run retry-test         3  -
run retry-test         -  5
run retry-test         3  5
run retry-test         0  -

Before, at `1e7b39d15c`

| model              | header | body | upstream requests |
| ------------------ | ------ | ---- | ----------------- |
| retry-test-nomodel | -      | -    | 2 |
| retry-test-nomodel | 3      | -    | 4 |
| retry-test-nomodel | -      | 5    | 6 |
| retry-test         | -      | -    | 3 |
| retry-test         | 3      | -    | 3 |
| retry-test         | -      | 5    | 3 |
| retry-test         | 3      | 5    | 3 |
| retry-test         | 0      | -    | 3 |

The four retry-test rows that carry a header or a body value all send 3 requests, which is the deployment's num_retries: 2 plus the initial attempt. The header and the body are ignored, including x-litellm-num-retries: 0, which should have disabled retries entirely. The retry-test-nomodel rows show the precedence is already correct as soon as no deployment value exists

After, at `eaad030fe6`

| model              | header | body | upstream requests |
| ------------------ | ------ | ---- | ----------------- |
| retry-test-nomodel | -      | -    | 2 |
| retry-test-nomodel | 3      | -    | 4 |
| retry-test-nomodel | -      | 5    | 6 |
| retry-test         | -      | -    | 3 |
| retry-test         | 3      | -    | 4 |
| retry-test         | -      | 5    | 6 |
| retry-test         | 3      | 5    | 4 |
| retry-test         | 0      | -    | 1 |

Every row now matches header > body > model_list > litellm_settings. The rows that were already correct are unchanged, so the deployment value keeps its place directly above litellm_settings when the request says nothing (retry-test with no header and no body still sends 3), and x-litellm-num-retries: 0 now disables retries

One row verbatim, both sides

Same command, same rig, the only difference being which commit the proxy is running. retry-test carries num_retries: 2, and the request asks for 3 via the header

Before, at 1e7b39d15c

$ curl -s -X POST http://127.0.0.1:8772/reset
{"count": 0}
$ curl -s http://127.0.0.1:4772/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -H 'x-litellm-num-retries: 3' \
    -d '{"model":"retry-test","messages":[{"role":"user","content":"hi"}]}'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - upstream boom. Received Model Group=retry-test\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}
$ curl -s http://127.0.0.1:8772/count
{"count": 3}

After, at eaad030fe6

$ curl -s -X POST http://127.0.0.1:8772/reset
{"count": 0}
$ curl -s http://127.0.0.1:4772/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -H 'x-litellm-num-retries: 3' \
    -d '{"model":"retry-test","messages":[{"role":"user","content":"hi"}]}'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - upstream boom. Received Model Group=retry-test\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}
$ curl -s http://127.0.0.1:8772/count
{"count": 4}

3 upstream requests is the deployment's num_retries: 2 plus the initial attempt, with the header discarded. 4 is the header's 3 plus the initial attempt

The other entry points keep their deployment retries

Dropping the pre-fill from _update_kwargs_before_fallbacks alone would have left the six entry points that repeat it ranking their own default above the deployment, silently disabling a model_list num_retries on image generation, adapter completion, and the file and batch routes. Measured through router.aimage_generation against a counting upstream, deployment num_retries: 4 with the router default at 0, counting every upstream request (the provider SDK contributes its own attempts per router attempt on this route, hence the multiples of 3)

tree	upstream requests
base `1e7b39d15c`	15
helper fix only, the six pre-fills left in place	3
this PR at `eaad030fe6`	15

15 is 5 router attempts, so the deployment's 4 retries are honoured before and after. The middle row is the regression this PR does not ship, and a test pins it: restoring the aimage_generation pre-fill alone fails test_deployment_num_retries_applies_to_image_generation

On the second question in the report, request body `max_retries`

num_retries and max_retries are different knobs: num_retries is the router's own retry loop, max_retries is the provider SDK's internal retry count. For a routed call the router is the sole retry owner, so the provider client is pinned to max_retries: 0; that is deliberate, and it is what stops a deployment num_retries: N from being applied twice and turning one call into (1 + N) ** 2 upstream requests. A request body max_retries therefore has no effect through the proxy, which this PR does not change. Same rig, after the fix, at eaad030fe6

| model              | request                | upstream requests |
| ------------------ | ---------------------- | ----------------- |
| retry-test-nomodel | body max_retries=5     | 2 |
| retry-test         | body max_retries=5     | 3 |

Both counts are exactly what the same request sends without max_retries, so the value is inert rather than partially applied

Type

🐛 Bug Fix

Changes

Router._update_kwargs_before_fallbacks used to fill kwargs["num_retries"] in with self.num_retries whenever the caller omitted it. That collapsed "the request asked for N retries" and "nobody asked, use the global" into one indistinguishable value, so by the time async_function_with_retries ran it had no way to rank a request value against a deployment one. It now leaves num_retries exactly as the caller passed it; async_function_with_retries already resolves the router default (and remains the safety net for an explicit num_retries=None, so the None > int TypeError guard is untouched)

Six entry points repeated the same pre-fill a line above their own call to that helper: image_generation, aimage_generation, aadapter_completion, acreate_file, acreate_batch, acancel_batch. All six reach async_function_with_retries, so leaving them would have made the request value never None there and permanently suppressed a deployment num_retries on those routes, which is a regression rather than the fix. They are dropped too. The one remaining pre-fill, in the sync text_completion, stays deliberately: that path resolves a deployment and calls litellm.text_completion directly without entering the retry loop, so no request-versus-deployment ranking happens there and there is nothing for this fix to correct. Removing the line would only change which value is forwarded into litellm.text_completion, which is a behaviour change this bug does not call for. Two pre-existing quirks of that path, neither introduced nor addressed here: the final spread puts **kwargs after **data, so the router default already shadows the deployment's own litellm_params.num_retries there, and because the same function sets metadata.model_group, the router-call check in litellm/main.py forces max_retries = 0 on the chat funnel anyway

async_function_with_retries keeps the request-level value it popped and adopts the deployment's exception-stamped num_retries only when the request carried none. Nothing else about the deployment path changes: it still beats litellm_settings, still accepts a string value from an env var, and still applies on the mock_testing_rate_limit_error path

Tests extend the mapped file. Three new behavioural tests drive router.acompletion through the real retry loop and count attempts via a callback: request beats deployment (and request 0 disables retries), deployment still beats global when the request is silent, and the request value also wins on the rate-limit mock path, which is the second place a deployment value is stamped onto an exception. All three fail on the unfixed tree and pass with the fix. Two existing tests asserted the old kwargs-filling contract directly and were rewritten against the new one; the invariants they protected (num_retries=None must not raise, an explicit 0 must survive) stay covered by the behavioural tests in the same class

Two more cover the non-completion entry points through aimage_generation, counting real upstream requests: a deployment num_retries still applies there when the request is silent, and a request value still wins. The provider SDK contributes a fixed number of its own attempts per router attempt on that route, so those two calibrate that factor from a single-attempt run rather than hard coding it. Restoring the pre-fill on aimage_generation alone fails the first of them, which is the regression the six deletions close

One knock-on worth calling out: a key/team router_settings_override.num_retries is merged into the request data by route_llm_request.py only when the request itself did not set one, so it reaches the router as the same num_retries kwarg and now also outranks a deployment value. That reads as the intent of a per-key/team override, which is described there as overriding the global router settings for that request, and it lands in the same slot a body value would. Distinguishing it from a real request value would need a second kwarg, which is not something this fix needs; flagging it so the ranking is a deliberate call rather than a surprise

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

CLAassistant · 2026-08-01T19:37:16Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

greptile-apps · 2026-08-01T19:39:51Z

Greptile Summary

The PR corrects retry-count precedence so an explicit request value is retained over a deployment value while omitted values can still fall back through deployment and global settings

Stops eagerly inserting the router retry default before shared retry resolution
Applies the same behavior to completion, image, adapter, file, and batch routes
Adds behavioral coverage for request, deployment, global, zero-retry, rate-limit mock, and image-generation cases

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/router.py	Adjusts shared retry resolution and affected entry points so request-level retry counts take precedence without suppressing deployment defaults
tests/test_litellm/test_router_per_deployment_num_retries.py	Adds focused regression coverage for retry precedence and preserves coverage for deployment retries on image generation

_{Reviews (4): Last reviewed commit: "fix(router): honor request-level num_ret..." | Re-trigger Greptile}

yassin-berriai · 2026-08-01T19:43:58Z

@greptileai please review the current head 9422ad5

No logic changed since your 5/5. The branch was rebased onto current staging to pick up #35479, which removes a duplicate Sequence import in team_endpoints.py that was failing lint on the merge ref for a file this diff never touches. litellm/router.py is byte identical across the rebase; the PR body now also carries a verbatim before/after curl transcript for one row.

codecov · 2026-08-01T19:50:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

yassin-berriai · 2026-08-01T20:06:03Z

@greptileai please review the current head f950382

Your 5/5 was against 9422ad526b. Since then a teammate's review found the first version incomplete in a way that would have regressed a different path, and I confirmed it by measurement rather than by reading, so the diff has grown.

Removing the num_retries pre-fill from _update_kwargs_before_fallbacks was not enough: six entry points (image_generation, aimage_generation, aadapter_completion, acreate_file, acreate_batch, acancel_batch) repeat the same pre-fill a line above their own call to that helper, and they all reach async_function_with_retries. With the pre-fill still there the request value is never None on those routes, so the new guard permanently suppressed a deployment num_retries. Measured through aimage_generation with deployment num_retries: 4: 15 upstream requests on base, 3 with the helper fix alone, 15 again now. Those six pre-fills are dropped and two tests cover it.

The seventh pre-fill, in the sync text_completion, is deliberately kept: that path resolves a deployment and calls litellm.text_completion directly without entering the retry loop, so its num_retries is the provider-SDK knob and removing it would change what reaches the provider.

The live-proxy proof in the body was re-captured at this head, and the PR body now also flags that a key/team router_settings_override.num_retries shares the request slot and therefore now outranks a deployment value.

…llm_params value A failing deployment stamps its own litellm_params.num_retries onto the raised exception, and async_function_with_retries adopted that value unconditionally. So a model_list num_retries outranked both the x-litellm-num-retries header and the request body, inverting the documented precedence to model_list > header > body > litellm_settings. The router could not tell a request-level value from its own default because the entry points filled num_retries in with self.num_retries whenever the caller omitted it, collapsing "the request asked for N" and "nobody asked". Drop that pre-fill from _update_kwargs_before_fallbacks and from the six entry points that also did it a line above their own call to it (image generation sync and async, adapter completion, file create, batch create, batch cancel), all of which reach async_function_with_retries, where the router/global default is already resolved. Leaving them would have made the request value never None on those routes and permanently suppressed a deployment num_retries there. The sync text_completion pre-fill stays. That path resolves a deployment and calls litellm.text_completion directly, never entering the retry loop, so no request-versus- deployment ranking happens there and there is nothing to fix; removing the line would only change which value is forwarded to litellm.text_completion, a behaviour change this bug does not call for. async_function_with_retries then adopts the deployment's value only when the request carried none. Precedence is now header > body > model_list > litellm_settings, with the deployment value still beating litellm_settings when the request is silent, on every entry point that retries. Resolves LIT-4772

yassin-berriai · 2026-08-01T20:08:10Z

@greptileai please review the current head eaad030

Superseding my previous re-trigger for f95038231f. The tree is identical between those two (git diff f95038231f eaad030fe6 is empty); only the commit message changed, to correct one overclaiming clause about the sync text_completion pre-fill that this PR deliberately keeps. The PR body carries the same correction plus the two pre-existing quirks of that path a reviewer might otherwise ask about.

codspeed-hq · 2026-08-01T20:50:23Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_lit4772_num_retries_precedence (eaad030) with litellm_internal_staging (e204e62)¹}

No successful run was found on litellm_internal_staging (97ec047) during the generation of this report, so e204e62 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…es is inert (#734) Spell out the four places num_retries can come from and how they rank (header > request body > model_list litellm_params > litellm_settings), and explain that max_retries is the provider SDK knob rather than a second spelling of num_retries, so it has no effect on a routed proxy request. Companion to BerriAI/litellm#35483 (LIT-4772)

yassin-berriai force-pushed the litellm_lit4772_num_retries_precedence branch from 39144e0 to 9422ad5 Compare August 1, 2026 19:43

yassin-berriai mentioned this pull request Aug 1, 2026

docs(routing): document num_retries precedence and why body max_retries is inert BerriAI/litellm-docs#734

Merged

yassin-berriai mentioned this pull request Aug 1, 2026

fix(helm): render pod-level securityContext on the migration Job #35482

Merged

5 tasks

yassin-berriai force-pushed the litellm_lit4772_num_retries_precedence branch from 9422ad5 to f950382 Compare August 1, 2026 20:02

yassin-berriai force-pushed the litellm_lit4772_num_retries_precedence branch from f950382 to eaad030 Compare August 1, 2026 20:07

tin-berri approved these changes Aug 1, 2026

View reviewed changes

yassin-berriai merged commit a8cc6a9 into litellm_internal_staging Aug 1, 2026
80 checks passed

yassin-berriai deleted the litellm_lit4772_num_retries_precedence branch August 1, 2026 20:51

This was referenced Aug 18, 2026

feat: governance v3——威胁重排+角色重构（ADR-0010） Cloudbird-Software/agent-registry#5

Merged

fix: stop demo query streams from dying mid-flight (research#86) FalkorDB/QueryWeaver#714

Open

Navigation Menu

Uh oh!

Uh oh!

fix(router): honor request-level num_retries over a deployment's litellm_params value - #35483

fix(router): honor request-level num_retries over a deployment's litellm_params value#35483
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_lit4772_num_retries_precedence

yassin-berriai commented Aug 1, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Aug 1, 2026

Uh oh!

greptile-apps Bot commented Aug 1, 2026 •

edited

Loading

Important Files Changed

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

codecov Bot commented Aug 1, 2026 •

edited

Loading

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

codspeed-hq Bot commented Aug 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

yassin-berriai commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before, at 1e7b39d15c

After, at eaad030fe6

One row verbatim, both sides

The other entry points keep their deployment retries

On the second question in the report, request body max_retries

Type

Changes

Final Attestation

Uh oh!

CLAassistant commented Aug 1, 2026

Uh oh!

greptile-apps Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

codecov Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

yassin-berriai commented Aug 1, 2026

Uh oh!

codspeed-hq Bot commented Aug 1, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yassin-berriai commented Aug 1, 2026 •

edited

Loading

Before, at `1e7b39d15c`

After, at `eaad030fe6`

On the second question in the report, request body `max_retries`

greptile-apps Bot commented Aug 1, 2026 •

edited

Loading

codecov Bot commented Aug 1, 2026 •

edited

Loading