fix(router): stop per-deployment num_retries from double-counting as provider max_retries

[yassin-berriai]

Relevant issues

Regression from #18975 (which resolved #18968). That change taught the Router to read a per-deployment num_retries from litellm_params, but the same value also reached litellm.completion and became the provider client's max_retries, so the two retry layers multiplied

Linear ticket

Resolves LIT-4385

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This is a retry-counting bug, so the proof counts the actual upstream requests one proxy call produces. A real provider cannot be made to fail deterministically on every attempt, so (exactly as the ticket's own reproduction does) the deployment points at a tiny local upstream that returns 500 on every POST /v1/chat/completions and keeps a request counter. The proxy is real, the OpenAI SDK path is real, only the upstream is a controllable failing stand-in

Config used for both runs (matches the ticket: global num_retries: 1, deployment num_retries: 5):

model_list:
  - model_name: mock
    litellm_params:
      model: openai/mock
      api_base: http://127.0.0.1:8885/v1
      api_key: sk-fake
      num_retries: 5
litellm_settings:
  num_retries: 1
general_settings:
  master_key: sk-litfix-4385

Before (unfixed, commit 212a9213c4)

$ curl -s -X POST http://127.0.0.1:4785/v1/chat/completions \
    -H "Authorization: Bearer sk-litfix-4385" -H "Content-Type: application/json" \
    -d '{"model":"mock","messages":[{"role":"user","content":"Hello!"}]}'
{"error":{"message":"litellm.InternalServerError: ... mock upstream always fails ...","code":"500"}}

$ curl -s http://127.0.0.1:8885/count
{"count": 36}

One proxy request produced 36 upstream requests, which is (1 + 5) ** 2

After (this PR, commit d10a956cab)

$ curl -s -X POST http://127.0.0.1:4785/v1/chat/completions \
    -H "Authorization: Bearer sk-litfix-4385" -H "Content-Type: application/json" \
    -d '{"model":"mock","messages":[{"role":"user","content":"Hello!"}]}'
{"error":{"message":"litellm.InternalServerError: ... mock upstream always fails ...","code":"500"}}

$ curl -s http://127.0.0.1:8885/count
{"count": 6}

One proxy request produced exactly 6 upstream requests, which is 1 + num_retries. The Router owns the retries and the provider SDK no longer retries on top of it

Edge case on the same fixed proxy, proving retries do not nest when a max_retries is also supplied in the request body:

# request body adds "max_retries": 3 on top of deployment num_retries: 5
# upstream counter delta for this single request: 6 (not 24)

Independent e2e verification

Devin independently reproduced the same before/after on a fresh clone: one proxy call against an always-500 counting mock upstream produced exactly 36 upstream requests on the unfixed base and exactly 6 on this branch, same config and same single curl, only the code differing

Full-resolution recording: https://raw.githubusercontent.com/yassin-berriai/litellm-pr-media/main/lit-4385/num_retries_amplification.mp4

Type

🐛 Bug Fix

Changes

For a routed call, litellm.completion previously copied the incoming num_retries onto max_retries unconditionally, and max_retries is what gets written onto the provider client (_set_dynamic_params_on_client). A deployment num_retries: 5 therefore drove both the Router's retry loop and the provider SDK's internal retries, so every one of the Router's 1 + num_retries attempts fanned out into 1 + num_retries upstream calls

The Router is the sole retry owner for routed requests, so completion now forces the provider-SDK max_retries to 0 whenever the call originates from the Router or proxy (detected via model_group in the request metadata, the same signal the @client wrapper already uses). The num_retries to max_retries alias is kept only for direct, non-routed litellm calls, which is the documented instructor use case. Because max_retries is forced to 0 for routed calls, a max_retries set at the request or deployment level can no longer nest provider-SDK retries under the Router either, so num_retries (Router-owned) and max_retries (provider-SDK, direct calls only) now have distinct, non-overlapping meanings

Tests extend the existing mapped file and count real upstream requests through the full Router completion path by injecting a counting transport via litellm.aclient_session. They cover the deployment, request, deployment-max_retries, retry-policy-present, and global-num_retries cases, and assert the direct-call alias is preserved

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR prevents provider retries from multiplying Router retries. The main changes are:

• Detect routed calls through both metadata fields
• Disable provider-level retries for routed requests
• Preserve retry aliases for direct completion calls
• Add request-counting tests for retry configurations
• Close the injected async HTTP session after each test

Confidence Score: 5/5

This looks safe to merge.

• Both metadata sources are checked independently.
• Routed calls no longer pass retries to the provider layer.
• Direct-call retry behavior remains covered.
• The injected test session is closed during teardown.
• No blocking issues were found in the updated code.

Important Files Changed

Filename	Overview
litellm/main.py	Checks both metadata sources for the Router marker and disables provider retries for routed calls.
tests/test_litellm/test_router_per_deployment_num_retries.py	Adds request-counting tests for routed and direct retry behavior and closes the injected async session.

_{Reviews (2): Last reviewed commit: "fix(router): make router-origin check ro..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

Thanks, addressed the two robustness items in d10a956:

• Router-origin detection now checks model_group in metadata and litellm_metadata independently (any("model_group" in (kwargs.get(k) or ()) for k in ("metadata", "litellm_metadata"))), so a non-empty metadata without model_group can no longer hide a model_group that lives in litellm_metadata
• The test fixture now closes the injected async client on teardown

On the "direct calls using model_group metadata lose their requested retries" point: model_group in the request metadata is the exact signal LiteLLM already uses to identify router/proxy-originated calls. The @client wrapper in litellm/utils.py computes _is_litellm_router_call = "model_group" in (kwargs.get("metadata") or {}) and skips its own retry loop on that basis, so a direct call carrying metadata["model_group"] is already treated as router-originated by LiteLLM's retry machinery. The documented instructor use case (the reason the num_retries -> max_retries alias exists) does not set model_group, so its alias is preserved and covered by test_direct_completion_still_forwards_num_retries_to_provider. This reuses the established discriminator rather than introducing a new one

Please re-review the current head d10a956

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_num_retries_amplification (688b064) with litellm_internal_staging (212a921)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (30ed840) during the generation of this report, so 212a921 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

CI status: 76/77 green. The single red, misc / Run tests, is pre-existing base drift unrelated to this PR: it fails at tests/test_litellm/interactions/test_openapi_compliance.py::test_status_enum_values because the Responses API status enum gained a queued value on litellm_internal_staging without that compliance test's expected list being updated. It fails identically on every currently-open PR (e.g. #34126, #34127, #34128, #34131, #34132) and this diff touches only litellm/main.py (completion retry path) and a router test file, nothing near that enum. The fix belongs in a separate PR on the base branch

[yassin-berriai]

@greptileai please review the current head d10a956 — it addresses the metadata/litellm_metadata detection robustness and closes the injected async test clients since your last pass on e346c83