Repository navigation
fix(router): backport #39491 to stable/1.100.x so retry breadcrumbs stop retaining every earlier request - #40455
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Greptile SummaryThis backport removes Router-level retry breadcrumb state, keeps breadcrumbs scoped to each request, excludes the aliased proxy request snapshot, and retains only the four most recent attempts. Regression tests cover cross-request isolation, bounded retention, credential masking, and fallback walk-state exclusion Confidence Score: 5/5The PR appears safe to merge because retry breadcrumbs are bounded and request-local without breaking metadata object identity No new actionable failures remain. The prior test-helper typing thread was manually resolved without explanation. The line-length dispute was correct because the cited line is within the configured 120-character limit
|
| Filename | Overview |
|---|---|
| litellm/router.py | Replaces shared Router breadcrumb storage with bounded per-request metadata while preserving in-place metadata write-back |
| tests/test_litellm/test_router.py | Adds regression coverage for breadcrumb isolation, flattening, retention limits, credential masking, and fallback behavior |
Reviews (3): Last reviewed commit: "test(router): type the breadcrumb test h..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a9ea571. Configure here.
Devin e2e QATested at: a9ea571 Both legs ran the same rig on a Linux box (32GB RAM, no swap pressure): a proxy booted from the named commit with model_list:
- model_name: flaky-chat
litellm_params:
model: openai/gpt-5-mini
api_key: os.environ/OPENAI_API_KEY
timeout: 0.02
- model_name: flaky-chat
litellm_params:
model: openai/gpt-5-mini
api_key: os.environ/OPENAI_API_KEY
api_base: https://api.openai.com/v1/
timeout: 0.02
- model_name: healthy-chat
litellm_params:
model: openai/gpt-5-mini
api_key: os.environ/OPENAI_API_KEY
router_settings:
num_retries: 2
disable_cooldowns: true
fallbacks: [{"flaky-chat": ["healthy-chat"]}]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEYPORT=44788
python litellm/proxy/proxy_cli.py --config lit7392_config.yaml --port $PORT --host 127.0.0.1 --num_workers 2
PIDS=($(ss -ltnp "sport = :$PORT" | grep -o 'pid=[0-9]*' | cut -d= -f2 | sort -u))
rss() { for p in ${PIDS[@]}; do awk '/VmRSS/{printf "%d\n", $2/1024}' /proc/$p/status; done | awk '{s+=$1; per=per $1 "/"} END{printf "%dMB(%s)", s, per}'; }
echo "proxy pids ${PIDS[*]} rss_start=$(rss)"
while true; do echo "$(date -u +%T) liveliness $(curl -s -o /dev/null -w '%{time_total}' http://127.0.0.1:$PORT/health/liveliness)s"; sleep 0.5; done > liveness.log &
for i in $(seq 1 300); do
out=$(curl -s -o /dev/null -D hdr.txt -w '%{http_code} %{time_total}' -m 300 -X POST http://127.0.0.1:$PORT/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @lit7392_req.json)
code=${out%% *}; t=${out#* }
retries=$(tr -d '\r' < hdr.txt | awk 'BEGIN{r="?";f="?"} tolower($1)=="x-litellm-attempted-retries:"{r=$2} tolower($1)=="x-litellm-attempted-fallbacks:"{f=$2} END{printf "%s %s", r, f}')
printf 'req %03d http=%s %ss retries/fallbacks=%s rss=%s\n' $i $code $t "$retries" $(rss)
done
python3 -c 'v=sorted(float(l.split()[2][:-1]) for l in open("liveness.log"));n=len(v);print(f"probes={n} p50={v[n//2]:.3f}s p99={v[int(n*.99)]:.3f}s max={v[-1]:.3f}s")'Before (ecc04bf)Both workers grow steadily and never come back down: worker A goes 435MB -> 520MB and worker B 460MB -> 515MB over 300 requests, about 19MB summed per 50 requests, with the slope unchanged at the end of the run Retry-heavy /v1/chat/completions load with a flaky deployment that fails over
All 300 requests returned 200 with /health/liveliness latency across 2 workers during the load
On this box (no swap, 300 requests) the growth did not yet slow the probe; the memory slope is the signal here After (a9ea571)Same rig, same config, same 300 requests: after the workers finish warming up around request 50, RSS is flat. Worker A goes 464MB -> 468MB and worker B 463MB -> 469MB from request 50 to 300 (10MB summed), against 95MB summed over the same span before Retry-heavy /v1/chat/completions load with a flaky deployment that fails over
All 300 requests returned 200 with /health/liveliness latency across 2 workers during the load
Verdict: the fallback path behaves identically on both legs (200, retries 0, fallbacks 1 on every request), and the per-request memory growth on the merge base is gone at the tip |
…00.1) (#544) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.100.0` → `v1.100.1` | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/141) for more information. --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.100.1`](https://github.com/BerriAI/litellm/releases/tag/v1.100.1) [Compare Source](BerriAI/litellm@v1.100.0...v1.100.1) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.100.1 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.100.1/cosign.pub \ ghcr.io/berriai/litellm:v1.100.1 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#​39568](BerriAI/litellm#39568) and [#​39572](BerriAI/litellm#39572) to stable/1.100.x and cut 1.100.1 by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​40176](BerriAI/litellm#40176) - fix(router): backport [#​39491](BerriAI/litellm#39491) to stable/1.100.x so retry breadcrumbs stop retaining every earlier request by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​40455](BerriAI/litellm#40455) - revert: drop the spend attribution backports from stable/1.100.x by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​40495](BerriAI/litellm#40495) **Full Changelog**: <BerriAI/litellm@v1.100.0...v1.100.1> </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC43NS4wIiwidXBkYXRlZEluVmVyIjoiNDQuNzkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.hayden.moe/hayden/phoebe/pulls/544
1 participant
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a team running the stable 1.100.x image behind a flaky deployment watches their proxy pod slow down over the day and then crash-loop on out-of-memory
"model": "flaky-chat"and a normal 20KB chat payload; the flaky deployment times out and the proxy fails over to the healthy onex-litellm-attempted-retries: 0andx-litellm-attempted-fallbacks: 1(the flaky deployment fails, the router moves to the healthy one), and their app carries onAfter: the same traffic against the same config keeps a flat memory footprint and fast health checks
"model": "flaky-chat"and the same 20KB payload; the flaky deployment times out and the proxy fails over to the healthy onex-litellm-attempted-retries: 0andx-litellm-attempted-fallbacks: 1(the flaky deployment fails, the router moves to the healthy one), and their app carries onRelevant issues
Backports #39491 (merged to litellm_internal_staging as
7bc2d0b06e2, shipped in v1.101.0-rc.1) onto stable/1.100.x. Also carries the router hunk of7c87451eadf, which binds the breadcrumb tuple to aFinallocal and marks the write-back# rebind-ok, so the port matches what staging runs today. The regression entered the line in721227e9eeLinear ticket
Resolves LIT-7392
What is included
7bc2d0b06e2on internal staging with the original author preservedlitellm/router.pyhunk of7c87451eadf(chore(router): document the breadcrumb write-back and ratchet lint budgets), folded into the same commit; its budget JSON edits were not picked, since budgets are never edited on a PR brancha9ea5713ab9test(router): type the breadcrumb test helpers, a follow-up commit on this branch only: the three helpers arrived untyped from the source commit (they still are on internal staging) and Greptile flagged them hereAdaptation notes
The pick applied without conflicts. The only divergence from the source commits is the parameter and return annotations on the three test helpers in
a9ea5713ab9.make checkpasses on the branch andpytest tests/test_litellm/test_router.py -k breadcrumbpasses (6 tests)Known noise on this line
tests/test_litellm/test_router_per_deployment_num_retries.py::TestNoProviderRetryAmplification::test_direct_completion_still_forwards_num_retries_to_providerfails in this rig at both the merge baseecc04bf8112and the tip with the sameassert 3 > (2 + 1), while a second pass ran the same file at both commits in a clean worktree and got 28 passed, so the failure is environment or ordering on this box and not from this PR. The other 90 tests in the router retry and fallback files pass at the tipCircleCI at the tip:
local_testing_part1andlocal_testing_part2fail onbedrock/cohere.command-r-plus-v1:0, which Bedrock now answers with "This model version has reached the end of its life";tests/local_testing/test_completion.pyandtest_streaming.pystill name that model on stable/1.100.x and no longer do on internal staging, and the same two jobs failed the same way at76b5fec1c51a.litellm_router_testingfailed once ontests/local_testing/test_router.py::test_router_timeout, which asks for a 0.0001s timeout and got a connection error first; the job passed at76b5fec1c51aon the samerouter.py, and the rerun from failed passed it at this tip (CircleCI workflow dd9cde72), leaving the two Bedrock EOL failures as the only red jobs.e2e_openai_endpointsfails ontests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_bad_request_error, which expects a 400 fornon-existent-modeland now gets OpenAI's 404; that test no longer exists in this form on internal staging and nothing in this PR touches the Responses pathRelease
v1.100.1 has not been cut yet: pyproject on this line already reads 1.100.1 from #40176, so there is no version bump here. Cutting v1.100.1 from the merged tip is the human-gated release step
Pre-Submission checklist
Screenshots / Proof of Fix
Both legs ran the same rig: a proxy booted from the named commit with
--num_workers 2on 127.0.0.1:34698 (no database), threeopenai/gpt-5-minideployments where the twoflaky-chatones carrytimeout: 0.02so every request times out there and fails over tohealthy-chat(num_retries: 2,disable_cooldowns: true,fallbacks), and 300 sequential POST /v1/chat/completions with a 21.8KB payload (max_tokens: 5) against real OpenAI. Memory isphys_footprintfrom/usr/bin/footprint, summed over the three processes listening on the port (the uvicorn parent and its two workers) and also printed per process, sampled after every request. A second loop hitGET /health/livelinessevery 0.5s during the load and recorded each probe'stime_total. The box was under heavy swap pressure from other work during both legs, so read the footprint columns as a trend, not as absolute numbersBefore (ecc04bf)
Memory grows request after request on both workers and the health probe gets slower: the summed footprint climbs from 2412MB at request 50 to 2566MB at request 300 (worker A 786MB -> 897MB, worker B 799MB -> 886MB), and
/health/livelinessreaches 0.45secc04bf8112(the stable/1.100.x tip) and read the listening pidspython3 -c 'v=sorted(float(l.split()[2][:-1]) for l in open("liveness.log"));n=len(v);print(f"probes={n} p50={v[n//2]:.3f}s p99={v[int(n*.99)]:.3f}s max={v[-1]:.3f}s")'299 of 300 requests returned 200 with
x-litellm-attempted-retries: 0andx-litellm-attempted-fallbacks: 1(12 of them retried once or twice first); request 140 came back 408 after 6.4s from the upstream sideAfter (76b5fec)
Same rig, same config, same 300 requests: the footprint stays flat once the second worker finishes warming up (2350MB at request 50, 2362MB at request 300; worker A 783MB -> 786MB, worker B 784MB -> 793MB) and the health probe stays at a few milliseconds
76b5fec1c51a(this branch) and read the listening pidspython3 -c 'v=sorted(float(l.split()[2][:-1]) for l in open("liveness.log"));n=len(v);print(f"probes={n} p50={v[n//2]:.3f}s p99={v[int(n*.99)]:.3f}s max={v[-1]:.3f}s")'299 of 300 requests returned 200 with
x-litellm-attempted-retries: 0andx-litellm-attempted-fallbacks: 1(5 retried once first). Request 32's upstream call to the healthy deployment stalled and curl gave up at its 300s limit; the 527 health probes sent during that stall averaged 22ms with a 0.28s max, so the event loop was serving while it waited, and the next request answered in 1.6sObservations from the run:
For reference only (a debugging callback, not part of the User Flow): on the before leg the breadcrumbs on the last request reached 78,564 nested containers and took 198ms to walk on one worker, against 94 containers and under 1ms on every request of the after leg
Dependents of the breadcrumb shape, stable tip vs this branch
The only in-repo reader of
previous_modelsthat decides anything with it islitellm.num_retries_per_request(litellm/utils.py:1451-1455), which compares the setting againstlen(previous_models). Same rig as above withnum_retries: 1andnum_retries_per_request: 4inlitellm_settings, two workers, eight sequential requests per sideStable tip
ecc04bf8112: the router-wide breadcrumb list is already at its cap of 4 by the second request each worker sees, so the per-request check fires on the first retry and the request never reaches the fallbackThis branch
a9ea5713ab9: every request carries only its own three breadcrumbs (two failedflaky-chatattempts plus the fallback hop,LIT7392CB success breadcrumbs=3on all eight), so the check stays under 4 and the fallback answersThe other readers (
litellm/integrations/s3.py,datadog/datadog.py,logfire_logger.pydrop the key while cleaning metadata,langfuse/langfuse.pydenies it as a steering key) never look inside the value, and theLIT7392CBcustom logger above received the breadcrumbs on both sides. Spend log rows were not checked, since the rig runs without a databaseType
🐛 Bug Fix
Caveats (if any)
Low
previous_modelsis now a tuple, not a list, on the stable line too.appendon it would breakproxy_server_requestlitellm/,enterprise/,ui/, or the tests indexes a breadcrumb for it; spend logs still read the snapshot off the request itselflitellm.num_retries_per_requestnow counts this request's failures onlyrun_async_fallbackhands the next hop adict(...)copy on stable and staging alike; the proxy and every logger still see the full chain through that copyFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
a9ea571 passes /live-pr-risk