test(e2e): memory regression test for failing requests on the release gate

[devin-ai-integration]

TLDR

Problem this solves:

• v1.100.0 shipped a retry-breadcrumb leak the e2e gate never looked at
• Every failed attempt copied the whole request; the proxy grew until OOM-killed
• Nothing on the release gate measured proxy memory or stored request size

How it solves it:

• New e2e test drives 300 failing chat requests with retries and a fallback
• One probe request first: its stored spend-log request body must stay under 64 KB
• Then warmup phase, measured phase: per-worker RSS growth must stay under 48 MB, each checkpoint sampling until no new worker has answered for the settle window, and both checkpoints must see the same workers so a leaking worker reached by one checkpoint alone cannot drop out of the comparison
• Runs on the default-collected e2e suite, so the release gate fails on a leaky RC
• The gateway now serves the authed, read-only /debug/memory/summary on the data plane, since the release stack's serving workers are the gateway replicas and their RSS is the reading that matters; /debug/memory/details and /debug/memory/gc/configure stay backend-only
• The helm ingress routes Exact /debug/memory/summary to the gateway too, since the release gate reaches the stack through the chart's ingress and the base chart sent that path to the backend, and the summary now names its hostname so a worker is (replica, pod, pid) across replicas behind one address
• The release image installs no psutil (it is only a locust and mirakuru dev dependency), so the summary reads /proc/self/statm and /proc/meminfo when psutil is missing instead of answering with an install hint and no RSS

User Flow

Before: a release cut with a v1.100.0-style leak passes the e2e gate and ships

1. The release owner tags a v1.100.0-style RC and the litellm-e2e stage runs its router suite against it
2. Every test in the suite is about routing verdicts, so the stage goes green with no request ever checked for memory or stored size
3. The RC ships and an operator with retries and fallbacks configured sends POST https://litellm-domain/v1/chat/completions against a deployment whose provider refuses; it comes back HTTP 500 as expected
4. GET https://litellm-domain/spend/logs?request_id= shows a proxy_server_request of about 520 KB for an 80-message transcript, one copy of the request per failed attempt nested inside metadata.previous_models
5. After 300 such failing calls GET https://litellm-domain/debug/memory/summary shows the worker at 1496 MB where it stood at 1357 MB before them, and the pod is OOM-killed a few hours into production traffic

After: the same release cut fails the e2e gate on its first failing request

1. The release owner tags the same RC and the litellm-e2e stage runs its router suite against it
2. The new test registers two model groups whose deployments refuse at the socket, sends one POST https://litellm-domain/v1/chat/completions with an 80-message transcript, 2 retries, and a fallback, and gets the expected HTTP 500 with an x-litellm-call-id
3. It reads GET https://litellm-domain/spend/logs?request_id= and fails the stage with "the spend log of one failing request stored a 298 KB request body, past the 64 KB budget", so the RC never ships
4. On a build with the fix the same request stores a 31 KB body, the test then sends 300 failing calls as warmup and 300 more as the measured phase, reads GET https://litellm-domain/debug/memory/summary per worker before and after, sees under 1 MB of growth against the 48 MB budget, and the stage goes green

Relevant issues

The leak this guards was fixed by #39491 (per-request retry breadcrumbs); v1.100.1 carries it as #40455

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: two python:3.12-slim containers, each with pip install litellm[proxy]==<version> and the release stage's config shape (store_prompts_in_spend_logs: true, json_logs: true, prometheus and otel callbacks, num_retries: 3, allowed_fails: 5, cooldown_time: 30, STORE_MODEL_IN_DB=True, its own Postgres database), plus two static model groups memory-primary and memory-fallback whose deployments point at http://127.0.0.1:9/v1 with cooldown_time: 0. body.json is a 40-turn chat transcript plus one user turn with "model": "memory-primary", "num_retries": 2, "fallbacks": ["memory-fallback"]. LITELLM_MASTER_KEY is the container's master key. The leaky build is v1.100.0 (the release the customer hit), the fixed build is v1.101.0rc1 (the first tag carrying #39491). Both legs run single worker, the shape the release stage runs, with one process per container

Before (ac66754)

the leaky release passes the gate

1. ls tests/e2e/router/ at the merge base: reliability_support.py test_reliability_e2e.py, no test reads /debug/memory/summary or a stored request size, so python -m pytest tests/e2e/router against v1.100.0 goes green on routing verdicts alone
2. What that green hides, on the v1.100.0 container:

   $ docker exec mat335-linux-v100-db pip show litellm | grep ^Version
   Version: 1.100.0
   
   $ curl -s -D - -o /dev/null http://127.0.0.1:22377/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @body.json | grep -iE '^HTTP|x-litellm-call-id'
   HTTP/1.1 500 Internal Server Error
   x-litellm-call-id: 4ad0151a-6f16-4897-b170-89184e12401f
   
   $ curl -s "http://127.0.0.1:22377/spend/logs?request_id=4ad0151a-6f16-4897-b170-89184e12401f" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.[0].proxy_server_request | tojson | length'
   520746
   
   $ curl -s "http://127.0.0.1:22377/spend/logs?request_id=4ad0151a-6f16-4897-b170-89184e12401f" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.[0].proxy_server_request.metadata.previous_models | length'
   4
   
   $ curl -s http://127.0.0.1:22377/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c .memory
   {"summary":"1357.4 MB (17.1% of system memory)","ram_usage_mb":1357.45,"system_memory_percent":17.13}
   
   $ seq 300 | xargs -P4 -I{} curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:22377/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @body.json | sort | uniq -c
    300 500
   
   $ sleep 15; curl -s http://127.0.0.1:22377/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c .memory
   {"summary":"1496.2 MB (18.9% of system memory)","ram_usage_mb":1496.25,"system_memory_percent":18.88}
   

3. One failing request stored a 520 KB copy of an 8 KB request and 300 of them grew the worker by 139 MB, and the gate had nothing that would notice either

After (93e2393)

the leaky release fails the gate

1. Same v1.100.0 container, same curls as above (520,746 char stored body, 4 nested request copies, 1357 MB to 1496 MB over 300 failing calls)
2. The gate now runs the new test against it, with LITELLM_PROXY_URL=http://127.0.0.1:22377 and the container's master key:

   $ cd tests/e2e && python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider
   router/test_reliability_memory_e2e.py::test_failing_requests_do_not_grow_rss_or_stored_request FAILED
   E       AssertionError: the spend log of one failing request stored a 2050 KB request body, past the 64 KB budget for a 80-message transcript with 2 retries and a fallback; the retry breadcrumbs are copying the whole request into the stored snapshot the way the v1.100.0 ones did
   router/test_reliability_memory_e2e.py:216: AssertionError
   ========================= 1 failed in 71.92s (0:01:11) =========================
   

3. The probe read 298 KB on a fresh proxy at dcf8228 (the very first failing request) and 2050 KB on this re-run against the same container after the earlier runs' failing calls: every further failing call nests one more copy, which is the leak

the fixed release passes the gate

1. On the v1.101.0rc1 container:

   $ docker exec mat335-linux-rc1-db pip show litellm | grep ^Version
   Version: 1.101.0rc1
   
   $ curl -s -D - -o /dev/null http://127.0.0.1:28082/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @body.json | grep -iE '^HTTP|x-litellm-call-id'
   HTTP/1.1 500 Internal Server Error
   x-litellm-call-id: b18b2a33-16ca-4858-8fde-46a6b54ccc1b
   
   $ curl -s "http://127.0.0.1:28082/spend/logs?request_id=b18b2a33-16ca-4858-8fde-46a6b54ccc1b" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.[0].proxy_server_request | tojson | length'
   31648
   
   $ curl -s "http://127.0.0.1:28082/spend/logs?request_id=b18b2a33-16ca-4858-8fde-46a6b54ccc1b" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.[0].proxy_server_request.metadata.previous_models | length'
   4
   
   $ curl -s http://127.0.0.1:28082/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c .memory
   {"summary":"668.7 MB (8.4% of system memory)","ram_usage_mb":668.68,"system_memory_percent":8.44}
   
   $ seq 300 | xargs -P4 -I{} curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:28082/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d @body.json | sort | uniq -c
    300 500
   
   $ sleep 15; curl -s http://127.0.0.1:28082/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c .memory
   {"summary":"668.8 MB (8.4% of system memory)","ram_usage_mb":668.75,"system_memory_percent":8.44}
   

2. Same four breadcrumbs, but each one is a per-request tuple rather than a nested copy of the whole request: 31 KB stored, and 300 failing calls move the worker by 0.1 MB
3. The gate's verdict on it, with LITELLM_PROXY_URL=http://127.0.0.1:28082:

   $ cd tests/e2e && python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider
   router/test_reliability_memory_e2e.py::test_failing_requests_do_not_grow_rss_or_stored_request PASSED
   ======================== 1 passed in 152.26s (0:02:32) =========================
   

Observations from the run:

• v1.100.0's spend-log writer drops its own rows under this traffic
• 106 "spend log queue is at its 64000000 byte budget" lines during 600 calls
• 23 "Error creating spendlogs object - maximum recursion depth exceeded" lines
• Hence the probe runs first, while the writer is idle; PR leaves the writer alone
• v1.100.0 and v1.100.1 idle at 1.3 GB RSS with a DB, rc1 at 560 MB; PR leaves it alone
• v1.100.1 (published 2026-09-10) already carries fix(router): keep retry breadcrumbs per request and out of the request snapshot #39491 as fix(router): backport #39491 to stable/1.100.x so retry breadcrumbs stop retaining every earlier request #40455 and passes the test

the two-gateway release stack can read every serving worker's RSS (e4f958d)

Same stack shape as the stage-mirror e2e job, on this branch's proxy code: backend.main:app on 45127 as the control plane, two gateway.main:app --workers 1 replicas on 45125 and 45126 behind nginx on 45124, a fresh Postgres 16.6 migrated with migrations/run.py, STORE_MODEL_IN_DB=True, DISABLE_SCHEMA_UPDATE=true, and the stage config minus its redis cache and vendor callbacks. LITELLM_PROXY_URL=http://127.0.0.1:45124 LITELLM_CONTROL_PLANE_URL=http://127.0.0.1:45127 LITELLM_PROXY_REPLICA_URLS=http://127.0.0.1:45125,http://127.0.0.1:45126

1. At 93e2393 the gateways trim every /debug route, so the test's RSS read 404s on both replicas after the probe and warmup had already passed (this is the failure the stage-mirror job showed on that tip):

   $ for p in 45125 45126 45124; do curl -s -o /dev/null -w "$p %{http_code}\n" http://127.0.0.1:$p/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY"; done
   45125 404
   45126 404
   45124 404
   $ cd tests/e2e && python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider
   router/test_reliability_memory_e2e.py::test_failing_requests_do_not_grow_rss_or_stored_request FAILED
   E               AssertionError: kind='unknown' status_code=404 body='{"detail":"Not Found"}'
   e2e_http.py:216: AssertionError
   ========================= 1 failed in 79.38s (0:01:19) =========================
   

2. At e4f958d, with the two gateways restarted on the new allowlist, the same curls and the same run:

   $ for p in 45125 45126 45124; do curl -s -o /dev/null -w "$p %{http_code}\n" http://127.0.0.1:$p/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY"; done
   45125 200
   45126 200
   45124 200
   $ curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:45125/debug/memory/summary
   401
   $ curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:45125/debug/memory/details -H "Authorization: Bearer $LITELLM_MASTER_KEY"
   404
   $ curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:45125/debug/memory/gc/configure?generation_0=700" -H "Authorization: Bearer $LITELLM_MASTER_KEY"
   404
   $ cd tests/e2e && python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider
   router/test_reliability_memory_e2e.py::test_failing_requests_do_not_grow_rss_or_stored_request PASSED
   ======================== 1 passed in 113.88s (0:01:53) =========================
   

3. The v1.100.0 and v1.101.0rc1 legs above ran at 93e2393; e4f958d, 5fdb086 and eca7bb1 change the gateway allowlist, the helm chart, debug_utils.py, the test's worker key and unit tests, none of which those pip-installed release containers load. 77a0053 changes the test's checkpoint rule, 9375719 merges staging into the branch, and 9ba7ec2 moves the test into a class, so both legs ran again at 9ba7ec2 below

the release gate's ingress sends the RSS read to the gateways (5fdb086)

The Buildkite gate reaches the stack through nginx rendered from the chart's ingress (helm template -s templates/ingress.yaml, then litellm_release ingress-to-nginx). Same three processes as above, with nginx on 45124 rebuilt from the base chart's ingress and then from this branch's, six summary reads each, and the summary's new hostname next to the pid:

$ zsh router_proof.sh
## process map (direct, bypassing the router)
gateway-1 :45125 -> Mac.localdomain pid 73371
gateway-2 :45126 -> Mac.localdomain pid 73376
backend :45127 -> Mac.localdomain pid 73366

## router built from the base ingress: 6x GET /debug/memory/summary through http://127.0.0.1:45124
Mac.localdomain pid 73366 rss_mb 72.0
Mac.localdomain pid 73366 rss_mb 74.0
Mac.localdomain pid 73366 rss_mb 74.0
Mac.localdomain pid 73366 rss_mb 74.1
Mac.localdomain pid 73366 rss_mb 74.1
Mac.localdomain pid 73366 rss_mb 77.0

## router built from the tip ingress: 6x GET /debug/memory/summary through http://127.0.0.1:45124
Mac.localdomain pid 73376 rss_mb 63.8
Mac.localdomain pid 73371 rss_mb 62.7
Mac.localdomain pid 73376 rss_mb 66.2
Mac.localdomain pid 73371 rss_mb 63.7
Mac.localdomain pid 73376 rss_mb 63.0
Mac.localdomain pid 73371 rss_mb 63.8

## chat through the tip router still lands on the gateways (a real gpt-5.5 call, 200)
POST /v1/chat/completions -> 200

The base ingress hands every summary read to the backend (pid 73366, the control plane, not a serving worker); the tip ingress alternates the two gateway workers, and a chat completion through the same router still lands on the gateways. helm unittest helm/litellm: 142 passed, including the new assertion on the Exact path and the rejection of an extraPaths entry that would take it over

the release image reports RSS without psutil (eca7bb1)

Buildkite e2e-tests build #11761 at e4f958d failed on this test with AssertionError: no /debug/memory/summary read carried ram_usage_mb, so the proxy cannot report its RSS after the probe and the warmup had passed: the release image is built with uv sync --frozen --no-default-groups --extra proxy ... and psutil is only a locust and mirakuru dependency in uv.lock, so the summary answered Install psutil for memory monitoring and no RSS. The stage-mirror job never saw it because its uv sync installs the dev group. With the fallback, the /proc reader inside a psutil-free Linux image against the kernel's own VmRSS:

$ docker run --rm -i python:3.13-slim python - < proc_probe.py
statm rss kB 15512 status VmRSS kB 15760
statm vms kB 19952 status VmSize kB 19952
percent 0.195
psutil absent

The summary on this branch's stack (macOS, psutil present, so the psutil branch), direct on each gateway and through the tip router, plus the details route staying backend-only:

$ zsh tip_summary_proof.sh
## eca7bb11ce: the summary on each serving worker, direct and through the tip router
:45125 -> Mac.localdomain pid 12619 status healthy ram_usage_mb 244.08 system_memory_percent 0.99 error None
:45126 -> Mac.localdomain pid 12626 status healthy ram_usage_mb 242.78 system_memory_percent 0.99 error None
:45124 -> Mac.localdomain pid 12626 status healthy ram_usage_mb 241.08 system_memory_percent 0.98 error None
## /debug/memory/details stays backend-only
:45125 details -> 404

both release legs again at 9ba7ec2, the tree merged with staging and the test in a class

9375719 merges litellm_internal_staging into the branch (both sides kept in tests/e2e/CLAUDE.md and tests/e2e/models.py), a8ffc85 and 9ba7ec2 trim then drop the helper docstrings, and 9ba7ec2 moves the test into TestReliabilityMemory, so both release legs ran again with the test at 9ba7ec2. Same two python:3.12-slim containers as the release legs above (v1.100.0 on 22377, v1.101.0rc1 on 28082, each with its own Postgres and the release stage's config shape), pointed at one address each:

$ cd tests/e2e && LITELLM_PROXY_URL=http://127.0.0.1:28082 python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider --durations=5
router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request PASSED [100%]
175.32s call     router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request
======================== 1 passed in 175.57s (0:02:55) =========================

$ cd tests/e2e && LITELLM_PROXY_URL=http://127.0.0.1:22377 python -m pytest router/test_reliability_memory_e2e.py -o addopts="" -v -p no:cacheprovider --durations=5
router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request FAILED [100%]
E       AssertionError: the spend log of one failing request stored a 568 KB request body, past the 64 KB budget for a 80-message transcript with 2 retries and a fallback; the retry breadcrumbs are copying the whole request into the stored snapshot the way the v1.100.0 ones did
E       assert 568.423828125 <= 64.0
========================= 1 failed in 66.01s (0:01:06) =========================

The merged debug_utils.py (staging's #40713 rewrote the cache counts next to the summary) on this branch's three-process stack booted from a8ffc85 (9ba7ec2 changes only tests/e2e, so the proxy, gateway, and chart bytes are the same), the summary through the tip router (nginx on 45124) and direct, the other debug routes, and a plain virtual key from /key/generate on the new gateway route:

$ zsh lpr_curls.sh
gateway 45125 authed summary: 200
gateway 45126 authed summary: 200
gateway 45124 authed summary: 200
gateway 45125 no auth summary: 401
gateway 45125 bad key summary: 401
gateway 45125 authed details: 404
gateway 45125 authed gc/configure: 404
backend 45127 authed summary: 200
backend 45127 authed details: 200
gateway 45125 chat still served: 200
$ for i in 1 2 3 4; do curl -s http://127.0.0.1:45124/debug/memory/summary -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '"\(.hostname) pid \(.worker_pid) rss_mb \(.memory.ram_usage_mb)"'; done
Mac.localdomain pid 52833 rss_mb 63.11
Mac.localdomain pid 52813 rss_mb 112.08
Mac.localdomain pid 52833 rss_mb 65.64
Mac.localdomain pid 52813 rss_mb 112.34
$ zsh plain_key_probe.sh
gateway 45125 plain key summary: {"error":{"message":"Authentication Error, Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/debug/memory/summary. Your role=unknown. ...
backend 45127 plain key details: {"error":{"message":"Authentication Error, Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/debug/memory/details. Your role=unknown. ...
gateway 45125 plain key /health: 200

helm unittest -f 'tests/*_tests.yaml' helm/litellm at a8ffc85: 150 passed; the harness's own tests (tests/e2e/test_*.py) at 9ba7ec2: 182 passed

the real release gate runs it and passes (9ba7ec2)

The Buildkite e2e-tests gate that the RC and stable tags wait on: build #11761 (e4f958d) failed on the release image having no psutil, so every summary read came back without ram_usage_mb, which is what eca7bb1 fixes; build #11828 at 77a0053 triggered litellm-e2e-pr build 411, which deployed the chart at that commit and ran the full suite, this test included, in 57m36s; build #12285 at the final tip 9ba7ec2 triggered litellm-e2e-pr build 449, which deployed the chart at this commit and ran the full suite again, the test now under its class node id, in 58m39s:

$ gh api repos/BerriAI/litellm/commits/9ba7ec2964/status --jq '.statuses[] | select(.context=="buildkite/e2e-tests") | .description'
Build #12285 passed (58 minutes, 39 seconds)

litellm-e2e-pr build 449, job "e2e — selected tests":
router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request
[gw4] [ 99%] PASSED router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request
================== 926 passed, 58 skipped, 14 warnings in 2335.88s (0:38:55) ==================

The stage-mirror GitHub Actions job (e2e-changed-tests, run 34668570996) ran the selected files three times at 9ba7ec2 (tests/e2e/router/test_reliability_memory_e2e.py: 1 collected, 0 skipped in each pass, then 20 passed in 306.62s, 306.44s, and 313.37s); run 34556104459 did the same at 77a0053

Type

✅ Test

Caveats (if any)

Low

• RSS alone would not have failed v1.100.0 on the release stack: +8 to +15 MB per 300 calls against the 48 MB budget, so a leak that copies less than the whole request per attempt could stay under it
  • The stored-request probe is the deterministic catch for the breadcrumb shape that shipped, and 48 MB is only there for a gross leak
  • Tightening the budget toward 15 MB is net-negative: the gate runs the test three times per job on shared runners, and a budget inside the allocator's noise turns the release gate flaky
• The budgets are chosen from the measured runs, not derived: 64 KB is about twice the 31 KB the fixed build stores for the 80-message transcript and a fifth of the 298 KB v1.100.0 stores on its first failing request; 48 MB is about three times the largest v1.100.0 growth seen on the CI-shaped stack and about a third of the 139 MB seen with no warmup. Both are env-tunable (E2E_MEMORY_STORED_REQUEST_BUDGET_KB, E2E_MEMORY_RSS_BUDGET_MB)
• The stored-request check needs store_prompts_in_spend_logs: true and a spend-log writer, and /model/new needs STORE_MODEL_IN_DB=True; the stage-mirror config has both, and a stack without them fails on "no spend log row appeared" or "stored no request body" rather than passing vacuously
• The release legs run one container with one worker each; RSS growth is keyed by replica address and worker pid, and the two-replica shape (two gateways, one worker each, behind one address) is the pasted e4f958d run plus this PR's stage-mirror e2e job, which runs the test three times
• The gateway gains one authed read-only route on the data plane, /debug/memory/summary (worker pid, hostname, RSS, gc counters); leaving it backend-only is net-negative because the backend's RSS is not the serving worker's, and the mutating and heavier debug routes stay off the gateway
• The ticket's shape named a custom callback that json-dumps kwargs and a 413-sized transcription leg; the release stack already serializes kwargs through its datadog, otel and arize callbacks, and the transcription leg is dropped since the chat leg reproduces the same breadcrumb copies
• The ticket's p99-latency check is dropped: on both builds the failing-call p99 moved within noise across the phases, so a budget on it would only add a flaky assertion
• Two 400 Missing required parameter: 'messages' in an earlier rc1 curl loop were a curl body artifact, not a proxy behavior; the pasted run has none
• The chart's ingress and the gateway allowlist name the gateway routes independently, so a route added to one has to be added to the other by hand; the helm test that asserts the Exact /debug/memory/summary path is the guard for this one, and the real gate (build Add Elasticsearch Logging Tutorial #11761) is what caught the mismatch
• Only the summary gets the /proc fallback; /debug/memory/details still needs psutil for its file-handle and thread counts and keeps answering with the install hint on the release image, which nothing on the gate reads
• On a non-Linux host with no psutil the summary still carries error and no ram_usage_mb, and the test fails on "no /debug/memory/summary read carried ram_usage_mb" rather than passing vacuously
• The two checkpoints must see the same workers, so a worker that restarts between them fails the test rather than dropping out: a restart under failing traffic is what an OOM kill looks like, so that failure is the point, and a stack with more workers than a 15-read window reaches fails on "first seen after it" with E2E_MEMORY_RSS_SETTLE_SAMPLES as the knob; the sampling is capped at four windows so a stack that keeps producing new pids cannot hang the gate
• Running the test on macOS against a single address fails for reasons that do not exist on Linux: the memory compressor lowers the warm minimum (62.7 MB against a 115 MB idle) and the 300-request batch bulges about 400 MB of in-flight work that drains in about 25 s; the same run against a Linux container holds RSS flat (667.8 to 668.3 MB through both batches, 1 s sampler), which is the shape the gate runs on

QA runbook

• tests/e2e/router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request - one failing request with retries and a fallback stores a request body under 64 KB in its spend log, and 300 more such requests after a 300-request warmup grow no worker's RSS by more than 48 MB
  • Boot the proxy with STORE_MODEL_IN_DB=True, store_prompts_in_spend_logs: true under general_settings, and a database, then export LITELLM_PROXY_URL and LITELLM_MASTER_KEY
  • Register two deployments through POST /model/new, each with api_base: http://127.0.0.1:9/v1, a placeholder api_key, and cooldown_time: 0 in litellm_params, under two fresh model group names (primary and fallback)
  • Send one POST /v1/chat/completions with the primary model, an 80-message transcript, "num_retries": 2, and "fallbacks": ["<fallback group>"], and expect HTTP 500 with an x-litellm-call-id response header
  • GET /spend/logs?request_id= and expect one row whose proxy_server_request serializes to under 64 KB (about 31 KB on a build with fix(router): keep retry breadcrumbs per request and out of the request snapshot #39491; about 300 KB on v1.100.0)
  • GET /debug/memory/summary with the master key and note memory.ram_usage_mb per worker_pid
  • Send 300 of the same failing requests at concurrency 4 as warmup, read the summary once a second until no new worker has answered for 15 reads, and keep the lowest ram_usage_mb per worker as the baseline
  • Send 300 more, sample the same way, and expect the same set of workers as the baseline with every worker's lowest reading within 48 MB of its baseline (rc1 moves under 1 MB; v1.100.0 moves about 8 to 15 MB on this stack and 139 MB with no warmup)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• 93e2393 passes /live-pr-risk
• e4f958d passes /live-pr-risk
• 77a0053 passes /live-pr-risk
• a8ffc85 passes /live-pr-risk
  • 9ba7ec2 after it changes only tests/e2e (the test moves into a class and four helper docstrings go), so the proxy, gateway, and chart it exercised are byte-identical there

Reopened from #40640 with the same commits so the approval can come from someone other than the author

---

Note

Medium Risk
Exposes one authenticated read-only debug route on the data-plane gateway and changes memory summary behavior (proc fallback, hostname); ingress/allowlist must stay in sync or RSS reads misroute.

Overview
Adds a release-gate e2e memory regression (reliability.perf.memory.under_slo) that reproduces v1.100.0-style retry traffic: failing chat completions with retries, fallbacks, and a long transcript must keep the spend-log stored request under ~64 KB and per-worker RSS growth (warmup vs measured batch) under ~48 MB, sampling /debug/memory/summary on every replica with settle logic keyed by (replica, hostname, pid).

Routing and observability so the test works on gateway-heavy stacks: /debug/memory/summary is allowlisted on the gateway and routed Exact to gateway in Helm ingress (other /debug/memory/* stay backend-only). The summary endpoint gains hostname, refactored RSS helpers, and a /proc fallback when psutil is absent (release images). Harness additions include memory_summary_everywhere, typed MemorySummaryResponse, env-tunable E2E_MEMORY_* knobs, and reliability helpers (cooldown_time: 0 refusing deployments, optional chat history).

^{Reviewed by Cursor Bugbot for commit 9ba7ec2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[mateo-berri]

LGTM

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_memory_regression_failing_requests (9ba7ec2) with litellm_internal_staging (e65b7ff)}

[greptile-apps]

Greptile Summary

This PR adds a release-gate regression test for request-copy memory growth and stored request size during retries and fallbacks. It exposes authenticated per-worker memory summaries through the gateway, routes that exact endpoint through Helm ingress, and supports RSS reporting from Linux procfs when psutil is unavailable

• Adds warmup and measured failure-traffic phases with per-worker RSS comparisons
• Verifies the stored request snapshot remains within a fixed size budget
• Keeps heavier and mutating memory debug routes backend-only
• Adds Helm, gateway allowlist, procfs reader, and e2e coverage
• Recent changes remove unnecessary helper docstrings and place the feature test in a focused class

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule failures

The recent changes fully address the prior style findings without changing test behavior. The earlier access-control concern is not valid because non-admin keys without an explicit allowed route reach the admin-only rejection path, consistent with mateo-berri's live verification

Important Files Changed

Filename	Overview
tests/e2e/router/test_reliability_memory_e2e.py	Adds the regression scenario and now places it in the required focused test class
litellm/proxy/common_utils/debug_utils.py	Adds hostname identification and a Linux procfs fallback for process RSS reporting
gateway/routes/allowlist.py	Exposes only the authenticated read-only memory summary route on gateway workers
helm/litellm/templates/ingress.yaml	Routes the exact memory summary path to gateway workers while preserving backend handling for other debug paths
tests/e2e/router/reliability_support.py	Adds refusing-deployment and transcript support used by the memory regression test

_{Reviews (3): Last reviewed commit: "test(e2e): keep the memory regression ca..." | Re-trigger Greptile}

[greptile-apps]

Operational Data Exposed

Ordinary unscoped virtual keys can access this gateway route, exposing worker hostnames, PIDs, memory, GC state, and cache counts.

How this was verified: The route is absent from master-only lists, and unscoped virtual keys pass route authorization.

Rule Used: What: Fail any PR which may contains a security incident on litellm's authentication layer Why: Do not cause security incidents Bad: ```python # Check cache first cache_key = ( f"oidc_userinfo_{token[:20]}" # Use fi... (source)

[mateo-berri]

Verified live at a8ffc85: a plain virtual key gets 401 on the gateway summary route, user_api_key_auth admits proxy admins only

[greptile-apps]

Unnecessary Helper Docstrings

These docstrings restate helper behavior. Repository guidance permits comments only for necessary complex logic, so this requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_utils/debug_utils.py	85.71%	7 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9ba7ec2. Configure here.}