fix(caching): close evicted LLM clients so their connections are reclaimed

[yassin-berriai]

TLDR

Problem this solves:

• Evicted LLM HTTP clients kept their connection pools open
• SDK clients are reference cycles, so nothing frees them promptly
• Sockets and TLS buffers pile up until a generational GC sweep
• Long-running proxies grow until the pod hits its memory limit

How it solves it:

• Eviction now hands the client to a deferred closer
• It closes the client once it is out of grace and idle
• Only clients litellm created are closed, never a caller's
• The queue holds clients weakly, so it retains nothing

Relevant issues

Linear ticket

Resolves LIT-4883

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Everything below runs inside ghcr.io/berriai/litellm:main-latest against a real TLS upstream, so the allocator, the socket accounting and the TLS buffers are the ones a pod actually sees. The "before" arm is the litellm the image ships; the "after" arm is this branch mounted at /wt and put on PYTHONPATH, so the image, the config, the traffic and the upstream are identical between arms. Captured at a2c4d54b3c.

The leak

Each call uses a distinct client-cache key, so past 200 calls every call evicts one entry. Nothing forces a collection, because production never forces one; that turned out to be the whole point, since an earlier run of mine that called gc.collect() between batches showed a perfectly flat curve and hid the bug entirely.

docker run --rm -v /tmp/litfix-4883:/w -w /w \
  -e LITELLM_LOCAL_MODEL_COST_MAP=True -e SSL_VERIFY=False \
  -e API_BASE=https://host.docker.internal:8884 \
  --entrypoint python ghcr.io/berriai/litellm:main-latest /w/leak_ab.py 1200

litellm loaded from: /app/.venv/lib/python3.13/site-packages/litellm
BASE tree: no deferred close
calls,cached_entries,live_azure_clients,live_httpx_clients,open_sockets,rss_mb
200,200,200,203,202,277
400,200,400,403,402,338
600,200,600,603,602,404
800,200,227,230,229,457
1000,200,427,430,429,457
1200,200,627,630,629,457

The cache is doing its job; cached_entries never moves off 200. Everything it evicts is still there: live clients and open sockets climb with the call count, and RSS goes from 277 MB to 457 MB. The dip at 800 calls is a gen2 sweep, which is the only thing that ever reclaims them, and RSS does not come back down after it.

After the fix

Same command with this branch on PYTHONPATH, and the grace window shortened to one second so a short run can show what the shipped 900-second window does after fifteen minutes:

litellm loaded from: /wt/litellm
FIXED tree: deferred close armed, grace=1.0s
calls,cached_entries,live_azure_clients,live_httpx_clients,open_sockets,rss_mb
200,200,200,203,202,277
400,200,400,403,402,342
600,200,600,603,422,355
800,200,800,803,414,357
1000,200,338,341,339,358
1200,200,538,541,402,358

Open sockets stop tracking the call count and settle around 400 where the base arm reaches 629. RSS grows 81 MB where the base grew 180 MB, and the curve is flat from 600 calls on rather than climbing. The Python client objects still wait for a gen2 sweep, which is correct: what the fix reclaims is the connection pool each one was holding, and that is where the bytes are.

The fix never costs anything while it waits

The same after-arm at the shipped 900-second grace, so that nothing is reaped during the run:

litellm loaded from: /wt/litellm
FIXED tree: deferred close armed, grace=900.0s
calls,cached_entries,live_azure_clients,live_httpx_clients,open_sockets,rss_mb
200,200,200,203,202,280
400,200,400,403,402,345
600,200,600,603,602,411
800,200,270,273,272,449
1000,200,470,473,472,449
1200,200,670,673,672,449

That reproduces the base arm: 672 sockets and RSS 449 MB, against base's 629 and 457. Which is the point. The queue holds its clients weakly, so a client waiting out its grace window is not retained by it, and if the collector would have got there first it still does.

Live proxy, side by side

Two proxies from the same image and the same 250-deployment config, one on base and one on this branch, taking identical traffic at the same instant. Two sweeps over all 250 deployments make the 200-entry cache evict continuously, then a steady one request a second against a single deployment keeps the cache being read. The fixed proxy is launched through a two-line wrapper that shortens the grace window to 60 seconds so the run fits in minutes rather than the shipped 15.

phase                    base sockets    fixed sockets
idle                                3                3
after sweep 1                     258              258
after sweep 2                     508              508
trickle, 60s in                   505              505
trickle, 90s in                   505              204
trickle, 2 min in                 505              204

The base proxy holds all 508: 200 cached clients plus every client it has evicted. The fixed one settles at 204, which is the 200 entries the cache is actually holding, one grace window after the sweeps stopped. Read this as how long the base proxy holds evicted pools open rather than as a permanent difference: a minute later both arms fall to 5 sockets, because the trickle only touches one deployment and the mock upstream drops the other 500 idle keep-alives. The sustained-load numbers above are the ones that answer the ticket.

The regression this must not repeat

tests/test_litellm/caching/test_llm_client_cache_e2e.py is the suite that exists because closing on eviction broke in-flight requests before. It still passes unchanged, and the mutation that closes evicted clients as soon as the grace window ends, ignoring whether a request is still on the wire, fails the two new in-flight tests:

$ python -m pytest tests/test_litellm/caching/ tests/test_litellm/llms/openai/ \
    tests/test_litellm/llms/azure/ tests/test_litellm/llms/custom_httpx/ -q
1380 passed, 33 skipped in 126.61s

Type

🐛 Bug Fix

Changes

LLMClientCache._remove_key hands the evicted value to EvictedClientCloser, a new leaf module in litellm/caching/. The closer keeps a weak reference and a deadline, and on the cache's read path closes anything that is both past its deadline and idle.

Two conditions, not one. Closing at eviction time is what caused the earlier Cannot send a request, as the client has been closed incident, so a request that was handed the client just before it was evicted has to be allowed to finish. A deadline alone cannot promise that: litellm.request_timeout defaults to 6000 seconds and a stream is bounded only by how long the upstream keeps sending, so any fixed window is beatable. So the deadline covers the case a pool cannot see, a request that holds the client but is momentarily not on the wire, and the pool covers the rest: httpcore reports a non-idle connection for the whole of a response including a stream, and aiohttp holds the connection in _acquired over the same span. A client that is still working is requeued for another window rather than closed.

The queue is a deque per deadline class rather than one list. Every entry shares the same window, so each deque is ordered by deadline and a reap pops what is due and stops, instead of filtering the whole queue on every cache read; with an eviction per request that difference is quadratic. Dead entries a client leaves behind sit at the front and are dropped when the next entry is queued, and the queue is bounded, past which an evicted client is left to the collector rather than letting a cache-churning workload grow it without bound.

A client the caller passed in is left alone, because litellm does not own its lifecycle and the caller goes on using it. Ownership is explicit rather than inferred: set_cache takes a litellm_owned_client flag. It is not enough to ask whether the caller passed the SDK client, because _get_async_http_client / _get_sync_http_client hand back litellm.aclient_session / litellm.client_session when one is configured, and the SDK's close() closes whatever http client it was given. So ownership is derived from the wrapped http client, and a wrapper built around one of those shared sessions is never queued.

The closer only ever acts on the event loop the client was evicted on, and a client evicted on another loop is left alone rather than having work scheduled onto a loop it does not belong to. A client whose close is synchronous carries no loop constraint and is closed from wherever the cache is next used.

QA runbook

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR adds deferred cleanup for LiteLLM-owned clients evicted from the LLM client cache while preserving caller-owned and active clients

• Queues evicted clients weakly and closes them after a grace period when their connection pools are idle
• Supports synchronous cleanup outside an event loop and loop-affine asynchronous cleanup
• Propagates explicit ownership metadata through the HTTPHandler, OpenAI, and Azure client creation paths
• Adds coverage for ownership, event-loop handling, active requests, bounded queue behavior, and synchronous eviction cleanup

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains; synchronous owned clients evicted without a running event loop are queued in the closable-anywhere bucket and subsequently closed after the grace period

Important Files Changed

Filename	Overview
litellm/caching/evicted_client_closer.py	Adds bounded, weak-reference deferred cleanup with synchronous and event-loop-specific queues; the previously reported synchronous cleanup gap is fixed
litellm/caching/llm_caching_handler.py	Integrates ownership tracking and deferred scheduling into LLM client cache eviction and read paths
litellm/llms/custom_httpx/http_handler.py	Marks newly created synchronous and asynchronous HTTP handlers as LiteLLM-owned
litellm/llms/openai/common_utils.py	Adds explicit wrapped-session ownership detection before cached SDK clients become eligible for cleanup
litellm/llms/openai/openai.py	Preserves the selected HTTP client and propagates its ownership when caching OpenAI SDK clients
litellm/llms/azure/common_utils.py	Propagates Azure SDK client ownership without claiming caller-provided clients or shared sessions
tests/test_litellm/caching/test_evicted_client_closer.py	Covers deferred cleanup, synchronous eviction without an event loop, loop affinity, active requests, weak retention, and queue bounds
tests/test_litellm/caching/test_llm_caching_handler.py	Verifies cache integration closes owned clients after grace while preserving caller-owned clients

_{Reviews (4): Last reviewed commit: "fix(caching): close evicted LLM clients ..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

[codecov]

Codecov Report

❌ Patch coverage is 94.51220% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/evicted_client_closer.py	95.00%	7 Missing ⚠️
litellm/caching/llm_caching_handler.py	87.50%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head d4f889fd10.

You were right about the synchronous path. _get_httpx_client caches an HTTPHandler whose entry is evicted from call sites with no running loop, and schedule() dropped exactly those, so sync clients kept leaking their pools.

The loop requirement now applies only where it is real. A queued close records whether the client's close is a coroutine; one that closes synchronously is closed from wherever the cache is next used, with no loop and no loop affinity, while an async close still waits for the loop it was evicted on rather than being scheduled onto a foreign one. An async client evicted with no loop running is queued rather than discarded, and closes on the next call from a loop. Values with nothing to close are never queued at all.

Two tests cover it: test_sync_client_evicted_outside_an_event_loop_is_still_closed drives the whole cycle with no loop anywhere, and test_an_async_client_waits_for_a_loop_rather_than_being_dropped schedules and reaps from a thread with no loop, asserts the client is still open and still queued, then reaps from the loop and sees it closed.

Also folded in a lock around the pending queue, since the cache is reachable from every worker thread's loop, and rebased onto current staging, which clears a lint failure from a duplicate import my diff never touched.

[yassin-berriai]

@greptileai please review the current head 7b26028518.

One change since the 5/5: the ownership test is now guarded. schedule() asked client in self._owned directly, and WeakSet.__contains__ only swallows the TypeError from non-weak-referenceable values, not the one from hashing a weak-referenceable but unhashable referent. This cache holds arbitrary values, so that would have raised inside _remove_key and broken eviction outright rather than just skipping the value. _is_owned now returns False in that case, covered by test_an_unhashable_cached_value_does_not_break_eviction, which fails against the unguarded membership test.

Also added the live-proxy numbers to the description.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_evicted_llm_client_close (a2c4d54) with litellm_internal_staging (b1fd20f)}

[yassin-berriai]

@greptileai please review the current head a2c4d54b3c.

Three security findings from veria, all real, all fixed; veria has re-reviewed and is now at 0 open / 0-10 risk.

The queue was rebuilt on every eviction and filtered in full on every cache read. With an eviction per request that pair is quadratic: 20k evictions took 3.98s against 0.037s for the new structure. It is now a deque per deadline class, so a reap pops what is due and stops, dead entries are dropped as the next entry is queued, and the queue is bounded past which an evicted client is left to the collector.

The grace window alone was not a safe close signal. litellm.request_timeout defaults to 6000 seconds and a stream is bounded only by the upstream, so a 900-second deadline can fire under a live request. Past its deadline a client is now closed only if its own pool reports nothing in flight, and requeued otherwise. Both backends account for this already: httpcore reports a non-idle connection for the whole of a response including a stream, aiohttp holds it in _acquired over the same span. Two tests drive real requests against a local trickling server, one per backend.

Ownership was hardcoded true at the OpenAI call site. _get_async_http_client hands back litellm.aclient_session when the caller configured one, and OpenAI.close() closes whatever http client it was given, so eviction would have closed the caller's shared session. Ownership now derives from the wrapped http client; azure had the same defect at common_utils.py:594 and is fixed too.

Mutation checks: dropping the in-flight test, the bound, or the ownership derivation each fails its tests, and restoring the full-queue scan trips the reap-cost test at 501500 comparisons against 2001.

[shivamrawat1]

#35492