fix: don't close HTTP/SDK clients on LLMClientCache eviction

[ishaan-jaff]

Relevant issues

Cherry-pick of #22925 onto litellm_rc_branch.

Fixes streaming requests crashing with RuntimeError: Cannot send a request, as the client has been closed. after ~1 hour in production. Regression from commit fb72979432 which re-introduced client closing on cache eviction.

Pre-Submission checklist

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix
✅ Test

Changes

LLMClientCache._remove_key() was closing httpx/OpenAI SDK clients on eviction. If a streaming request still held a reference, it'd crash with the error above after the 1-hour TTL expired.

Fix: remove the _remove_key override entirely — evicted clients are garbage-collected naturally, shutdown cleanup via close_litellm_async_clients() is unchanged.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Error		Mar 5, 2026 8:02pm

[greptile-apps]

Greptile Summary

This PR fixes a production crash (RuntimeError: Cannot send a request, as the client has been closed.) by removing the _remove_key override in LLMClientCache that was eagerly closing HTTP/SDK clients on cache eviction. In-flight streaming requests that still hold a reference to an evicted client would crash after the 1-hour TTL expired. The fix lets evicted clients be garbage-collected normally, while explicit shutdown cleanup remains handled by close_litellm_async_clients().

Key changes:

• litellm/caching/llm_caching_handler.py: Deletes the _remove_key override (and its asyncio.Task-based async-close machinery); replaces with a detailed docstring explaining the intentional no-close policy.
• tests/test_litellm/caching/test_llm_caching_handler.py: All assertions inverted from "client must be closed after eviction" → "client must NOT be closed after eviction"; test_remove_key_no_event_loop and test_background_tasks_cleaned_up_after_completion replaced with simpler, correct tests.
• tests/test_litellm/caching/test_llm_client_cache_e2e.py: asyncio.sleep(0.15) guards added to existing tests to catch any future create_task-based regression; two new tests cover the AsyncOpenAI SDK client eviction and TTL-expiry paths.
• AGENTS.md / CLAUDE.md: Codified the no-close-on-eviction rule to prevent future regressions.

The implementation is minimal and surgical — a single method override is deleted and its behaviour (no-close) is the correct one. Both unit and e2e tests verify the fix from multiple angles, and documentation is updated.

Confidence Score: 5/5

• This PR is safe to merge — it removes a well-identified crash-causing code path and replaces it with the correct intentional no-op eviction behavior.
• The change is minimal and surgical: a single method override is deleted and its behaviour (no-close) is the correct one. Both unit and e2e tests verify the fix from multiple angles, documentation is updated, and the shutdown cleanup path is untouched. The PR passes all the supplied tests and directly addresses the reported production issue.
• No files require special attention

Sequence Diagram

sequenceDiagram
    participant Req as In-Flight Request
    participant Cache as LLMClientCache
    participant Client as HTTP Client
    participant GC as Python GC
    participant Shutdown as close_litellm_async_clients()

    Note over Cache,Client: Before fix — eviction closed the client
    Req->>Client: streaming read...
    Cache->>Cache: TTL expires / cache full
    Cache->>Client: ❌ _remove_key() → client.close()
    Req->>Client: next chunk read
    Client-->>Req: RuntimeError: client has been closed

    Note over Cache,Client: After fix — eviction only removes the cache reference
    Req->>Client: streaming read...
    Cache->>Cache: TTL expires / cache full
    Cache->>Cache: _remove_key() removes key (no close call)
    Req->>Client: next chunk read (still works ✅)
    Client-->>Req: response chunk
    Note over Client,GC: When Req drops its reference, GC collects the client
    GC->>Client: garbage collected
    Note over Shutdown: At process shutdown
    Shutdown->>Client: close() / aclose() called explicitly

Loading

_{Last reviewed commit: 335e834}