fix(redis): reset only the failed node on a cluster client timeout, not the whole client

[yassin-berriai]

TLDR

Problem this solves:

• One Redis Cluster node's transient client-side timeout tears down every node's connections
• Every other concurrent request then blocks behind one shared reinitialization lock

How it solves it:

• Reset only the node that actually timed out, mirroring how a plain (non-cluster) Redis client already behaves

User Flow

Before: a customer's proxy pods share one Redis Cluster connection; when one node's response is briefly slow under load, every concurrent request stalls for seconds, not just the ones talking to that node

1. The customer sends many concurrent requests to POST https://proxy-domain/v1/chat/completions while one Redis Cluster node is briefly slow to respond (e.g. under host contention)
2. Requests that never touch the slow node also stall for multiple seconds before completing, even though those other nodes answer in under a millisecond
3. Dashboards show the proxy's Redis and auth latency metrics spike into the seconds-to-many-seconds range across the whole pod, not just for the affected node's own traffic

After: only requests actually waiting on the transiently slow node are affected; every other request keeps its normal latency

1. The customer sends the same concurrent requests while the same node is briefly slow
2. Requests that never touch the slow node keep completing in under a millisecond, unaffected
3. The proxy's Redis and auth latency metrics stay flat outside of the handful of requests actually routed to the slow node

Relevant issues

Linear ticket

Resolves LIT-5836

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is a Redis Cluster client-internals fix with no LLM call in its path, so there is no provider spend to demonstrate; the proof instead drives the exact redis.asyncio cluster client construction function the proxy itself calls (litellm._redis.get_redis_async_client), against a real, unmocked local 3-master/3-replica Redis Cluster (redis-server --cluster-enabled yes, no mocks). One master is paused with the real Redis CLIENT PAUSE command to force a genuine client-side socket timeout while the server itself stays healthy, then the run measures whether concurrent traffic against the other two, untouched masters is affected.

Setup (shared by both runs): a local 3-master/3-replica cluster on 127.0.0.1:27000-27005, socket_timeout=1.0s matching the customer's config, one task looping commands against the node about to be paused (27000) and 40 tasks looping commands against the two untouched masters (27001/27002).

Before (9821b45)

1. python3 repro_storm.py builds the cluster client via get_redis_async_client, pauses node 27000 for 3000ms with CLIENT PAUSE, then drives all 41 tasks concurrently for 4.5s
2. Output: client type: <class 'redis.asyncio.cluster.RedisCluster'> (the unpatched upstream client)
3. Output: RedisCluster.aclose() was invoked 13 time(s) during the run (each tears down every node's connections)
4. Output: 40 tasks hitting HEALTHY, unpaused nodes (27001/27002): ... healthy-node calls >1s: 40/65303 and max single-call latency observed on a healthy-node task: 2.081s — every one of the 40 tasks against the untouched nodes was hit with a multi-second stall caused entirely by node 27000's pause

After (368abcd)

1. Same command, same cluster, same pause on node 27000
2. Output: client type: <class 'litellm.caching.redis_cluster_node_isolation.get_litellm_async_redis_cluster_class.<locals>.LiteLLMAsyncRedisCluster'>
3. Output: RedisCluster.aclose() was invoked 0 time(s) during the run (each tears down every node's connections)
4. Output: 40 tasks hitting HEALTHY, unpaused nodes (27001/27002): ... healthy-node calls >1s: 0/116359 and max single-call latency observed on a healthy-node task: 0.007s — zero collateral impact on the untouched nodes; the paused node's own task still correctly waits out the pause and succeeds

Type

🐛 Bug Fix

Caveats (if any)

• Requires overriding a private redis-py method (RedisCluster._execute_command); a version guard logs a warning if the installed redis-py version falls outside the set this override was verified against
• This PR covers the generic Redis Cluster teardown-storm mechanism only; LIT-5944 (project-scoped virtual-key auth latency) tracks whether a project-specific hot path remains once this lands, as a separate follow-up
• Does not add a full live-proxy end-to-end load test; the proof drives the exact client-construction function the proxy uses, against a real (unmocked) Redis Cluster

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR changes async Redis Cluster timeout handling so only the failed node is disconnected while topology-related errors retain full-cluster reinitialization

• Routes async cluster construction through the node-isolating subclass
• Adds focused coverage for timeout, connection, success, cluster-down, and MOVED behavior
• Updates Redis client-construction tests for the new class factory

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/_redis.py	Async Redis Cluster construction now uses the node-isolating subclass factory
litellm/caching/redis_cluster_node_isolation.py	Adds the Redis Cluster override that limits connection-error cleanup to the failed node
tests/test_litellm/caching/test_redis_cluster_node_isolation.py	Adds typed regression coverage for node-local resets and unchanged cluster-level error behavior
tests/test_litellm/test_redis.py	Updates cluster-construction mocks to patch the new subclass factory

_{Reviews (3): Last reviewed commit: "fix(redis): reset only the failed node o..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 81.69014% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/redis_cluster_node_isolation.py	80.88%	13 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head 042d7d5 — addressed the typing feedback on the test scaffolding.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit5836_redis_cluster_teardown_storm (368abcd) with litellm_internal_staging (91f2382)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (0c50286) during the generation of this report, so 91f2382 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai please review the current head 368abcd — rebased onto litellm_internal_staging (picks up the #37864 lint fix) and squashed into a single commit.