fix(redis): log a timeout streak once per interval instead of one line per cache call

[devin-ai-integration]

TLDR

Problem this solves:

• A short Redis latency blip logs one ERROR per cache call until the breaker opens
• A 15s blip under load produced 385 ERROR lines, all the same timeout
• The cost tracking callback added its own traceback per request on top

How it solves it:

• Redis timeout lines are throttled: first one logs, later ones go to DEBUG
• Every 5s (REDIS_TIMEOUT_LOG_INTERVAL) one line summarizes how many were suppressed
• Non-timeout Redis errors keep their per-call line, breaker-open refusals stay at DEBUG
• Spend counter increments treat a Redis timeout like an open breaker: invalidate and return
• The generic Cache.add_cache failure log only joins the throttle when its backend is Redis

User Flow

Before: an operator whose proxy sits behind a Redis that slows down for a few seconds sees their log volume explode

1. Their apps keep sending POST http://litellm-domain/v1/chat/completions with {"model":"haiku","messages":[{"role":"user","content":"say hi 1"}],"max_tokens":5} and every one returns 200 with a normal completion
2. Redis round trips go from sub-millisecond to 70ms for 15 seconds, so each cache call hits the 100ms socket timeout
3. The proxy log gains about 450 lines in 17 seconds: 385 at ERROR, almost all Got exception from REDIS Timeout reading from ..., plus 27 Error in tracking cost callback - Timeout reading from ... tracebacks
4. Their log-based alert on ERROR rate fires and the on-call spends time confirming nothing was actually failing

After: the same blip shows up as a handful of lines that say how big it was

1. Their apps send the same POST http://litellm-domain/v1/chat/completions requests and every one still returns 200
2. Redis slows down the same way for 15 seconds
3. The proxy log gains 9 lines: one WARNING for the first timeout, one ERROR summary ... Timeout reading from ... (202 more Redis timeouts since the previous Redis timeout line were logged at DEBUG), one WARNING that the breaker opened, and the rest are unrelated spend queue warnings. No cost tracking tracebacks
4. If Redis is actually down instead of slow, every failed cache call still logs its own ERROR line, so a real outage looks the same as before

Relevant issues

Docs row for REDIS_TIMEOUT_LOG_INTERVAL: BerriAI/litellm-docs#1436 (merged)

Affected release

Linear ticket

Resolves LIT-7520

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both runs. The proxy talks to Redis through a small TCP relay on 127.0.0.1:31521 that forwards to the real Redis on 31520 and can add a fixed per-direction delay read from a file. Postgres is a real local database. The model is Anthropic claude-haiku-4-5 behind the alias haiku, so every request costs real money

# config.yaml
model_list:
  - model_name: haiku
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: 127.0.0.1
    port: 31521
general_settings:
  master_key: sk-lit7520

# env
LITELLM_LOG=WARNING JSON_LOGS=True REDIS_SOCKET_TIMEOUT=0.1 REDIS_CONNECTION_POOL_TIMEOUT=5
REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5 REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT=60 REDIS_CIRCUIT_BREAKER_TIMEOUT_MIN_DURATION=5.0
PYTHONPATH=<checkout under test>

Both legs print which tree served the proxy and whether the new symbol exists before starting:

python -c "import litellm, litellm.caching.redis_cache as r; print('served-from', litellm.__file__, 'MARKER_present' if hasattr(r, '_RedisTimeoutLogThrottle') else 'MARKER_absent')"

The blip storm is 1200 identical requests over 40 parallel curls, with the relay delay set to 70ms from second 5 to second 20:

seq 1 1200 | xargs -P 40 -I{} sh -c "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:20520/v1/chat/completions \
  -H 'Authorization: Bearer sk-lit7520' -H 'Content-Type: application/json' \
  -d '{\"model\":\"haiku\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi {}\"}],\"max_tokens\":5}'" > codes.txt &
sleep 5; echo 0.07 > delay.txt; sleep 15; echo 0 > delay.txt; wait
sort codes.txt | uniq -c

Log lines below are the JSON log rendered as time level source message

Before (b27d2cc)

Redis latency blip during 1200 requests

1. bash start_proxy.sh /home/ubuntu/repos/litellm before.log, output served-from /home/ubuntu/repos/litellm/litellm/__init__.py MARKER_absent
2. Run the storm above. Output: 1200 200
3. wc -l before.log grew by 449 lines between the first request and the last one (17 seconds). By level and message:

    299 ERROR   redis_cache.py  LiteLLM Redis Caching: async set() - Got exception from REDIS Timeout reading from 127.0.0.1:31521, Writing value=...
     54 ERROR   redis_cache.py  LiteLLM Redis Caching: async increment_pipeline() - Got exception from REDIS Timeout reading from 127.0.0.1:31521
     27 ERROR   spend_log_error_logger.py  Error in tracking cost callback - Timeout reading from 127.0.0.1:31521   (each with a full traceback)
      5 ERROR   redis_cache.py  Error occurred in async batch get cache - Timeout reading from 127.0.0.1:31521
     28 WARNING redis_cache.py  TTL preservation failed, falling back to regular pipeline: Timeout reading from 127.0.0.1:31521
     27 WARNING redis_cache.py  Redis async_increment_cache_pipeline failed, falling back to in-memory result: Timeout reading from 127.0.0.1:31521
      1 WARNING redis_cache.py  Redis circuit breaker OPENED after 1412 consecutive failures (0 hard connectivity) — fast-failing Redis calls for 60s
      8 WARNING spend_update_queue.py / daily_spend_update_queue.py  Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   

   385 ERROR lines for one 15 second blip, every one of them the same timeout

Redis hard down (non-timeout failure)

1. Same proxy, docker stop litfix-redis-7520, then three sequential curls of the same body with say hi 1, say hi 2, say hi 3. Output: 200 200 200
2. New log lines:

   01:28:59 WARNING config_sync_pubsub.py:256 config sync subscriber redis error: Connection closed by server.; reconnecting in 5s
   01:28:59 WARNING auth_cache_invalidation_pubsub.py:175 auth cache invalidation subscriber redis error: Connection closed by server.; reconnecting in 5s
   01:29:01 ERROR redis_cache.py:1577 Error occurred in async batch get cache - Connection closed by server.
   01:29:02 ERROR redis_cache.py:1047 LiteLLM Redis Caching: async set() - Got exception from REDIS Connection closed by server., Writing value={'timestamp': ...
   01:29:02 WARNING redis_cache.py:250 Redis circuit breaker OPENED after 5 consecutive failures (5 hard connectivity) — fast-failing Redis calls for 60s
   01:29:04 WARNING config_sync_pubsub.py:256 config sync subscriber redis error: Connection closed by server.; reconnecting in 10s
   01:29:04 WARNING auth_cache_invalidation_pubsub.py:175 auth cache invalidation subscriber redis error: Connection closed by server.; reconnecting in 10s
   

After (bea22df, head a4e34d6 only rewraps one test line on top of it and leaves every litellm/ file byte-identical)

Redis latency blip during 1200 requests

1. bash start_proxy.sh /home/ubuntu/repos/litellm_redis_timeout_log_throttle after_v7.log, output served-from /home/ubuntu/repos/litellm_redis_timeout_log_throttle/litellm/__init__.py MARKER_present
2. Control request, curl ... -d '{"model":"haiku","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}' returns HTTP 200 with "content":"Hi! 👋"
3. Run the identical storm. Output: 1200 200
4. wc -l after_v7.log grew from 17 to 26 lines over the same window. All 9 new lines:

   20:00:55 WARNING redis_cache.py TTL preservation failed, falling back to regular pipeline: Timeout reading from 127.0.0.1:31521
   20:00:58 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   20:01:00 ERROR redis_cache.py LiteLLM Redis Caching: async set() - Got exception from REDIS: Timeout reading from 127.0.0.1:31521 (202 more Redis timeouts since the previous Redis timeout line were logged at DEBUG)
   20:01:03 WARNING redis_cache.py:252 Redis circuit breaker OPENED after 1515 consecutive failures (0 hard connectivity) — fast-failing Redis calls for 60s
   20:01:04 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   20:01:06 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   20:01:14 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   20:01:15 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   20:01:16 WARNING spend_update_queue.py:47 Spend update queue is full. Aggregating all entries in queue to concatenate entries.
   

   Two Redis timeout lines (a single timeout under load 3 seconds before the blip, then the 5 second summary with 202 suppressed) instead of 385 ERROR lines, zero Error in tracking cost callback lines (grep -c returns 0), and nothing new after the breaker opened even though requests kept flowing at 200

Redis hard down (non-timeout failure)

1. Same proxy, docker stop litfix-redis-7520, then the same three sequential curls. Output: 200 200 200
2. New log lines, one ERROR per failed cache call exactly as before:

   20:02:30 WARNING config_sync_pubsub.py:256 config sync subscriber redis error: Connection closed by server.; reconnecting in 5s
   20:02:30 WARNING auth_cache_invalidation_pubsub.py:175 auth cache invalidation subscriber redis error: Connection closed by server.; reconnecting in 5s
   20:02:32 ERROR redis_cache.py Error occurred in async batch get cache: Connection closed by server.
   20:02:33 ERROR redis_cache.py LiteLLM Redis Caching: async set() - Got exception from REDIS: Connection closed by server.
   20:02:33 ERROR redis_cache.py Error occurred in async batch get cache: Connection closed by server.
   20:02:33 WARNING redis_cache.py:252 Redis circuit breaker OPENED after 5 consecutive failures (5 hard connectivity) — fast-failing Redis calls for 60s
   20:02:35 WARNING config_sync_pubsub.py:256 config sync subscriber redis error: Connection closed by server.; reconnecting in 10s
   20:02:35 WARNING auth_cache_invalidation_pubsub.py:175 auth cache invalidation subscriber redis error: Connection closed by server.; reconnecting in 10s
   

Type

🐛 Bug Fix

Caveats (if any)

Medium

• The throttle is process-wide, one window shared by every Redis cache instance and message
  • A proxy with two Redis caches timing out at once gets one summary line, not two
• The async set() message no longer includes the value being written (it was the bulk of every ERROR line)

Low

• A Redis timeout in the spend counter increment no longer reaches the cost callback's error path or its alert. The counters are invalidated and reloaded from the database on the next read, same as an open breaker
• _is_redis_timeout_failure is now public as is_redis_timeout_failure so the proxy can import it without a private-usage type error
• Cache.add_cache wraps whichever backend it was built with. It checks isinstance(self.cache, RedisCache) before joining the throttle, so a timeout from a non-Redis backend (S3, disk, Qdrant) keeps its per-call ERROR line. The dynamic_cache_object argument to async_add_cache is not inspected; a Redis object passed that way while self.cache is something else logs per call, which is the pre-PR behavior
• Before was captured at b27d2cce77, the branch's original base, not at the current merge base. git diff b27d2cce77 origin/main -- litellm/caching/redis_cache.py shows main kept every per-call verbose_logger.error this PR replaces and added one more (async_set_cache_pipeline_with_ttls), so the current base can only log more lines per blip, not fewer
• CodeQL (not a required check) flags 3 py/cyclic-import alerts on the two import lines this PR touches. Both lines already exist on main with the same cycle: proxy_server.py:253 already imports RedisCircuitBreakerOpenError from redis_cache (one of the 3 alerts is about that pre-existing name) and redis_cache.py already imports 5 constants from litellm.constants in the same block. The PR only adds a name to each existing statement. python -c "import litellm.constants, litellm.caching.redis_cache, litellm.proxy.proxy_server" succeeds in each of the 3 import orders
• Case 1 exercised async batch get, async_increment_pipeline() and, on earlier runs of the same storm, async set(), the rate limiter TTL pipeline and the DualCache increment fallback. Sync get_cache(), the batch-get sync path and the remaining write and list paths (async_set_cache_pipeline, async_set_cache_pipeline_with_ttls, async_set_cache_sadd, async_increment, async_rpush, async_lpop and their pipeline variants) are covered by unit tests, not the live run
  • The storm reuses the master key, so the auth prefetch write async_set_cache_pipeline_with_ttls never fires live. Its timeout path is the async_set_cache_pipeline_with_ttls case of test_write_path_timeouts_inside_the_interval_stay_at_debug, which fails on the previous head

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Changes are mostly logging volume and graceful handling of transient Redis timeouts; spend counters are invalidated on timeout like an open breaker, which could briefly skew in-memory spend until reload.

Overview
Redis cache failures now rate-limit timeout noise instead of emitting one ERROR per call during a slow-Redis blip.

log_redis_failure is the shared path: non-timeout Redis errors still log at the caller’s level every time; timeouts emit one line per REDIS_TIMEOUT_LOG_INTERVAL (default 5s, REDIS_TIMEOUT_LOG_INTERVAL), with repeats at DEBUG and a periodic summary of how many were suppressed. Circuit-breaker-open refusals stay DEBUG. Most RedisCache exception handlers that used raw verbose_logger.error now call this helper (messages no longer embed cache values on some paths).

Cache.add_cache / async variants use _log_add_cache_failure, which only joins the Redis throttle when self.cache is a RedisCache; other backends keep per-call ERROR logging.

is_redis_timeout_failure is public (was _is_redis_timeout_failure) for proxy spend logic: _apply_spend_counter_increments now treats Redis timeouts like an open circuit breaker—invalidate counters and return without raising—reducing cost-callback tracebacks during transient slowness.

Tests cover throttle behavior on Redis ops, add_cache, DualCache fallbacks, and spend-counter timeouts.

^{Reviewed by Cursor Bugbot for commit a4e34d6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/eba56c82452f40a2943f80e119789fb7
Open in Devin Desktop: https://app.devin.ai/desktop/session/eba56c82452f40a2943f80e119789fb7?variant=devin
Requested by: @yassin-berriai

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_redis_timeout_log_throttle (a4e34d6) with main (34fe9f7)}

[greptile-apps]

Greptile Summary

This PR reduces Redis timeout log storms while retaining per-call error visibility for non-timeout Redis failures and non-Redis cache backends.

• Adds a process-wide, thread-safe timeout throttle driven by a monotonic clock and configurable interval.
• Routes Redis cache operation failures through centralized logging.
• Treats spend-counter Redis timeouts like an open circuit breaker after invalidating in-memory counters.
• Keeps generic cache failures at ERROR unless the configured backend is Redis.
• Adds coverage for timeout summaries, write paths, fallback behavior, spend counters, and backend-specific logging.

Confidence Score: 5/5

The PR appears safe to merge; the prior throttling and style findings are resolved, and no new actionable failures remain.

The timeout throttle now uses a monotonic clock, authentication-prefetch pipeline failures use the centralized throttle, generic non-Redis cache failures remain at ERROR, and the previously flagged test constructor is wrapped. devin-ai-integration[bot] accepted the process-wide shared-throttle behavior because it intentionally matches the process-wide circuit breaker and documented the tradeoff. The final line-length thread was manually resolved without explanation, and the current line is compliant.

Important Files Changed

Filename	Overview
litellm/caching/redis_cache.py	Centralizes Redis failure logging and throttles repeated timeout messages while preserving ordinary error handling.
litellm/caching/caching.py	Restricts generic add-cache timeout throttling to configured Redis backends.
litellm/proxy/proxy_server.py	Handles spend-counter Redis timeouts like breaker-open failures after counter invalidation.
litellm/constants.py	Adds the configurable Redis timeout logging interval.
tests/test_litellm/caching/test_redis_cache.py	Covers throttling intervals, summaries, non-timeout errors, and Redis write/list operations.
tests/test_litellm/caching/test_caching.py	Verifies that only Redis-backed generic cache failures join the timeout throttle.
tests/test_litellm/caching/test_dual_cache.py	Verifies timeout throttling when Redis operations fall back to memory and wraps the previously flagged long line.
tests/test_litellm/proxy/proxy_server/test_spend_counters.py	Verifies timeout invalidation and propagation behavior for spend-counter updates.

_{Reviews (6): Last reviewed commit: "test(caching): wrap the DualCache fixtur..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 92.53731% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/redis_cache.py	94.23%	3 Missing ⚠️
litellm/caching/caching.py	77.77%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai please re-review f681a97: monotonic clock and shorter test docstrings

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai please re-review 28f2d1f: adds unit tests for the async write and list timeout paths, no production code change

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai please re-review 53c8a82: merge of main, conflict resolution keeps is_redis_timeout_failure and the spend counter timeout return

[mateo-berri]

bugbot run

[devin-ai-integration]

Same static cycle main already has for the neighbouring imports on these lines; importing either module first works at runtime, so leaving as is

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai please re-review c4354c2, which routes async_set_cache_pipeline_with_ttls through log_redis_failure

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@veria-ai please review c4354c2 for the throttled Redis timeout logging and spend counter timeout handling

[devin-ai-integration]

@greptileai please re-review a4e34d6: wraps the flagged test line, generic add_cache stays at ERROR for non-Redis backends

[mateo-berri]

bugbot run

[devin-ai-integration]

@veria-ai please review a4e34d6 for the throttled Redis timeout logging and the generic add_cache backend check

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a4e34d6. Configure here.}