fix(caching): write the response-cache SET to Redis at once instead of on the post-call batch

[devin-ai-integration]

TLDR

Problem this solves:

• An identical /v1/messages request sent right after the first misses the cache
• Response-cache SET waited on the post-call Redis batch since perf(proxy): one post-call Redis pipeline per backend for spend, rate-limit, routing and response-cache writes #43779
• That batch only goes out after every success callback finishes, or at 1s
• Fails test_cache_hit_post_call_reject_keeps_deployment_labels_anthropic in CircleCI integration-extensions on main

How it solves it:

• Response-cache writes go straight to Redis again, as before perf(proxy): one post-call Redis pipeline per backend for spend, rate-limit, routing and response-cache writes #43779
• Spend counters, rate limits, deployment TPM and slot releases still ride the post-call batch

User Flow

Before: a developer behind a proxy with a Redis cache and a slow success callback pays for the same answer twice

1. They send POST https://litellm-domain/v1/messages with a prompt and get msg_A back
2. Right after, they send the exact same POST https://litellm-domain/v1/messages
3. A fresh provider call comes back with a new id msg_B, billed again

After: the second request is served from the cache

1. They send POST https://litellm-domain/v1/messages with a prompt and get msg_A back
2. Right after, they send the exact same POST https://litellm-domain/v1/messages
3. The same msg_A comes back from the cache, no second provider call

Affected release

Regression in v1.105.0-dev.1 (introduced by #43779)

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Live proxy, real Redis at 127.0.0.1:6379, real Anthropic and OpenAI calls, and a success callback that sleeps 0.5s (the shape of a slow logging integration)

model_list:
  - model_name: claude-haiku-4-5
    litellm_params: {model: anthropic/claude-haiku-4-5, api_key: os.environ/ANTHROPIC_API_KEY}
  - model_name: gpt-5.4-mini
    litellm_params: {model: openai/gpt-5.4-mini, api_key: os.environ/OPENAI_API_KEY}
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
  callbacks: slow_callback.slow_logger   # async_log_success_event: await asyncio.sleep(0.5)
  cache: true
  cache_params: {type: redis, host: 127.0.0.1, port: 6379}

Each run sends two back-to-back identical requests with a fresh random marker in the prompt, so every run starts with a cold key. A cache hit returns the first call's provider id, a miss returns a new one

curl -s localhost:4000/v1/messages -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' \
  -d '{"model":"claude-haiku-4-5","max_tokens":16,"messages":[{"role":"user","content":"say hi <uuid>"}]}'
curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' \
  -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"say hi <uuid>"}]}'

Before (424bfd8, merge base)

/v1/messages

1. Run 1: call 1 msg_011Cfaknn3vifvxKf2aDK6k9, call 2 msg_011CfaknqY2AbmJFpM1ec3P6, a second provider call
2. Run 2: call 1 msg_011CfakntqTc9jb75mu3EmmQ, call 2 msg_011CfaknwFKLkKv5ATVtdxxP, a second provider call
3. Run 3: call 1 msg_011CfaknypbuZtmC7MoFrCFv, call 2 msg_011Cfako2Ndvbt4UJLJg1KAG, a second provider call

/v1/chat/completions

1. Run 1: both calls chatcmpl-EU19GJrS2fJXeeQKFLrIyiG5SgMwA, call 2 carries x-litellm-cache-key, cache hit
2. Run 2: both calls chatcmpl-EU19HGgJQKro90iE8tcmyExQDnHkM, cache hit
3. Run 3: both calls chatcmpl-EU19ILT3hqgtPB4f4n7fS5gQOc6S9, cache hit

After (e791607)

/v1/messages

1. Run 1: both calls msg_011CfamqVedscTbbbtPjbQ1f, cache hit
2. Run 2: both calls msg_011CfamqYUnzFmpgVQwajbR3, cache hit
3. Run 3: both calls msg_011CfamqbLwaRxu5YBXZ2mhb, cache hit

/v1/chat/completions

1. Run 1: both calls chatcmpl-EU1MVgHhyCzsckKQWHSm5GGSA62Wh, call 2 carries x-litellm-cache-key, cache hit
2. Run 2: both calls chatcmpl-EU1MVeK8gFd0x4E2xesvuqP6458RC, cache hit
3. Run 3: both calls chatcmpl-EU1MW49WX0hjDM8UjvleUvG3SqoAR, cache hit

Integration suite

1. tests/integration/observability/test_cache_hit_guardrail_metrics.py against a local proxy, Postgres, Redis and the scripted upstream
2. Merge base: test_cache_hit_post_call_reject_keeps_deployment_labels_anthropic fails with assert 2 == 1 provider requests
3. This branch: 18 passed, 1 skipped

Unit tests and mutation check

1. tests/unit/caching: 560 passed, 1 skipped
2. With the merge-base caching.py and dual_cache.py copied back in, both new tests in test_request_redis_batch_post_call.py fail (response not in Redis before the post-call flush)

Type

🐛 Bug Fix

Caveats (if any)

Low

• Chat completions did not reproduce live before or after, only /v1/messages missed
• /v1/responses covered by the integration suite only, it passed on both sides
• Unrelated _slot_stash line in the test file got reflowed by ruff format

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/ca842b11eda64990a199ffbd982d8826
Open in Devin Desktop: https://app.devin.ai/desktop/session/ca842b11eda64990a199ffbd982d8826?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Changes when response cache writes hit Redis.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR restores direct Redis writes for response-cache entries while leaving spend, rate-limit, and deployment updates in the post-call batch. It updates the unit tests to check that response entries are available before that batch flushes.

_{Reviews (1) · Last reviewed commit: "fix(caching): re-export DualCache explic..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_response_cache_set_direct (e791607) with main (ae60fd1)}