fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts

[devin-ai-integration]

TLDR

Problem this solves:

• With semantic caching on, every turn of an agent loop embedded the same text, so the cache replayed the turn 1 answer on every later turn
• On /v1/messages the tool_result block text was dropped from the embedded prompt, so turn 2 embedded byte-identical to turn 1
• Nothing stopped a 5, 7, 9 message agent history from being looked up or stored
• On /v1/responses a function_call_output whose output is a list of input_text blocks contributed nothing to the embedded prompt, only the string form did

How it solves it:

• cache_params.max_messages (default 4): a request with more messages, or more Responses API input items, than this is neither looked up nor stored. Cache.should_use_cache returns False, which is the shared gate for every backend, exact and semantic, Python and the Rust native cache
• get_semantic_cache_prompt_from_messages in common_utils.py is get_str_from_messages plus the text inside Messages API tool_result blocks. Redis, Valkey and Qdrant semantic caches embed that, and the Rust semantic cache mirrors it
• The Responses prompt walker in redis_semantic_cache.py recurses into a list-form output the same way it does into content, so block-form tool outputs embed like string ones. The Rust walker mirrors it
• convert_content_list_to_str and the provider-facing helpers are unchanged

A message cap was chosen over encoding tool calls into the embedded text: it is one shared check, it also stops long agent histories from filling the store, and it holds for every backend

User Flow

Before: an agent loop with semantic caching on gets the same cached answer for every tool turn

1. The agent sends POST http://localhost:4000/v1/messages with one user message asking to write hello to /tmp/alpha.txt
2. It gets a real tool_use for write_file
3. The next turn adds the tool_use and a tool_result saying PermissionError (3 messages)
4. It gets the turn 1 tool_use back from the cache, same tool id, same cost header, so the agent writes the file again
5. Every later turn (5, 7, ... messages) gets the same cached tool_use back, forever

After: only short conversations are cached, and a tool turn no longer embeds like the turn before it

1. The agent sends POST http://localhost:4000/v1/messages with one user message asking to write hello to /tmp/alpha.txt
2. It gets a real tool_use for write_file; sending the same request again is a cache hit
3. The next turn adds the tool_use and a tool_result saying PermissionError (3 messages, under the cap, so the cache is checked)
4. It gets a fresh answer that reacts to the error, because the embedded text now includes the tool result
5. Turns with 5 or more messages skip the cache both ways: each one reaches the provider and none is stored

The same cap applies to /v1/chat/completions (each tool_calls message and each role: "tool" message counts) and to /v1/responses (each input item counts). Exact caching stops caching past 4 messages too

Linear ticket

Resolves LIT-8373

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Tests: tests/integration/caching/test_cache_max_messages.py runs against the shared integration proxy (cache: redis) and its scripted provider: for each of /v1/messages, /v1/chat/completions and /v1/responses it sends turn 1 of an agent session twice (1 or 2 messages, the repeat is served from the cache) and turn 3 twice (5 or 6 messages, both reach the provider), so the provider sees exactly 3 of the 4 requests. Removing the cap fails it on the past-cap assertion. tests/unit/caching/test_caching.py covers should_use_cache at 4, 5, Responses input counting, embedding string inputs, a custom max_messages, and None disabling the cap. test_litellm_core_utils_prompt_templates_common_utils.py and tests/unit/caching/test_redis_semantic_cache.py pin the embedded text for string and block tool_result, tests/unit/caching/test_redis_semantic_cache.py also pins a block-form function_call_output, and litellm-rust/crates/cache/tests/semantic.rs pins the Rust side of both. Mutations checked: changing > to >= in should_use_cache, dropping the tool_result branch, dropping the list-form output branch, and counting input strings each fail a test

Screenshots / Proof of Fix

Setup: proxy with cache: true, type: redis-semantic on Redis Stack, similarity_threshold: 0.95, text-embedding-3-small, real Anthropic (claude-haiku-4-5), Redis flushed before each leg. Five requests to /v1/messages: turn 1 (1 message), turn 1 again, turn 2 (3 messages: tool_use plus tool_result saying PermissionError), turn 3 (5 messages: a second tool_use plus a tool_result saying the write succeeded), turn 3 again. The response id and the tool_use id show whether Anthropic answered or the cache did

U='{"role":"user","content":"Write hello to /tmp/alpha.txt, then tell me in one sentence which path you wrote."}'
tu() { echo "{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_$1\",\"name\":\"write_file\",\"input\":{\"path\":\"/tmp/alpha.txt\",\"text\":\"hello\"}}]}"; }
tr_() { echo "{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"toolu_$1\",\"content\":\"$2\"}]}"; }
ERR="PermissionError: [Errno 13] Permission denied: /tmp/alpha.txt"; OK="wrote 5 bytes to /tmp/alpha.txt"
body() { echo "{\"model\":\"claude-haiku-4-5\",\"max_tokens\":200,\"tools\":$TOOLS,\"messages\":[$1]}"; }
T1=$(body "$U"); T2=$(body "$U,$(tu 1),$(tr_ 1 "$ERR")"); T3=$(body "$U,$(tu 1),$(tr_ 1 "$ERR"),$(tu 2),$(tr_ 2 "$OK")")
for req in "$T1" "$T1" "$T2" "$T3" "$T3"; do
  echo "$req" | curl -sS -D /dev/stderr http://localhost:4000/v1/messages -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d @- \
    | python3 -c 'import json,sys; d=json.load(sys.stdin); print("id=" + d["id"], json.dumps(d["content"])[:300])'; sleep 4
done

Before (merge base d7e7f40): one Anthropic call, then the cache answers everything, including the 3 and 5 message tool turns

== turn 1 (1 message): user asks
   id=msg_011CfmrMSzVpyKjfD21umLAR [{"type": "tool_use", "id": "toolu_01V1aLidiZMoNA567X2Z7EhU", "name": "write_file", "input": {"path": "/tmp/alpha.txt", "text": "hello"}}]
   x-litellm-response-cost: 0.000968
== turn 1 again (1 message)
   id=msg_011CfmrMSzVpyKjfD21umLAR [{"type": "tool_use", "id": "toolu_01V1aLidiZMoNA567X2Z7EhU", ...}]
== turn 2 (3 messages): tool_use + tool_result PermissionError
   id=msg_011CfmrMSzVpyKjfD21umLAR [{"type": "tool_use", "id": "toolu_01V1aLidiZMoNA567X2Z7EhU", ...}]
== turn 3 (5 messages): second tool_use + tool_result ok
   id=msg_011CfmrMSzVpyKjfD21umLAR [{"type": "tool_use", "id": "toolu_01V1aLidiZMoNA567X2Z7EhU", ...}]
== turn 3 again (5 messages)
   id=msg_011CfmrMSzVpyKjfD21umLAR [{"type": "tool_use", "id": "toolu_01V1aLidiZMoNA567X2Z7EhU", ...}]

After (394abff): the repeat of turn 1 is the only cache hit; the tool turns reach Anthropic, and the 5 message turn is never stored, so its repeat reaches Anthropic too

== turn 1 (1 message): user asks
   id=msg_011CfmrDKDTZzbux2F8SM9mp [{"type": "tool_use", "id": "toolu_01WonzdfwuuDeBQh7txZLicG", "name": "write_file", "input": {"path": "/tmp/alpha.txt", "text": "hello"}}]
   x-litellm-response-cost: 0.000968
== turn 1 again (1 message)
   id=msg_011CfmrDKDTZzbux2F8SM9mp [{"type": "tool_use", "id": "toolu_01WonzdfwuuDeBQh7txZLicG", ...}]
   x-litellm-response-cost: 0.000968
== turn 2 (3 messages): tool_use + tool_result PermissionError
   id=msg_011CfmrE1NwN7u2PGufzws3U [{"type": "text", "text": "I encountered a permission error trying to write to /tmp/alpha.txt. Let me try writing to a different location that should be accessible:"}, {"type": "tool_use", "id": "toolu_01495wdq8NLGRZwFpM7EsKim", "name": "write_file", "input": {"path": "./alpha.txt", "text": "hello"}}]
   x-litellm-response-cost: 0.001211
== turn 3 (5 messages): second tool_use + tool_result ok
   id=msg_011CfmrEPEauipUfjctYKy5T [{"type": "text", "text": "I wrote \"hello\" to the file located at /tmp/alpha.txt."}]
   x-litellm-response-cost: 0.000895
== turn 3 again (5 messages)
   id=msg_011CfmrEio4j226CCLeCWEVC [{"type": "text", "text": "I wrote \"hello\" to the file at the path /tmp/alpha.txt."}]
   x-litellm-response-cost: 0.0009

Type

🐛 Bug Fix
✅ Test

Caveats

A 3 or 4 message tool turn is still under the cap, so it is still looked up. Its embedded text now differs from turn 1 by the tool result text, and in the live run above it missed at similarity_threshold: 0.95, but a very short tool output with a lower threshold could still score above it and replay the turn 1 answer once. Turns with 5 or more messages cannot

Tool call names and arguments are not part of the embedded text, as on main. Two short turns that differ only in the command they ran, with the same tool output, embed the same and can share an answer. Encoding tool calls was considered and set aside for the cap

Exact caching changes too: a plain chat with 5 or more messages that hit the exact cache before no longer does. Set cache_params.max_messages higher, or to null to turn the cap off

Existing semantic entries for /v1/messages tool turns were embedded without the tool_result text, so after upgrade those turns miss once and are re-stored. Requests without tool_result blocks embed exactly as before

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/ed7571b353934c67b185cf3b2b06ec77
Open in Devin Desktop: https://app.devin.ai/desktop/session/ed7571b353934c67b185cf3b2b06ec77?variant=devin
Requested by: @kerry-berri

---

Note

Medium Risk
Changes default caching behavior for conversations with more than four messages and alters semantic-cache keys for tool turns, which can cause one-time misses and different hit rates after upgrade.

Overview
Fixes agent loops getting stale semantic cache hits by capping which conversations are cached and by embedding tool-turn text that was previously dropped.

max_messages (default 4) on Cache turns off lookup and storage when a request has more chat messages or Responses API input items (mapping/model objects only; embedding string lists are not counted). should_use_cache enforces this before mode/opt-in, so exact and semantic backends all skip long histories. Set max_messages=None to disable the cap.

Semantic embedding text now uses get_semantic_cache_prompt_from_messages (Python) and matching Rust logic: Messages API tool_result content (string or nested text blocks) is included so a tool turn does not embed like the prior turn. Responses prompt collection also recurses into list-shaped output on items such as function_call_output, so block-form tool outputs contribute text like string output did.

Redis, Valkey, and Qdrant semantic caches call the new helper; get_str_from_messages is unchanged for non-cache paths.

^{Reviewed by Cursor Bugbot for commit 580a970. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

Stale Bugbot comment from a previous run.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adjusts semantic cache prompt extraction and adds message-count limits.

The PR should satisfy the repository's type-discipline requirement before merging; no new behavioral failure was established

Findings

1.  Image content is dropped ▶
2.  Custom tool inputs vanish ▶
3.  Legacy function calls vanish ▶

Summary

The PR caps cache use for long conversations and includes tool-result text in semantic-cache prompts

• Adds the shared message-count gate across Chat Completions, Messages, and Responses requests
• Updates Python and Rust prompt extraction and adds unit and integration coverage
• The latest change inlines the Responses walker without changing its behavior, but introduces a type-discipline violation

_{Reviews (22) · Last reviewed commit: "style(caching): suppress C901 on the Res..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_semantic_cache_tool_blocks (580a970) with main (086bcd2)}

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/caching.py	33.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litellm-stampbot]

@greptileai

[litellm-stampbot]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 580a970. Configure here.}