fix(responses): preserve prompt cache reuse in chat bridge

[sorgfresser]

TLDR

Problem this solves:

• Moving breakpoints stop reusing the previous tool prefix
• Growing agent histories keep caching only the static prefix

How it solves it:

• Preserve cache breakpoints through the Chat-to-Responses bridge
• Default gateway-injected breakpoints to implicit lookup
• Preserve caller-selected cache modes and TTLs

User Flow

Before: an agent with growing tool history keeps rewriting previously cached input

1. The developer sends POST https://litellm-domain/v1/chat/completions with tools, medium reasoning, and automatic cache injection
2. The first 2,283-token turn writes 2,280 cache tokens
3. The 2,838-token turn reads only 1,725 cached tokens and rewrites 1,110
4. The 3,393-token turn still reads only 1,725 cached tokens and rewrites 1,665

After: each turn reuses the previous eligible tool boundary and writes only its new suffix

1. The developer sends the same growing tool conversation
2. The first turn still writes 2,280 cache tokens
3. The second turn reads 2,280 cached tokens and writes 555
4. The third turn reads 2,835 cached tokens and writes 555

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5

Screenshots / Proof of Fix

The same three growing tool turns were captured from the exact production image and replayed through the real upstream provider. The six bounded calls cost $0.1265075 total

Before (cdcbf2f)

1. Captured requests used prompt_cache_options.mode: explicit
2. Cached tokens by turn were 0, 1725, and 1725
3. Cache writes by turn were 2280, 1110, and 1665

After (b1084f0, patch-equivalent live run at 6d84883)

1. Captured requests used prompt_cache_options.mode: implicit while retaining explicit breakpoints
2. Cached tokens by turn were 0, 2280, and 2835
3. Cache writes by turn were 2280, 555, and 555
4. The rebase changed test paths and commit identity without changing the production patch

Test Plan

• 20 passed, 735 deselected across the focused cache and Responses tests on the rebased tip
• All required CI checks pass on the rebased tip
• Restoring the old explicit default makes the new cache-mode assertion fail
• Caller-provided explicit and implicit modes, including TTL, remain unchanged

Type

Bug Fix
Test

Final Attestation

• The tests check the right things, including caller overrides and growing tool histories

---

Note

Medium Risk
Changes upstream prompt-cache semantics (implicit vs explicit) and request payload shape for cached agent/tool conversations; incorrect behavior would affect cache hit rates and cost but is limited to OpenAI prompt-cache breakpoint flows with tests covering overrides.

Overview
Fixes prompt-cache reuse when chat completions are bridged to the Responses API and when LiteLLM auto-injects cache breakpoints for OpenAI GPT ≥5.6.

The chat→Responses converter now copies prompt_cache_breakpoint onto converted text, image_url, and file content blocks via with_prompt_cache_breakpoint, so injected marks are not dropped mid-bridge. On the OpenAI-dialect chat path, when the cache hook newly places breakpoints it defaults prompt_cache_options.mode to implicit instead of explicit, while per-block breakpoints stay explicit and caller-supplied prompt_cache_options (including TTL) are unchanged.

Tests assert breakpoint survival through transform_request and update expectations for the implicit default on chat/completions and Responses request bodies.

^{Reviewed by Cursor Bugbot for commit b1084f0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

All committers have signed the CLA.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing sorgfresser:litellm_preserve_responses_prompt_cache_breakpoint (b1084f0) with main (22b36cb)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Changes prompt cache mode from explicit to implicit in Anthropic integration.

The PR appears safe to merge; no outstanding finding or new actionable issue was established.

Summary

The PR preserves prompt-cache breakpoints when Chat content is converted to Responses input and changes the gateway-injected chat-path cache mode to implicit while retaining caller-provided options.

• Adds conversion coverage for text, image URL, and file blocks.
• Updates cache-mode assertions and tests caller-selected modes and TTLs.

_{Reviews (3) · Last reviewed commit: "test(cache): expect implicit chat lookup"}

[sorgfresser]

@greptileai Please review the updated cache-mode fix and confirm multimodal breakpoint preservation remains complete across the Chat-to-Responses bridge.

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b1084f0. Configure here.}

[ryan-crabbe-berri]

LGTM. Verified live against OpenAI that implicit mode keeps the explicit marks and reads back the previous turn's prefix. Docs follow-up incoming for making the implicitl default change and then also documenting what we do by default in the other scenarios here