fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge by devin-ai-integration[bot] · Pull Request #44119 · BerriAI/litellm

devin-ai-integration · 2026-10-02T01:28:39Z

TLDR

Problem this solves:

Tool-calling chats on gpt-5.6 go through the Responses bridge
The bridge drops every prompt_cache_breakpoint marker on text, image and file blocks
So those requests get no prompt-cache reads at all, on every turn

How it solves it:

Keep markers on converted text, image and file blocks (absorbed from fix(responses): preserve prompt cache reuse in chat bridge #42281)
Only the bridge keeps them. Other converter callers still drop them
Drop markers when neither the model nor its base_model supports breakpoints
Leave the injected default prompt_cache_options: {"mode": "explicit"} unchanged

This is a copy of #42281 without its default flip from explicit to implicit. That flip also changed plain /v1/chat/completions and direct /v1/responses, left /v1/messages on explicit, and contradicted the documented default, so it needs its own decision

User Flow

Before: an agent app running a tool loop on gpt-5.6 pays full price for its long system prompt on every turn

The app sends POST https://litellm-domain/v1/chat/completions with tools, a 3k-token system prompt and the growing tool history
Every turn comes back with usage.prompt_tokens_details.cached_tokens: 0
A caller who marks blocks with prompt_cache_breakpoint gets the same 0, since the markers never reach OpenAI

After: the same app reads the system prompt from cache on every turn after the first

The app sends the same POST https://litellm-domain/v1/chat/completions with tools and the growing history
Turn 1 writes about 3020 tokens to cache, and every later turn reads about 2995 cached tokens
A caller marking blocks on a model without breakpoint support (gpt-5.4-mini, an Azure gpt-5.4-nano deployment) still gets 200, because the markers are dropped before the request leaves the proxy

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks
- Red jobs at the head also fail on main or have no link to this diff, details in the audit
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Two proxies with 2 workers each, real Postgres, real OpenAI, Azure OpenAI and OpenRouter, no mocks. The base runs the merge base, the head runs the PR tip, same config. Every prompt starts with a per-leg random nonce so the legs never share the provider cache. What the provider received was read from the POST Request Sent from LiteLLM block of the proxy log

Before (`aa601ce`)

Tool loop on /v1/chat/completions (bridged to /v1/responses)

Four non-streaming turns on gpt-5.6 with tools and a growing history: cached tokens 0, 0, 0, 0 and cache writes 0, 0, 0, 0
Three streaming turns: cached 0, 0, 0
Wire: prompt_cache_options: {'mode': 'explicit'} sent with no prompt_cache_breakpoint anywhere in input

Image block with a caller marker on the bridge

Same image request twice on gpt-5.6: second request cached 0, write 0

Caller markers on models without breakpoint support

gpt-5.4-mini, Azure gpt-5.4-nano (azure/ and the /openai/v1 api_base): bridge returns 200, markers silently dropped
Native chat for the same models returns 400 prompt_cache_breakpoint is not supported on this model

After (`ccf03ee`)

Tool loop on /v1/chat/completions (bridged to /v1/responses)

Four non-streaming turns: cached 0, 2996, 2996, 2996 with writes 3022, 838, 1650, 2462
Three streaming turns: cached 2996, 2996, 2996
Wire: prompt_cache_options: {'mode': 'explicit'} plus prompt_cache_breakpoint on the marked input_text blocks

Image block with a caller marker on the bridge

Same image request twice: first writes 2135, second reads 2135 cached tokens. Wire shows the marker on input_image

Caller markers on models without breakpoint support

gpt-5.4-mini, Azure gpt-5.4-nano (azure/ and the /openai/v1 api_base): bridge still returns 200, wire has no prompt_cache_breakpoint
Native chat for the same models still returns the same 400, unchanged

Unchanged paths

Single-turn chat with a long system prompt: cached 2962 on warm requests, same as base, still mode: explicit
Caller-sent prompt_cache_options: {"mode": "explicit", "ttl": "30m"} goes out unchanged on both legs
Direct /v1/responses on gpt-5.6: same mode: explicit and markers on both legs
/v1/messages on gpt-5.6 uses its own Messages to Responses adapter, not this bridge. Deployment injection points send mode: explicit and two markers on both legs, client-only cache_control sends neither on both legs (LIT-9162)
OpenRouter on a custom api_base: 200 on both legs, head now forwards the caller markers and OpenRouter accepts them

Checkpoint cells (same two legs)

Caller marker as int 1, [1], "" or a 5120-character string on gpt-5.6: base bridge 200 with the marker dropped, head bridge forwards it and OpenAI returns 400, the same 400 native chat returns on both legs. A valid marker on two blocks reaches the wire twice on head
No Authorization header: 401 on both legs before any provider call
Azure gpt-5.4-nano streaming bridge: one attempt per leg, 200 on both, wire has no marker
Azure AI Foundry alias foundry-alias with base_model: gpt-5.6: head forwards the marker, base drops it. Alias with an unsupported base_model: no marker on both legs
Complexity router with a gpt-5.4-mini classifier on the native classifier path: classifier wire has no marker on both legs, final gpt-5.6 request keeps the caller marker on head and drops it on base, 200 on both
/v1/responses with a content-rewriting guardrail: text masked and explicit marker kept on both legs
Deprecated Noma v1 against a passive listener: no marker in the scan payload on both legs. Noma's own verdict was not tested, the vault has no Noma key
Dedicated Postgres stopped mid burst (30 mixed requests per leg): all 60 requests 200, head bridge requests carry markers, each leg loses 2 different spend rows with no duplicates, the pre-existing loss tracked in LIT-9050

Local providers suite

tests/integration/run.py providers with the CircleCI rig recipe, seeds 4106601: head 315 passed, 4 failed. Base 308 passed, 11 failed. The 4 head failures fail on base with the same owned-Redis readiness assertion

The four CircleCI reds with no main reproduction (unit print_verbose, retained OpenAI clients, Noma v2 timeout, Javelin timeout) and the TPM routing test each pass locally on both legs

Deterministic integration audit

Scripted upstream, real proxy with 2 workers per leg, real Postgres and Redis, no litellm component mocked. Same command on both legs: uv run --no-sync python tests/integration/run.py providers --seed 4106601 --order-seed 4242 over tests/integration/providers/test_responses_bridge_prompt_cache_breakpoint.py and tests/integration/providers/test_responses_bridge_prompt_cache_breakpoint_chaos.py

Base aa601ce (run 03:15:07Z): 7 failed, 5 passed. Every failure is the missing marker at the upstream
Head ccf03ee run 1 (03:16:00Z): 12 passed, 0 skipped
Head ccf03ee run 2 (03:16:53Z): 12 passed, 0 skipped, identical collected and passed node lists and order, no reruns
CircleCI pipeline 90753 at ccf03ee: all 12 audit nodes passed in integration-providers (job 2235383). The job is red on one unrelated test, test_concurrent_team_model_tpm_requests_reserve_tokens_before_reaching_the_provider, which also fails on main

Cell	Test node	Base	Head run 1, run 2
caller markers on text, image, file, non-stream, openai sync / async / httpx	`test_caller_prompt_cache_breakpoints_survive_chat_to_responses_bridge[False-openai_sync]`, `[False-openai_async]`, `[False-httpx]`	FAIL	PASS, PASS
same, streaming	`test_caller_prompt_cache_breakpoints_survive_chat_to_responses_bridge[True-openai_sync]`, `[True-openai_async]`, `[True-httpx]`	FAIL	PASS, PASS
deployment `cache_control_injection_points`, no options	`test_deployment_cache_control_injection_without_options_is_unchanged`	PASS	PASS, PASS
deployment `prompt_cache_options` implicit ttl 30m	`test_deployment_prompt_cache_options_override_is_unchanged`	PASS	PASS, PASS
unmarked bridge plus direct `/v1/responses` marker	`test_unmarked_bridge_and_direct_responses_marker_are_forwarded_unchanged`	PASS	PASS, PASS
unsupported model strips markers, non-stream and stream	`test_unsupported_model_drops_breakpoints_without_rejecting_the_request[False]`, `[True]`	PASS	PASS, PASS
chaos: 30 request mixed burst, worker SIGKILL, upstream outage and recovery, one spend row per id	`test_responses_bridge_prompt_cache_breakpoint_chaos.py::test_worker_and_peer_outages_preserve_markers_and_recover`	FAIL	PASS, PASS

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Medium

Bridge now sends caller prompt_cache_breakpoint to OpenAI-compatible backends whose model name passes the support check
- OpenRouter accepts it. Other custom api_base backends were not run
Azure deployments named with an alias need base_model set, or markers are dropped

Low

Native /v1/chat/completions still returns 400 for caller markers on unsupported models, unchanged from base (LIT-9161)
Malformed caller marker values now get OpenAI's 400 on the bridge, matching native chat, where base returned 200 by dropping them
Other callers of the shared converter keep dropping markers, as on base
- Deprecated Noma guardrail, complexity router, /v1/responses guardrail rewrite
- Live cells above, no integration cell
Live only, no integration cell: base_model alias, malformed marker values

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and !audit at ccf03ee: results above

REVIEWER MUST KNOW BEFORE APPROVING

Supported models (gpt-5.6 and newer, or a base_model that is) now receive caller and injected prompt_cache_breakpoint fields on bridged /v1/chat/completions requests, a new field on the provider wire, so cache writes and reads now show up where they were 0
Unsupported models get those requests with caller markers stripped, so they keep returning 200 instead of a provider 400
Malformed caller markers (int, list, empty or 5 KB string) on supported models now reach OpenAI and return its 400, where base returned 200 by dropping them
The injected default stays mode: explicit. The fix(responses): preserve prompt cache reuse in chat bridge #42281 flip to implicit is intentionally not included
The deprecated Noma guardrail and the other shared converter callers keep base behavior (no markers), proven by unit tests and live payload capture

Link to Devin session: https://app.devin.ai/sessions/fa4f9e145d28404ca20bef7772ba7011
Open in Devin Desktop: https://app.devin.ai/desktop/session/fa4f9e145d28404ca20bef7772ba7011?variant=devin
Requested by: @yucheng-berri

Note

Medium Risk
Changes bridged /v1/chat/completions provider payloads and prompt-cache behavior for supported models; unsupported models rely on correct model/base_model detection to avoid provider 400s.

Overview
Fixes the chat → Responses API bridge stripping caller prompt_cache_breakpoint markers on text, image, and file blocks (and tool output), which blocked prompt-cache reads on tool-loop traffic.

transform_request now keeps those markers on the converted input when supports_openai_prompt_cache_breakpoint passes for the model or litellm_params.base_model; otherwise it strips them before the upstream call so unsupported models still return 200. convert_chat_completion_messages_to_responses_api gains keep_prompt_cache_breakpoints (default off) so other callers of the converter still drop markers unless they opt in.

Adds unit coverage for keep/strip, alias base_model, and tool output; plus integration tests (including chaos/worker-kill) that assert upstream /v1/responses payloads and spend logging.

^{Reviewed by Cursor Bugbot for commit f4042d8. Bugbot is set up for automated code reviews on this repo. Configure here.}

…esponses bridge Preserve cache-breakpoint markers through the bridge for supported models and drop them for models without breakpoint support Co-authored-by: Simon Sorg <simonsorg13@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-02T01:28:42Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

yassin-berriai · 2026-10-02T01:28:47Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

greptile-apps · 2026-10-02T01:33:41Z

Confidence Score: 5/5

[Medium risk] Preserves prompt cache markers through API response transformation.

The PR appears safe to merge; no new actionable issue or outstanding previous finding was identified.

Summary

The PR preserves prompt-cache breakpoints on supported chat-to-Responses bridge requests while stripping them for unsupported models and, by default, for other converter callers.

Adds text, image, file, tool-output, and proxy integration coverage.
Keeps the existing explicit prompt-cache-options default.

_{Reviews (2) · Last reviewed commit: "fix(responses): keep cache breakpoints o..."}

codspeed · 2026-10-02T01:42:55Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_responses_bridge_keep_cache_breakpoint (f4042d8) with main (df23f11)¹}

No successful run was found on main (9e20264) during the generation of this report, so df23f11 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-10-02T01:43:51Z

Codecov Report

❌ Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...litellm_responses_transformation/transformation.py	90.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yassin-berriai · 2026-10-02T02:48:50Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…llers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-02T03:14:08Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

yassin-berriai · 2026-10-02T05:33:20Z

@greptileai please re-review the current head ccf03ee, the converter now strips markers for non-bridge callers

…nt blocks The marker read ran every chat content block through TypeAdapter(dict[str, object]).validate_python, which rejects dict blocks with non-string keys that chat completion callers passing Python dicts could previously send; the request then failed with a pydantic ValidationError on the bridge keep path, the strip path, and the image/file conversions alike. item is already isinstance-narrowed to a dict at every read site, so read the marker with dict.get directly and drop the adapter. Adds a regression test covering text/image_url/file blocks carrying non-string keys on both the keep (gpt-5.6) and strip (gpt-4o) paths.

yucheng-berri · 2026-10-05T07:04:06Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…oint basedpyright flags the raw dict.get read as reportUnknownArgumentType (+2 against the error budget); cast the isinstance-narrowed block to dict[str, object] first, matching the strip helpers' cast-ok idiom.

A Final assignment inside the conversion loop trips reportGeneralTypeIssues, and per-site casts trip the LIT006 budget; read the marker through one cast-narrowed local instead.

yucheng-berri · 2026-10-05T07:59:49Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

yucheng-berri · 2026-10-05T08:15:41Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…ort gate

yucheng-berri · 2026-10-07T23:17:16Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f4042d8. Configure here.}

…breakpoint an input item Since this branch was cut, main carries prompt_cache_breakpoint markers through the bridge on input items' content blocks (BerriAI#42281, BerriAI#44119, BerriAI#45500), and its tests expect a leading system message whose block carries one to stay an input item so the marker survives. Folding that message into `instructions` dropped the marker and broke eight of those tests. The fold now skips a leading system message any of whose blocks carries a prompt_cache_breakpoint, alongside the non-text case it already skipped. A block with only cache_control, the shape BerriAI#42171 reports, still folds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

yucheng-berri and others added 2 commits October 2, 2026 00:32

fix(responses): honor base_model when gating bridge cache breakpoints

21a0b3c

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team October 2, 2026 01:28

devin-ai-integration Bot assigned yucheng-berri Oct 2, 2026

cursor Bot reviewed Oct 2, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread litellm/completion_extras/litellm_responses_transformation/transformation.py Outdated

Comment thread litellm/completion_extras/litellm_responses_transformation/transformation.py

yucheng-berri and others added 2 commits October 2, 2026 01:54

fix(responses): avoid recursive cache-breakpoint stripping

c8720df

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(responses): justify bridge stripping casts

202c4fb

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 2, 2026

View reviewed changes

fix(responses): keep cache breakpoints out of non-bridge converter ca…

ccf03ee

…llers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 2, 2026

View reviewed changes

cursor Bot reviewed Oct 5, 2026

View reviewed changes

yucheng-berri added 3 commits October 5, 2026 00:13

style(responses): ruff-format the marker-read cast

5160b21

fix(responses): hoist one cast-ok content block read for the type gates

90daacf

A Final assignment inside the conversion loop trips reportGeneralTypeIssues, and per-site casts trip the LIT006 budget; read the marker through one cast-narrowed local instead.

cursor Bot reviewed Oct 5, 2026

View reviewed changes

merge main (#42281 landed): adopt its raw marker reads, keep the supp…

f4042d8

…ort gate

cursor Bot reviewed Oct 7, 2026

View reviewed changes

yucheng-berri approved these changes Oct 7, 2026

View reviewed changes

yucheng-berri merged commit d910653 into main Oct 7, 2026
102 of 105 checks passed

yucheng-berri deleted the litellm_responses_bridge_keep_cache_breakpoint branch October 7, 2026 23:25

This was referenced Oct 8, 2026

fix(responses): keep cache breakpoints on blocks the bridge stringifies #40032

Merged

fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge #45500

Merged

yuneng-berri mentioned this pull request Oct 9, 2026

test: fix stale and flaky tests across CircleCI, GHA and Buildkite #45530

Merged

3 of 26 tasks

poodle64 mentioned this pull request Oct 10, 2026

fix(responses-bridge): fold list-format system content into instructions again (regression of #21192) #42188

Open

6 tasks done

Conversation

devin-ai-integration Bot commented Oct 2, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Screenshots / Proof of Fix

Before (aa601ce)

Tool loop on /v1/chat/completions (bridged to /v1/responses)

Image block with a caller marker on the bridge

Caller markers on models without breakpoint support

After (ccf03ee)

Tool loop on /v1/chat/completions (bridged to /v1/responses)

Image block with a caller marker on the bridge

Caller markers on models without breakpoint support

Unchanged paths

Checkpoint cells (same two legs)

Local providers suite

Deterministic integration audit

Type

Caveats (if any)

Medium

Low

Final Attestation

REVIEWER MUST KNOW BEFORE APPROVING

Uh oh!

devin-ai-integration Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yassin-berriai commented Oct 2, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yassin-berriai commented Oct 2, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Oct 2, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yassin-berriai commented Oct 2, 2026

Uh oh!

yucheng-berri commented Oct 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Oct 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Oct 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

devin-ai-integration Bot commented Oct 2, 2026 •

edited by cursor Bot

Loading

Before (`aa601ce`)

After (`ccf03ee`)

devin-ai-integration Bot commented Oct 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 2, 2026 •

edited

Loading

codspeed Bot commented Oct 2, 2026 •

edited

Loading

codecov Bot commented Oct 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading