fix(caching): scope automatic breakpoints to supported Claude transports

[tin-berri]

TLDR

Problem this solves:

• Vertex Claude silently skips automatic prompt-caching breakpoints
• Non-Claude Bedrock models can receive incorrect gateway-caching attribution

How it solves it:

• Require Claude identity and prompt-caching capability on supported transports
• Include Anthropic, Bedrock, Vertex AI, and Azure AI
• Preserve existing client controls and explicitly configured injection points

The shared hook delegates provider eligibility to llms/anthropic/common_utils.py. Chat seeding and affinity judge SDK alias targets, while native Messages retains its existing routing semantics. Client controls in the normal request or extra_body suppress gateway injection. Generic supports_prompt_caching() remains unchanged for implicit-cache usage and pricing

User Flow

Before: OpenWebUI conversations with Vertex Claude do not receive automatic cache breakpoints

1. An admin enables enable_anthropic_prompt_caching on their gateway
2. A user opens OpenWebUI at http://localhost:16482, selects vertex-sonnet, and sends a long reference followed by a second question
3. Both answers succeed, but the requests receive no gateway-injected caching attribution

After: the same conversation receives cache breakpoints and gateway-attributed savings

1. An admin enables enable_anthropic_prompt_caching on their gateway
2. A user opens OpenWebUI at http://localhost:16482, selects vertex-sonnet, and sends a long reference followed by a second question
3. Both answers succeed, and the second turn reports 17,656 cached-read tokens with gateway-injected caching attribution

Relevant issues

Related: #33573 deferred Vertex support because the provider also hosts Gemini. #41419 already filters unsupported Bedrock wire cache points; this fixes the earlier eligibility and attribution decision

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is as isolated as possible
• I have received a Greptile Confidence Score of at least 4/5

The mapped hook and Anthropic utility suites pass all 537 cases. All four response-round mutations are caught: chat alias identity, affinity alias identity, extra-body ownership, and Claude-family eligibility. The four caller suites produce 372 passes and the same one failure on both base and fix: test_async_success_handler_truncates_large_base64_off_the_event_loop, asserting that the loop thread is absent from recorded scan threads

The response-round make check passes with CI's Python 3.12, including all four budget gates and formatting. Dashboard lint and both generated API artifacts passed on the initial revision; those files are unchanged by the response, whose staged scope skipped those checks. Final-revision CI and bot reviews are pending

Screenshots / Proof of Fix

All requests used synthetic content and paid upstream inference. Bedrock calls used AWS directly. Vertex requests traversed the actual Vertex adapter into a local recording relay, then a real gateway Vertex deployment; only local Google authentication was substituted. This verifies payloads, usage, attribution, and persistence, but is not a direct Google-credential test

The gateway enabled enable_anthropic_prompt_caching: true, with aliases vertex-sonnet for vertex_ai/claude-sonnet-5, bedrock-sonnet for bedrock/converse/global.anthropic.claude-sonnet-5, and bedrock-gpt for bedrock/us.openai.gpt-6-astra

Reproduce the payloads and requests with an authenticated gateway using the same aliases:

export PROXY=http://127.0.0.1:16480
export KEY='<your-local-test-key>'
python3 - <<'PY'
import json
from pathlib import Path
system = 'Cache eligibility verification. Reply only OK.\n' + '\n'.join(
    f'Reference record {i}: the amber service reads stable configuration and emits one concise acknowledgement.'
    for i in range(550)
)
messages = [{'role': 'system', 'content': system}, {'role': 'user', 'content': 'Reply only OK'}]
for model in ('vertex-sonnet', 'bedrock-sonnet', 'bedrock-gpt'):
    Path(f'{model}.json').write_text(json.dumps({'model': model, 'messages': messages, 'max_tokens': 32 if model == 'bedrock-gpt' else 8}))
Path('responses.json').write_text(json.dumps({'model': 'vertex-sonnet', 'instructions': system, 'input': 'Reply only OK', 'max_output_tokens': 32}))
Path('messages.json').write_text(json.dumps({'model': 'vertex-sonnet', 'system': system, 'messages': messages[1:], 'max_tokens': 32}))
PY
probe() {
  curl -sS "$PROXY/v1/$1" -H "Authorization: Bearer $KEY" \
    -H 'Content-Type: application/json' --data-binary "@$2.json"
}

Before (1d91fc2)

Chat completions

1. Run probe chat/completions vertex-sonnet
2. HTTP 200, 17,630 prompt tokens, zero cache-write/read tokens, zero outbound controls, no gateway attribution

Responses

1. Run probe responses responses
2. HTTP 200, 17,630 input tokens, zero cached/write tokens, zero outbound controls

Messages

1. Run probe messages messages
2. HTTP 200, 17,630 input tokens, zero cache-write/read tokens, zero outbound controls

Bedrock model-family boundary

1. Run probe chat/completions bedrock-sonnet, then probe chat/completions bedrock-gpt
2. Both return HTTP 200. Claude writes 17,628 cache tokens and receives gateway attribution. GPT also incorrectly receives gateway attribution

OpenWebUI conversation

1. Open http://127.0.0.1:16482, select vertex-sonnet, send a long synthetic handbook asking for READY ONE, then ask for READY TWO using the same reference
2. Both turns complete. Browser requests contain zero client controls and the gateway adds none

Existing Claude Code controls

1. Run a fresh isolated Claude Code 2.1.277 session against the local gateway, asking for READY ONE, then resume that session asking for READY TWO
2. Both turns exit successfully. Each sends three client cache controls; neither receives gateway-injection attribution

After (2b086dc)

Chat completions

1. Run probe chat/completions vertex-sonnet
2. HTTP 200, 17,628 cache-write tokens, two outbound controls, attribution to cache-rig-vertex

Responses

1. Run probe responses responses
2. HTTP 200, 17,628 cache-write tokens, two outbound controls, attribution to cache-rig-vertex

Messages

1. Run probe messages messages
2. HTTP 200, 17,628 cache-read tokens, two outbound controls, attribution to cache-rig-vertex

Bedrock model-family boundary

1. Run probe chat/completions bedrock-sonnet, then probe chat/completions bedrock-gpt
2. Both return HTTP 200. Claude writes 17,628 cache tokens with gateway attribution. GPT reports 9,369 cache-write tokens with no gateway attribution; generic cache usage and pricing remain unchanged

OpenWebUI conversation

1. Open http://127.0.0.1:16482, select vertex-sonnet, send a long synthetic handbook asking for READY ONE, then ask for READY TWO using the same reference
2. Both turns complete in a real Chrome browser. Client controls remain zero; each chat turn receives one gateway breakpoint
3. The first turn writes 17,656 cache tokens. The second reads 17,656 and writes 56. Its persisted response ID is chatcmpl-00be1ebe-bde9-4b88-974a-8b5ae070bf92, with attribution to cache-rig-vertex
4. Both selected user-turn rows persist the gateway marker and cache usage. Numeric savings were not recomputed for this latest run; background OpenWebUI requests are excluded from these observations

Existing Claude Code controls

1. Run a fresh isolated Claude Code 2.1.277 session against the local gateway, asking for READY ONE, then resume that session asking for READY TWO
2. Both turns exit successfully. Each sends three client cache controls; neither receives gateway-injection attribution. The first writes 8,394 cache tokens; the second reads 8,394 and writes 42. This supplementary check used the real CLI's -p and --resume modes, not an interactive TUI

Type

Bug Fix

Caveats

Medium

• Other relays and opaque model identities remain unchanged/ineligible
• Azure coverage is automated; live Azure inference was not run

Low

• The unrelated logging-thread test fails identically on the base
• Cache hits depend on upstream routing, TTL, and minimum size
• Cache writes cost money; savings remain net of write premiums

Final Attestation

• Tests cover Claude families, unsupported capability, non-Claude models, both provider-resolution modes, client controls, affinity, and wire preservation

---

Note

Medium Risk
Changes when the gateway mutates outbound LLM payloads and spend attribution for prompt caching; mis-scoping could skip caching on Vertex/Azure or inject on wrong models, though tests and stand-down logic mitigate that.

Overview
Automatic Anthropic cache_control injection is narrowed and extended so it only runs for Claude models on Anthropic, Bedrock, Vertex AI, and Azure AI, using a shared supports_anthropic_cache_control() helper instead of a Bedrock/Anthropic-only provider gate.

Client-owned breakpoints are detected in both the top-level request and extra_body, so gateway auto-injection stands down when the SDK puts cache_control in the envelope. Stand-down and default-injection paths now thread request_kwargs through configured injection-point judgment.

Router cache affinity resolves litellm.model_alias_map before deciding defaults, so aliased models (e.g. router replacements) get the same eligibility as their target Claude deployment.

Proxy/UI docs for enable_prompt_caching now list Vertex AI and Azure AI. Tests cover new transports, non-Claude Bedrock/Vertex models, extra_body controls, and alias-aware completion flows.

^{Reviewed by Cursor Bugbot for commit 2b086dc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score? Please review the final SHA and live evidence

[tin-berri]

bugbot run
Please review Claude-only eligibility across supported transports, client cache-control preservation, and the before-and-after live evidence on this final SHA

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_claude_auto_cache_providers (2b086dc) with main (209a780)¹}

Footnotes

1. No successful run was found on main (bd82d73) during the generation of this report, so 209a780 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[cursor]

Stale Bugbot comment from a previous run.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge with no outstanding actionable findings.

Summary

This revision centralizes Claude prompt-cache eligibility in the Anthropic provider boundary, resolves model aliases for chat seeding and affinity, and preserves client-owned cache controls supplied through either normal request fields or extra_body.

• Extends automatic breakpoints to supported Claude models on Anthropic, Bedrock, Vertex AI, and Azure AI.
• Excludes non-Claude models even when their transport advertises generic prompt-caching support.
• Adds synchronous and asynchronous regression coverage for aliases, client controls, configured injection points, and supported transports.
• Updates proxy descriptions and generated dashboard API declarations to reflect the expanded transport coverage.

_{Reviews (2) · Last reviewed commit: "fix(caching): scope automatic breakpoint..."}

[codecov]

Codecov Report

❌ Patch coverage is 88.23529% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/llms/anthropic/common_utils.py	75.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[tin-berri]

@greptileai review pls

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2b086dc. Configure here.}