fix(llm): don't emit explicit cache_control markers for Gemini - #3586
Conversation
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
||||||||||||||||||||||||||||||
juanmichelini
left a comment
juanmichelini
left a comment
There was a problem hiding this comment.
LGTM, tested here https://openhands-eval-monitor.vercel.app/?run=swebench%2Flitellm_proxy-gemini-3-5-flash%2F27218792462%2F&days=15
Costs are much lower. Less than <1 usd per instance
…it-cache_control-markers-3090-freeze-cache-at-static-prefix-614-cost
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
The SDK now suppresses explicit cache_control markers for Gemini prompt formatting while preserving Anthropic explicit caching behavior.
Does this PR achieve its stated goal?
Yes. I exercised the SDK's public LLM/message formatting path before and after the PR with a multi-turn, non-adjacent system/last-user conversation: on main, Gemini models reported prompt caching active and emitted 2 cache_control markers; on this PR, the same Gemini models reported prompt caching inactive and emitted 0 markers. The same PR run still emitted 2 markers for claude-sonnet-4-20250514, so the Anthropic path remains active.
| Phase | Result |
|---|---|
| Environment Setup | ✅ make build completed and installed the uv-managed workspace successfully |
| CI Status | |
| Functional Verification | ✅ Public SDK LLM formatting behavior verified before/after; no functional issues found |
Functional Verification
Test 1: Gemini explicit cache markers are removed
Step 1 — Reproduce / establish baseline without the fix:
Ran git checkout --detach origin/main && uv run python - <<'PY' ... with a script that imports LLM, Message, and TextContent, builds a 4-message Gemini conversation, calls format_messages_for_llm(), and counts cache_control keys.
Relevant output:
{
"gemini-2.5-pro": {
"cache_control_markers": 2,
"is_caching_prompt_active": true
},
"gemini-3.1-pro-preview": {
"cache_control_markers": 2,
"is_caching_prompt_active": true
},
"litellm_proxy/gemini-3.1-pro-preview": {
"cache_control_markers": 2,
"is_caching_prompt_active": true
}
}This confirms the regression shape exists on the base branch: Gemini prompt caching is active and the formatted payload contains explicit markers.
Step 2 — Apply the PR's changes:
Checked out PR commit 8317a84715b4f1712c8250d573fee2e58007dedd.
Step 3 — Re-run with the fix in place:
Ran the same SDK script on the PR commit.
Relevant output:
{
"gemini-2.5-pro": {
"cache_control_markers": 0,
"is_caching_prompt_active": false
},
"gemini-3.1-pro-preview": {
"cache_control_markers": 0,
"is_caching_prompt_active": false
},
"litellm_proxy/gemini-3.1-pro-preview": {
"cache_control_markers": 0,
"is_caching_prompt_active": false
}
}This confirms the PR delivers the stated SDK behavior: Gemini no longer receives explicit cache_control markers even when caching_prompt=True.
Test 2: Anthropic explicit prompt caching still works
The same before/after script included claude-sonnet-4-20250514 as a control model. On the PR commit it returned:
{
"claude-sonnet-4-20250514": {
"cache_control_markers": 2,
"is_caching_prompt_active": true
}
}This confirms the PR did not disable explicit prompt caching for the Anthropic model path.
Unable to Verify
I did not run a live Gemini request or inspect real cache_read_tokens billing behavior because that requires provider credentials/cost-bearing external LLM calls. This QA verified the SDK request-formatting behavior that determines whether explicit cache markers are sent. Future QA guidance in AGENTS.md could document whether QA agents are allowed to spend Gemini credits and which environment variable should be used for live cache-metric checks.
Issues Found
None from functional QA.
This review was created by an AI agent (OpenHands) on behalf of the user.
3 participants
AGENT:
Validation is deterministic + offline. The regression is a cache-billing behavior, so the proof is twofold:
test_gemini_prompt_caching_emits_no_markersbuilds a multi-turn Gemini conversation where the system message (index 0) and the last user message (index 3) are non-adjacent — the exactshape that froze the cache — and asserts
is_caching_prompt_active()isFalseand that nocache_controlmarker appears anywhere in the formatted payload. No markers → no explicitcachedContentattached → Gemini stays onGoogle's implicit-caching path.
re-run a live Gemini eval for this PR; the offline test reproduces the root cause deterministically.
Commands run:
uv run pytest tests/sdk/llm/test_model_features.py::test_prompt_cache_support tests/sdk/llm/test_prompt_caching_cross_conversation.py→ 28 passeduv run pytest tests/sdk/llm/test_model_features.py tests/sdk/llm/test_llm_completion.py→ 165 passed (confirms theprompt-cache-too-smallretry tests, which useclaude-sonnetas the real model, are unaffected)Why
#3090addedgemini-2.5/gemini-3toPROMPT_CACHE_MODELS, which made the SDK emit Anthropic-stylecache_controlmarkers for Gemini. For Anthropic, the second marker on the last user/tool message extends the cached prefixevery turn. For Gemini via LiteLLM it does the opposite:
separate_cached_messages) keeps only the first contiguous marked block. The SDK marks two non-adjacent messages (system + last user/tool), so after turn 1 the cachecollapses to just the static system prefix — frozen for the whole trajectory.
cachedContentto the request disables Google's implicit caching on the growing conversation body, so the body is re-billed as fresh input every turn.Net effect: Gemini runs since post-#3090 silently cost ~6–14× more, with no resolve-rate benefit. Before #3090 (no markers), Gemini's implicit caching covered the full growing prefix (~75–83% hit).
Summary
gemini-2.5/gemini-3fromPROMPT_CACHE_MODELSsosupports_prompt_cacheisFalsefor Gemini → nocache_controlmarkers → Gemini falls back to Google's implicit prefix caching (pre-Enable Gemini prompt cache markers #3090 behavior). Leaves a guardcomment so it isn't re-added.
_apply_prompt_cachingcomment (the moving breakpoint is Anthropic-only).test_prompt_cache_supporttoFalse; replace the old "Gemini emits markers" test with a multi-turn regression test asserting no markers are emitted.Issue Number
#3585
How to Test
cd software-agent-sdk && uv syncuv run pytest tests/sdk/llm/test_model_features.py::test_prompt_cache_support tests/sdk/llm/test_prompt_caching_cross_conversation.pymetrics.token_usages[*].cache_read_tokensper call — it should grow withprompt_tokensinstead of freezing at the static system+tools prefix.Type
Notes
disable_anthropic_gemini_context_caching_transform) exists but was rejected as the fix because it also disables Anthropic explicit caching; gating at the SDK feature table is the surgicalequivalent.
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:8317a84-pythonRun
All tags pushed for this build
About Multi-Architecture Support
8317a84-python) is a multi-arch manifest supporting both amd64 and arm648317a84-python-amd64) are also available if needed