Preserve cache_control for Qwen on OpenRouter + Anthropic adapter - #2
Merged
Conversation
Three related fixes so cache_control survives end-to-end when routing Anthropic /v1/messages traffic to Qwen/DeepSeek upstreams via OpenRouter (measured ~0% -> ~95% cache hit rate on Alibaba/DeepSeek routes): 1. Add QWEN to CacheControlSupportedModels so the OpenRouter chat handler stops stripping cache_control before the upstream call. Alibaba (Qwen) requires explicit cache_control breakpoints. DeepSeek is intentionally omitted: its caching is automatic/prefix-based and ignores cache_control, so listing it would be inert. 2. Widen is_anthropic_claude_model to recognize qwen so the Anthropic->OpenAI adapter applies cache_control for Qwen models. DeepSeek is likewise omitted here; this also keeps DeepSeek's thinking translated to reasoning_effort rather than passed through raw. 3. Drop the x-anthropic-billing-header system block in the adapter path (mirrors _filter_billing_headers_from_system in the messages adapter). Its per-request hash otherwise invalidates the prefix cache every turn, pinning cache_read_input_tokens at 0. This is model-agnostic and is what restores DeepSeek's automatic prefix caching. Adds unit tests for all paths, including that DeepSeek does not receive a cache_control block and is not treated as cache_control-capable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When Claude Code is pointed at Qwen/DeepSeek through the LiteLLM proxy (Anthropic
/v1/messages-> OpenRouter),cache_controlwas lost end-to-end, so every turn paid the full input rate (cache_read_input_tokensstayed at 0). These fixes let the prefix cache land (measured ~0% -> ~95% hit rate on Alibaba/DeepSeek routes).This supersedes #1, which incorrectly added
cache_controlsupport for DeepSeek. Per OpenRouter's prompt-caching docs, DeepSeek caching is automatic/prefix-based and ignorescache_control; only Alibaba (Qwen) requires explicitcache_controlbreakpoints. DeepSeek's caching is restored solely by the billing-header strip (fix 3).Changes
openrouter/chat/transformation.py— addQWEN(only) toCacheControlSupportedModels. Alibaba/Qwen requires explicitcache_controlcontent blocks; without this the handler strips the marker before the upstream call. DeepSeek is intentionally omitted (its caching is automatic and ignorescache_control, so listing it would be inert).anthropic/.../adapters/transformation.py—is_anthropic_claude_model— recognizeqwenso the Anthropic->OpenAI adapter appliescache_controlfor Qwen. DeepSeek is omitted here too; this also keeps DeepSeek'sthinkingtranslated toreasoning_effortrather than passed through raw. The one behavioral tradeoff (rawthinkingpassthrough) now applies to Qwen only; callers usedrop_params: trueand/or setreasoning_effortexplicitly, so it is benign in practice.anthropic/.../adapters/transformation.py—_add_system_message_to_messages— drop thex-anthropic-billing-header:system block. Claude Code injects this ahead of thecache_controlmarker with a per-request hash; left in, it invalidates the upstream prefix-cache key every turn. This is model-agnostic and is what restores DeepSeek's automatic prefix caching. Mirrors_filter_billing_headers_from_systemin the sibling messages adapter.Tests
is_anthropic_claude_modelrecognizes qwen and rejects deepseek (and still rejects gpt/gemini).transform_requestpreservescache_controlfor qwen and drops it for deepseek.cache_control.All touched files pass
black; the targeted unit tests pass.