fix(anthropic): map metadata.user_id to prompt_cache_key on the /v1/messages bridge

[mateo-berri]

TLDR

Problem this solves:

• /v1/messages maps metadata.user_id to OpenAI's user but never to prompt_cache_key
• OpenAI gets no cache-routing hint from Anthropic-SDK and Claude Code traffic
• Claude Code's 150-char user_id is longer than OpenAI's 64-char key limit anyway

How it solves it:

• Both bridges set prompt_cache_key to the first 64 chars of user_id
• The chat-completions bridge only sets it when the target provider accepts the param
• A prompt_cache_key the client sends itself still wins over the derived one
• litellm_proxy/ deployments are skipped: the downstream proxy's real provider is unknown and rejects the key unless it runs with drop_params

User Flow

Before: the request reaches OpenAI with user filled in but no prompt_cache_key, so GPT-5.6's cache routing never gets the per-session key the client is already sending

1. The gateway admin adds openai/gpt-5.6-luna to the proxy config under model_name: gpt-5.6-luna and starts the proxy
2. A developer points Claude Code at the gateway (ANTHROPIC_BASE_URL=https://litellm-domain, ANTHROPIC_AUTH_TOKEN=<virtual key>, ANTHROPIC_MODEL=gpt-5.6-luna) and types a prompt
3. Claude Code sends POST https://litellm-domain/v1/messages with model: gpt-5.6-luna, its system prompt, the message, and metadata.user_id set to its 150-character session identity ({"device_id":"8c33...","account_uuid":"...","session_id":"..."}), byte-identical on every turn of the session
4. A 200 comes back in Anthropic shape: id starts with resp_, the answer text, and usage.cache_creation_input_tokens on the first turn
5. The developer opens that call in OpenAI's request log (https://platform.openai.com/logs): the request carries user set to the first 64 characters of the user_id and no prompt_cache_key at all; the response object echoes "prompt_cache_key": null
6. The admin sees the same thing in the proxy's own --detailed_debug output: the POST Request Sent from LiteLLM body going to https://api.openai.com/v1/responses has user and no prompt_cache_key
7. With litellm_settings.use_chat_completions_url_for_anthropic_messages: true, the same POST https://litellm-domain/v1/messages goes out as POST https://api.openai.com/v1/chat/completions with user set and, again, no prompt_cache_key

After: the same request leaves the gateway with prompt_cache_key set to the first 64 characters of metadata.user_id, and OpenAI echoes it back

1. The gateway admin adds openai/gpt-5.6-luna to the proxy config under model_name: gpt-5.6-luna and starts the proxy
2. A developer points Claude Code at the gateway (ANTHROPIC_BASE_URL=https://litellm-domain, ANTHROPIC_AUTH_TOKEN=<virtual key>, ANTHROPIC_MODEL=gpt-5.6-luna) and types a prompt
3. Claude Code sends POST https://litellm-domain/v1/messages with model: gpt-5.6-luna, its system prompt, the message, and metadata.user_id set to its 150-character session identity ({"device_id":"8c33...","account_uuid":"...","session_id":"..."}), byte-identical on every turn of the session
4. A 200 comes back in Anthropic shape: id starts with resp_, the answer text, and usage.cache_creation_input_tokens on the first turn
5. The developer opens that call in OpenAI's request log (https://platform.openai.com/logs): the request carries user and prompt_cache_key, both set to the same first 64 characters of the user_id ({"device_id":"8c33...); the response object echoes "prompt_cache_key": "{\"device_id\":\"8c33..."
6. The admin sees the same thing in the proxy's own --detailed_debug output: the POST Request Sent from LiteLLM body going to https://api.openai.com/v1/responses has both user and prompt_cache_key
7. With litellm_settings.use_chat_completions_url_for_anthropic_messages: true, the same POST https://litellm-domain/v1/messages goes out as POST https://api.openai.com/v1/chat/completions with both user and prompt_cache_key set
8. A client that already sends its own prompt_cache_key in the POST https://litellm-domain/v1/messages body sees that exact value in the upstream request, not the derived one

Relevant issues

Fixes #37508

Linear ticket

Resolves LIT-5875

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live before/after QA, no mocks, real calls to openai/gpt-5.6-luna. Before is a proxy booted from the merge base 6fcdea03b0, After a proxy booted from this PR's tip 8e1c8c1c62. Each side ran two proxies from its commit on random free ports with --detailed_debug: proxy A with the default Responses-API bridge (config_responses.yaml), proxy C with the chat-completions bridge (config_chat.yaml)

config_responses.yaml:

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

config_chat.yaml is the same file plus:

litellm_settings:
  use_chat_completions_url_for_anthropic_messages: true

Clients: Anthropic Python SDK 0.125.0 pointed at the proxy, OpenAI Python SDK 3.3.1 to retrieve the stored upstream objects the way a developer checks what OpenAI received, Claude Code v2.1.237 driven in tmux. Every case sends a ~2860-token system prompt with a fresh nonce so OpenAI starts cold, then the same metadata.user_id on both calls of a pair, 3 s apart. The SDK pair used by cases 1, 2, 4 and 5 (case 2 adds extra_body={"store": True} so the completions can be retrieved, case 4 adds extra_body={"prompt_cache_key": ...}, case 5 adds thinking={"type": "enabled", "budget_tokens": 1024}):

client = anthropic.Anthropic(base_url=f"http://127.0.0.1:{port}", api_key=os.environ["LITELLM_MASTER_KEY"])
for _ in range(2):
    client.messages.create(
        model="gpt-5.6-luna", max_tokens=32, system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": "Reply with the single word ok."}],
        metadata={"user_id": user_id},
    )
    time.sleep(3)
openai.OpenAI().responses.retrieve(upstream_id)

The id the Responses bridge returns is a 241-char wrapper around the upstream resp_ id, which OpenAI's retrieve endpoint rejects (string_above_max_length), so the retrievals below use the upstream id decoded out of it. OpenAI's stored chat object (chat.completions.retrieve) exposes input_user and no prompt_cache_key field on either side, so case 2 shows the request body the gateway sent upstream as read from the admin's --detailed_debug log (POST Request Sent from LiteLLM) plus the raw usage OpenAI returned. Read the cache numbers honestly: cached_tokens rose on the second call on both sides because OpenAI also caches by prefix on a quiet account, so what this PR changes is the routing key OpenAI stores with the request, which is what keeps one user's hits together under load

Surprises seen during QA, none of them a failure:

• Bridged Responses id is a 241-char wrapper, OpenAI retrieve rejects it. PR leaves it alone
• Gateway GET /v1/responses/<id> returns a differently encoded id. PR leaves it alone
• Retrieved objects report cache_write_tokens 0, live responses showed thousands. PR leaves it alone
• Chat retrieve exposes input_user, no prompt_cache_key, null prompt_tokens_details. PR leaves it alone
• Case 2 second completion 404s on immediate retrieve, found on retry. PR leaves it alone
• Thinking reroute returns chatcmpl-<uuid>, hides upstream resp_ id. PR leaves it alone
• Thinking reroute logs "Error decoding response_id ... Incorrect padding". PR leaves it alone
• budget_tokens: 1024 becomes reasoning.effort: low upstream. PR leaves it alone
• Claude Code user and key cut mid-JSON at 64 chars. PR causes the key half, by design (OpenAI limit)
• Claude Code title side request also carries the key. PR causes it
• Client store: true in extra_body is forwarded upstream. PR leaves it alone
• OpenAI cached the second call even without a key. PR leaves it alone
• Chat-path debug curl block prints only https://api.openai.com/v1/. PR leaves it alone
• One-model config, yet /v1/models lists 35 (DB-stored models). PR leaves it alone
• Proxy re-reads the worktree .env, DATABASE_URL survives env -u. PR leaves it alone

Before (6fcdea0)

Case 1: Anthropic SDK, Responses-API bridge (proxy A)

1. Run the SDK pair against proxy A (port 32698) with user_id = lit5875-qa-before-2a635bbf-e707-4a3f-8d39-56e5641ee726, then retrieve both upstream ids with the OpenAI SDK and the second bridged id through the gateway itself
2. Observed:

### openai.responses.retrieve('resp_00afb75bc2b408db006a86e8245d2887d0a0e0b69fcfdf1a88')
"user": "lit5875-qa-before-2a635bbf-e707-4a3f-8d39-56e5641ee726"
"prompt_cache_key": null
"usage.input_tokens": 2864
"usage.input_tokens_details.cached_tokens": 0
### openai.responses.retrieve('resp_0692d2d40fe1c329006a86e828618c819b9614357123fab4e2')
"user": "lit5875-qa-before-2a635bbf-e707-4a3f-8d39-56e5641ee726"
"prompt_cache_key": null
"usage.input_tokens": 2864
"usage.input_tokens_details.cached_tokens": 2861
### GET http://127.0.0.1:32698/v1/responses/<bridged id>  (Authorization: Bearer <master key>)
HTTP 200
"user": "lit5875-qa-before-2a635bbf-e707-4a3f-8d39-56e5641ee726"
"prompt_cache_key": null
"usage.input_tokens_details.cached_tokens": 2861

3. The bridged Anthropic responses report usage.cache_creation_input_tokens: 2861 on call 1 and usage.cache_read_input_tokens: 2861 on call 2 (OpenAI caches by prefix even without a key)

Case 2: Anthropic SDK, chat-completions bridge (proxy C)

1. Run the SDK pair against proxy C (port 52572) with user_id = lit5875-qa-before-7b40ece8-d047-40f8-9ea2-3868fa0837b2 and extra_body={"store": True}, then read the two POST Request Sent from LiteLLM bodies and the raw usage from the proxy's --detailed_debug log
2. Observed:

[outbound 1] keys=['max_completion_tokens', 'messages', 'model', 'store', 'user']
             user='lit5875-qa-before-7b40ece8-d047-40f8-9ea2-3868fa0837b2' prompt_cache_key=<absent>
[outbound 2] keys=['max_completion_tokens', 'messages', 'model', 'store', 'user']
             user='lit5875-qa-before-7b40ece8-d047-40f8-9ea2-3868fa0837b2' prompt_cache_key=<absent>
chatcmpl-EEvSEhQFCjWH4THQelIEI47KC5F5y  prompt_tokens 2864  prompt_tokens_details {"cached_tokens": 0, "cache_write_tokens": 2861}
chatcmpl-EEvSIkhqqE92PUqhvU2aziV9FsIXc  prompt_tokens 2864  prompt_tokens_details {"cached_tokens": 2861, "cache_write_tokens": 0}

Case 3: Claude Code against proxy A

1. Launch Claude Code in tmux with a fresh config dir and cwd, ANTHROPIC_BASE_URL=http://127.0.0.1:32698, ANTHROPIC_AUTH_TOKEN=<master key>, ANTHROPIC_MODEL=gpt-5.6-luna, send Reply with the single word ok. then Now reply with the word again., then retrieve the two prompt requests' upstream ids (the title side request is omitted)
2. Observed in the pane:

❯ Reply with the single word ok.
⏺ ok
❯ Now reply with the word again.
⏺ again

3. Observed on retrieve (user is the first 64 chars of the 150-char user_id, no key):

resp_06ba3dc35262ae3d006a86e8e3661c87d0b1e1e0fe6a137516  user: {"device_id":"14ff65a7dbdf2c27d8cbe71d1202b24bc0263838ea205ee6d1  prompt_cache_key: null  input_tokens 21027  cached_tokens 0
resp_0fba1f46c67c5fa9006a86e8f8f16487d09aee22bb320063ab  user: {"device_id":"14ff65a7dbdf2c27d8cbe71d1202b24bc0263838ea205ee6d1  prompt_cache_key: null  input_tokens 21062  cached_tokens 19376

Case 4: explicit key wins (proxy A)

1. Run one SDK call against proxy A with metadata={"user_id": user_id} and extra_body={"prompt_cache_key": "explicit-lit5875-qa2-before"}, then retrieve the upstream id
2. Observed (unchanged by this PR, the explicit key already passed through):

resp_062b189ab3605d50006a86e9231b00819891f18277fbabc54c  user: lit5875-qa-before-2c89a0e2-c655-4cc0-bbbf-a56b4a9fce3e  prompt_cache_key: "explicit-lit5875-qa2-before"

Case 5: chat bridge with thinking enabled (proxy C)

1. Run one SDK call against proxy C with metadata={"user_id": user_id} and thinking={"type": "enabled", "budget_tokens": 1024} (the chat bridge reroutes this to /v1/responses), then retrieve the upstream id
2. Observed:

resp_01ba7eed2c31a209006a86e8c78dac87d0ab9fcf5151f0f877  user: lit5875-qa-before-9ed3962b-9bdc-4d9c-9573-34694c1c56f6  prompt_cache_key: null  reasoning: present

After (8e1c8c1)

Case 1: Anthropic SDK, Responses-API bridge (proxy A)

1. Run the SDK pair against proxy A (port 35912) with user_id = lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c, then retrieve both upstream ids with the OpenAI SDK and the second bridged id through the gateway itself
2. Observed:

### openai.responses.retrieve('resp_0c833379cda56c2d006a86e810ff58819b9bb041cc86da9657')
"user": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"prompt_cache_key": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"usage.input_tokens": 2866
"usage.input_tokens_details.cached_tokens": 0
### openai.responses.retrieve('resp_02beb47bc7e2158a006a86e814f0a481989e67f832b37c5d19')
"user": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"prompt_cache_key": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"usage.input_tokens": 2866
"usage.input_tokens_details.cached_tokens": 2863
### GET http://127.0.0.1:35912/v1/responses/<bridged id>  (Authorization: Bearer <master key>)
HTTP 200
"user": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"prompt_cache_key": "lit5875-qa-after-90a0a703-2528-46f9-ab3e-811ba4c5fd9c"
"usage.input_tokens_details.cached_tokens": 2863

3. The bridged Anthropic responses report usage.cache_creation_input_tokens: 2863 on call 1 and usage.cache_read_input_tokens: 2863 on call 2

Case 2: Anthropic SDK, chat-completions bridge (proxy C)

1. Run the SDK pair against proxy C (port 25467) with user_id = lit5875-qa-after-a95c5d9f-b481-410f-8a7b-ff7a571f533c and extra_body={"store": True}, then read the two POST Request Sent from LiteLLM bodies and the raw usage from the proxy's --detailed_debug log
2. Observed:

[outbound 1] keys=['max_completion_tokens', 'messages', 'model', 'prompt_cache_key', 'store', 'user']
             user='lit5875-qa-after-a95c5d9f-b481-410f-8a7b-ff7a571f533c' prompt_cache_key='lit5875-qa-after-a95c5d9f-b481-410f-8a7b-ff7a571f533c'
[outbound 2] keys=['max_completion_tokens', 'messages', 'model', 'prompt_cache_key', 'store', 'user']
             user='lit5875-qa-after-a95c5d9f-b481-410f-8a7b-ff7a571f533c' prompt_cache_key='lit5875-qa-after-a95c5d9f-b481-410f-8a7b-ff7a571f533c'
chatcmpl-EEvRwUA5kYw3g5i9XeoAFkyzrcC5O  prompt_tokens 2862  prompt_tokens_details {"cached_tokens": 0, "cache_write_tokens": 2859}
chatcmpl-EEvS06rNPrAhuOhb8LkPX1EnW6k6y  prompt_tokens 2862  prompt_tokens_details {"cached_tokens": 2859, "cache_write_tokens": 0}

Case 3: Claude Code against proxy A

1. Launch Claude Code in tmux with a fresh config dir and cwd, ANTHROPIC_BASE_URL=http://127.0.0.1:35912, ANTHROPIC_AUTH_TOKEN=<master key>, ANTHROPIC_MODEL=gpt-5.6-luna, send Reply with the single word ok. then Now reply with the word again., then retrieve the two prompt requests' upstream ids (the title side request is omitted)
2. Observed in the pane:

❯ Reply with the single word ok.
⏺ ok
❯ Now reply with the word again.
⏺ again

3. Observed on retrieve (user and prompt_cache_key are both the first 64 chars of the 150-char user_id):

resp_05dc7e3993afed48006a86e86043f48199b8244f201a934b00  user: {"device_id":"c146cfa1f87b4ec714fcce3646878f7c2d01c6b7a64ed00d21  prompt_cache_key: {"device_id":"c146cfa1f87b4ec714fcce3646878f7c2d01c6b7a64ed00d21  input_tokens 21027  cached_tokens 0
resp_093be5234b7b1521006a86e86e3cf0819aa270530c458286a2  user: {"device_id":"c146cfa1f87b4ec714fcce3646878f7c2d01c6b7a64ed00d21  prompt_cache_key: {"device_id":"c146cfa1f87b4ec714fcce3646878f7c2d01c6b7a64ed00d21  input_tokens 21062  cached_tokens 19376

Case 4: explicit key wins (proxy A)

1. Run one SDK call against proxy A with metadata={"user_id": user_id} and extra_body={"prompt_cache_key": "explicit-lit5875-after"}, then retrieve the upstream id
2. Observed:

resp_00e9ad99e39fcee6006a86e88ed0a887d08e50abf850ba1e41  user: lit5875-qa-after-13ef6437-cb4d-49f1-b01a-57d52a4fe8ac  prompt_cache_key: "explicit-lit5875-after"

Case 5: chat bridge with thinking enabled (proxy C)

1. Run one SDK call against proxy C with metadata={"user_id": user_id} and thinking={"type": "enabled", "budget_tokens": 1024} (the chat bridge reroutes this to /v1/responses), then retrieve the upstream id
2. Observed:

resp_04785407f76d7fb2006a86e83e38608198a1fe654829824360  user: lit5875-qa-after-ca6b3fe5-0db9-401f-ba0e-9acfc1cb8c19  prompt_cache_key: lit5875-qa-after-ca6b3fe5-0db9-401f-ba0e-9acfc1cb8c19  reasoning: present

Type

🐛 Bug Fix

Caveats (if any)

• Every provider advertising prompt_cache_key (azure, groq, deepseek, openrouter, vllm, ...) now gets it implicitly on /v1/messages, and there is no config switch to turn the derivation off
• A LiteLLM proxy chained in front of OpenAI through litellm_proxy/ gets no derived key
• Claude Code's key ends up per device: its session_id falls past the 64-char cut
• Guardrail and shadow-eval translations never derive the key, since they run without a provider

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

^{Cursor Bugbot is generating a summary for commit cc2013e. Configure here.}

• 8e1c8c1 passes /live-pr-risk

[greptile-apps]

Greptile Summary

The PR derives an OpenAI-compatible prompt_cache_key from Anthropic metadata.user_id across the chat-completions and Responses bridges while preserving explicitly supplied keys.

• Adds a shared 64-character cache-key conversion helper.
• Checks provider parameter support before deriving the key on the chat-completions path.
• Preserves explicit client keys through both bridge handlers.
• Adds transformation and handler coverage for supported providers, truncation, precedence, missing metadata, and Responses rerouting.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/adapters/handler.py	Passes the resolved provider into translation and restores an explicit client cache key after translated parameters are merged.
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py	Derives the cache key only when the resolved provider advertises support, excluding chained LiteLLM proxies with unknown backends.
litellm/llms/anthropic/experimental_pass_through/responses_adapters/handler.py	Preserves an explicit prompt_cache_key when constructing Responses API arguments.
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py	Maps Anthropic metadata.user_id into the Responses API user and prompt_cache_key fields.
litellm/llms/anthropic/experimental_pass_through/utils.py	Adds the shared conversion that stringifies non-null user IDs, truncates them to 64 characters, and omits empty results.
litellm/types/llms/openai.py	Extends the chat-completion request type to represent the cache key assigned by the bridge.

_{Reviews (2): Last reviewed commit: "fix(anthropic): skip the derived prompt_..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cc2013e. Configure here.}

[mateo-berri]

bugbot run

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit_5875_anthropic_prompt_cache_key (8e1c8c1) with litellm_internal_staging (6d47468)}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8e1c8c1. Configure here.}

[mateo-berri]

@greptileai