fix(litellm/caching/caching_handler.py): fix kwargs[litellm_params][preset_cache_key] not being propagated up the call stack during a cache hit, resulting in the x-litellm-cache-key header not being returned

[mirodrr2]

Title

fix(litellm/caching/caching_handler.py): fix kwargs["litellm_params"]["preset_cache_key"] not being propagated up the call stack during a cache hit, resulting in the x-litellm-cache-key header not being returned

Relevant issues

#8570

Type

🐛 Bug Fix

Changes

The problem is subtle and my fix is currently hacky. I don't know yet enough about the codebase to determine a more "correct" solution to this.

The problem is:

1. The kwargs passed into caching_handler._retrieve_from_cache() does not contain a litellm_params dictionary:

   litellm/litellm/caching/caching_handler.py

   Line 480 in 70923f7

   new_kwargs = kwargs.copy()

2. Further down the call stack, caching._set_preset_cache_key_in_kwargs() expects there to be a litellm_params dictionary, so it can use it to pass the preset_cache_key up the call stack. But it is missing.

   litellm/litellm/caching/caching.py

   Line 365 in 70923f7

   kwargs["litellm_params"]["preset_cache_key"] = preset_cache_key

3. As a result, the preset_cache_key is missing when we try to retrieve it in caching_handler._async_get_cache() using caching_get_preset_cache_key_from_kwargs

   litellm/litellm/caching/caching_handler.py

   Line 179 in 70923f7

   cache_key = litellm.cache._get_preset_cache_key_from_kwargs(

https://github.com/BerriAI/litellm/blob/main/litellm/caching/caching.py#L341

5. This ultimately results in the x-litellm-cache-key not being returned to the caller upon cache hit

The obvious fix for this is to just add an empty litellm_params dictionary to kwargs. However, in the case of a cache miss, this results in a downstream error, where litellm_params ends up being passed to bedrock, and throwing this error:

{"error": {"message": "litellm.BadRequestError: BedrockException - {\"message\":\"The model returned the following errors: Malformed input request: #: extraneous key [litellm_params] is not permitted, please reformat your input and try again.\"}. Received Model Group=anthropic.claude-3-5-sonnet-20240620-v1:0\nAvailable Model Group Fallbacks=None", "type": null, "param": null, "code": "400"}}

So to get around that, if we do not have a cache hit, we detect that, and remove the litellm_params dictionary from kwargs. I only remove it if preset_cache_key is the only key in its dictionary, in an attempt to not interfere with other potential flows I might not be considering

I don't like this fix, but I'm not sure how else to fix it without starting to modify existing interfaces, and turning this into a refactoring pull request. If this fix is too hacky, I'd appreciate some general suggestions as to what may be a better fix

Is there any way to gracefully drop the litellm_params dictionary before it gets passed to bedrock?

[REQUIRED] Testing - Attach a screenshot of any new tests passing locally

Sorry, I haven't been able to get the tests running yet, would appreciate help there. Below is output with the fix. You can see the x-litellm-cache-key header is returned on cache hit now

curl -i "$GATEWAY_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GATEWAY_API_KEY" \
-d '{
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "messages": [
        {
            "role": "user",
            "content": "what did I last say"
        }
    ]
}'

HTTP/2 200 
date: Wed, 26 Feb 2025 02:48:13 GMT
content-type: application/json
content-length: 924
server: uvicorn
server: uvicorn
x-litellm-call-id: b6678ec0-822f-4af6-bf51-e4d8cbcde629
x-litellm-model-id: b0251b3e34d2bc13aa961b90a2fdccf445358bc05d6ad130e7a8c452228f3849
x-litellm-version: 1.61.16
x-litellm-response-cost: 0.0014910000000000001
x-litellm-key-spend: 0.0
x-litellm-response-duration-ms: 2513.21
x-litellm-overhead-duration-ms: 23.606
x-litellm-model-group: anthropic.claude-3-5-sonnet-20240620-v1:0
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0

{"id": "chatcmpl-01bd0687-5b2e-46d2-8575-6dcafebb63db", "created": 1740538093, "model": "anthropic.claude-3-5-sonnet-20240620-v1:0", "object": "chat.completion", "system_fingerprint": null, "choices": [{"finish_reason": "stop", "index": 0, "message": {"content": "I apologize, but I don't have any information about what you last said. As an AI language model, I don't retain information from previous conversations or have access to any chat history beyond the current exchange. Each interaction starts fresh, so I can only respond to the information you provide in your current message. If you'd like to discuss a specific topic or ask a question, please feel free to do so, and I'll do my best to assist you.", "role": "assistant", "tool_calls": null, "function_call": null}}], "usage": {"completion_tokens": 97, "prompt_tokens": 12, "total_tokens": 109, "completion_tokens_details": null, "prompt_tokens_details": null}}%                                                                                                                           (fresh_venv) [ mirodrr | ~/genai-gateway-github-3/output] > curl -i "$GATEWAY_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GATEWAY_API_KEY" \
-d '{
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "messages": [
        {
            "role": "user",
            "content": "what did I last say"
        }
    ]
}'

HTTP/2 200 
date: Wed, 26 Feb 2025 02:48:18 GMT
content-type: application/json
content-length: 924
server: uvicorn
server: uvicorn
x-litellm-call-id: e5c7421f-1855-4b98-9dde-633bf0a9b577
x-litellm-cache-key: d7df71804288612c31a392c87841fb9b0a6dfa3dfee0b3a7f44787882100f7dc
x-litellm-version: 1.61.16
x-litellm-key-spend: 0.0014910000000000001
x-litellm-model-group: anthropic.claude-3-5-sonnet-20240620-v1:0
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0

{"id": "chatcmpl-01bd0687-5b2e-46d2-8575-6dcafebb63db", "created": 1740538093, "model": "anthropic.claude-3-5-sonnet-20240620-v1:0", "object": "chat.completion", "system_fingerprint": null, "choices": [{"finish_reason": "stop", "index": 0, "message": {"content": "I apologize, but I don't have any information about what you last said. As an AI language model, I don't retain information from previous conversations or have access to any chat history beyond the current exchange. Each interaction starts fresh, so I can only respond to the information you provide in your current message. If you'd like to discuss a specific topic or ask a question, please feel free to do so, and I'll do my best to assist you.", "role": "assistant", "tool_calls": null, "function_call": null}}], "usage": {"completion_tokens": 97, "prompt_tokens": 12, "total_tokens": 109, "completion_tokens_details": null, "prompt_tokens_details": null}}%                                                                                                                           

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 26, 2025 3:47am

[mirodrr2]

@ishaan-jaff can I get a review today?

[ghost]

Hi @mirodrr2 can you rebase with main and add a test for your change in here

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Eduard-Voiculescu]

Any updates to when this PR be merged?

[ghost]

ack @Eduard-Voiculescu - i'll look at this tomorrow

[dragonsahead]

Is it just me or there's no x-litellm-cache-key header at all in the response? using v1.74.0-stable

[Math2Compute]

Is it just me or there's no x-litellm-cache-key header at all in the response? using v1.74.0-stable

Same for me, tried many things but did not manage to get this x-litellm-cache-key header. I upgraded to v1.76.0 and did not have any luck.