Repository navigation
fix(bedrock): make prompt caching work on the Nova InvokeModel route - #41343
Merged
mateo-berri merged 4 commits intoSep 16, 2026
Merged
Conversation
Nova InvokeModel rejects the standalone cachePoint blocks the shared Converse transform emits, so each one is folded into the block it caches and tool_config injection points are dropped before the transform runs, since this route has no tool caching to credit. Usage reads Bedrock's Count-suffixed cache keys and adds cached tokens into prompt_tokens, streaming routes every wrapped InvokeModel event through the Converse chunk parser and tolerates the missing totalTokens, and the Nova 1 cost-map entries gain cache_read_input_token_cost at a quarter of the input rate
devin-ai-integration
Bot
requested a review
from mateo-berri
as a code owner
September 16, 2026 01:05
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
Contributor
Greptile SummaryThis PR enables prompt caching on the Bedrock Nova InvokeModel route by translating cache points into Nova’s accepted request shape, normalizing InvokeModel cache usage fields, preserving stream lifecycle events, and adding cache-read pricing metadata and regression coverage
Confidence Score: 5/5The PR appears safe to merge, with no outstanding correctness or repository-rule findings The latest changes replace vendor-rate assertions with stable range and billing invariants, and the prior thread was manually resolved after that correction Important Files Changed
Reviews (3): Last reviewed commit: "test(bedrock): assert the Nova cache-rea..." | Re-trigger Greptile |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…r stops flagging it
mateo-berri
requested review from
kerry-berri and
ryan-crabbe-berri
as code owners
September 16, 2026 19:30
…result to the last text block
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…of pinning the vendor ratio
Contributor
Contributor
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 6015437. Configure here.
mateo-berri
deleted the
litellm_lit5030_bedrock_invoke_nova_prompt_caching
branch
September 16, 2026 20:42
3 of 5 tasks
Stanley00
pushed a commit
to stanley-fork/litellm
that referenced
this pull request
Sep 16, 2026
…ext merge PR BerriAI#41343 and PR BerriAI#41112 both added cache_read_input_token_cost to the six amazon.nova-{micro,lite,pro}-v1:0 and us.amazon.nova-* entries, one at the top of each entry and one at the bottom. The merge kept both, so every PR now fails test_price_map_has_no_duplicate_keys. Both copies carried the same value, so this only removes the trailing duplicate in both price files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
bedrock/invoke/Nova requests withcache_controlget a Bedrock 400 on the cache pointHow it solves it:
cacheReadInputTokenCountandcacheWriteInputTokenCountnext to the Converse namescache_read_input_token_costto the 16 Nova entries that lacked ittool_configinjection points on Invoke, which has no tool cachingUser Flow
Before: a developer whose app sends the same long system prompt to Nova Pro through a
bedrock/invoke/deployment turns on prompt caching and every request fails, and the requests that do get through report a fraction of the tokens they used"model": "nova-pro"(abedrock/invoke/us.amazon.nova-pro-v1:0deployment), a 12k-token system message carrying"cache_control": {"type": "ephemeral"}, and the user messageReply with exactly: OK.Malformed input request: #/system/1: required key [text] not foundandx-litellm-response-cost: 0"stream": trueand"stream_options": {"include_usage": true}and get zero SSE chunks, only the same 400 error body#/messages/0/content/2: extraneous key [cachePoint] is not permitted"prompt_tokens": 5,"cached_tokens": 0, andx-litellm-response-costfor 5 input tokens, so the 12,262 cached tokens vanish from usage and spend"stream": trueandresponse.completedreports"input_tokens": 13at a cost of $0.0000168After: the same requests go through, the second one reads from the cache, and usage and spend carry the cached tokens at the cache read rate
cache_controlon the system message"cache_creation_input_tokens": 12262,"cached_tokens": 0, andx-litellm-response-cost-cache-creationin the response headers"cached_tokens": 12262,"cache_read_input_tokens": 12262, andx-litellm-response-cost: 0.002466, about a quarter of the first call, withx-litellm-response-cost-cache-readset"stream": truethe chunks arrive with"finish_reason": "stop"and the last chunk'susagecarries the samecached_tokensandcost"prompt_tokens": 12267,"cached_tokens": 12262, and the same cache read cost"stream": trueends withresponse.completedreporting"input_tokens": 12267and the real costRelevant issues
Related: #31693 maps the Count-suffixed usage fields only and its branch now carries unrelated commits; #34498 and #34542 cover the Claude Invoke route and stay separate
Affected release
Linear ticket
Resolves LIT-5030
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both sides ran the same seven cases in the same order against live Bedrock in us-east-1, each proxy booted with no database and
--num_workers 2from its own worktree and venv, withLITELLM_LOCAL_MODEL_COST_MAP=Trueso prices come from that tree's cost map: Before at the merge base 226b1e1 on port 20101, After at 0dff64c on port 38470. Commit 6015437 came after the run and changes two test assertions only, so it cannot change behavior and the After run stands at 0dff64cConfig on both sides (credentials via
os.environ):nova-pro-replaypoints at a local endpoint that replays one Nova InvokeModel response recorded from a live cache hit (cacheReadInputTokenCount: 12262), so both sides read the same cache-hit body regardless of Bedrock's cache timing. Payloads:req.json:nova-pro, a 69,318-character system message (Section 1: The quarterly compliance handbook ...) with"cache_control": {"type": "ephemeral"}, the user messageReply with exactly: OK.,max_tokens: 20req_stream.json:req.jsonplus"stream": true, "stream_options": {"include_usage": true}req_messages.json: the Anthropic Messages shape for /v1/messages, the same text as asystemblock withcache_control, the same user messagereq_responses_stream.json: the same text asinstructions,inputReply with exactly: OK.,"stream": true,max_output_tokens: 20req_image_tool.json: a user turn with a text block plus an image block carryingcache_control, an assistantget_weathertool call, a tool resultsunnycarryingcache_control, theget_weathertool,max_tokens: 30req_converse.json:req.jsonwith"model": "nova-pro-converse"and the text prefixedConverse control.req_replay.json:req.jsonwith"model": "nova-pro-replay"and nocache_controlJbelow is the jq summary filter{status: (.error.message // .choices[0].message.content), finish_reason: .choices[0].finish_reason, usage}Before (226b1e1)
Chat cache write then read
curl -s -o before_run1_body.json -w 'HTTP %{http_code}\n' -D before_run1_headers.txt localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req.jsonjq -c "$J" before_run1_body.jsongrep -i 'x-litellm-response-cost' before_run1_headers.txtcurl -s -o before_run2_body.json -w 'HTTP %{http_code}\n' -D before_run2_headers.txt localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req.jsonjq -c "$J" before_run2_body.jsongrep -i 'x-litellm-response-cost' before_run2_headers.txtChat stream usage
curl -s -N localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_stream.json | tee before_stream_raw.txt | grep -c '^data:'grep -o '"finish_reason":"[a-z_]*"' before_stream_raw.txt | sort | uniq -cgrep '"usage":{' before_stream_raw.txt | sed 's/^data: //' | jq -c '.usage // .error'tail -c 300 before_stream_raw.txtMessages API route
curl -s -o before_messages_body.json -w 'HTTP %{http_code}\n' localhost:20101/v1/messages -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_messages.jsonjq -c 'if .error then .error else {text: .content[0].text, stop_reason, usage} end' before_messages_body.jsonResponses API stream usage
curl -s -N localhost:20101/v1/responses -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_responses_stream.json | tee before_responses_raw.txt | grep -o '^data: {"type":"[a-z_.]*"' | sed 's/^data: //' | sort | uniq -cgrep '^data: {' before_responses_raw.txt | sed 's/^data: //' | jq -c 'select(.type == "response.completed") | {status: .response.status, usage: .response.usage}'Chat image and tool result cache
curl -s -o before_image_tool_body.json -w 'HTTP %{http_code}\n' localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_image_tool.jsonjq -c "$J" before_image_tool_body.jsonConverse route control
curl -s -o before_converse_body.json -w 'HTTP %{http_code}\n' localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_converse.jsonjq -c "$J" before_converse_body.jsonReplay cache-hit body
curl -s -o before_replay_body.json -w 'HTTP %{http_code}\n' -D before_replay_headers.txt localhost:20101/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_replay.jsonjq -c "$J" before_replay_body.jsongrep -i 'x-litellm-response-cost' before_replay_headers.txtAfter (0dff64c)
Chat cache write then read
curl -s -o after_run1_body.json -w 'HTTP %{http_code}\n' -D after_run1_headers.txt localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req.jsonjq -c "$J" after_run1_body.jsongrep -i 'x-litellm-response-cost' after_run1_headers.txtcurl -s -o after_run2_body.json -w 'HTTP %{http_code}\n' -D after_run2_headers.txt localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req.jsonjq -c "$J" after_run2_body.jsongrep -i 'x-litellm-response-cost' after_run2_headers.txtChat stream usage
curl -s -N localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_stream.json | tee after_stream_raw.txt | grep -c '^data:'grep -o '"finish_reason":"[a-z_]*"' after_stream_raw.txt | sort | uniq -cgrep '"usage":{' after_stream_raw.txt | sed 's/^data: //' | jq -c '.usage // .error'tail -c 300 after_stream_raw.txtMessages API route
curl -s -o after_messages_body.json -w 'HTTP %{http_code}\n' localhost:38470/v1/messages -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_messages.jsonjq -c 'if .error then .error else {text: .content[0].text, stop_reason, usage} end' after_messages_body.jsonResponses API stream usage
curl -s -N localhost:38470/v1/responses -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_responses_stream.json | tee after_responses_raw.txt | grep -o '^data: {"type":"[a-z_.]*"' | sed 's/^data: //' | sort | uniq -cgrep '^data: {' after_responses_raw.txt | sed 's/^data: //' | jq -c 'select(.type == "response.completed") | {status: .response.status, usage: .response.usage}'Chat image and tool result cache
curl -s -o after_image_tool_body.json -w 'HTTP %{http_code}\n' localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_image_tool.jsonjq -c "$J" after_image_tool_body.jsonConverse route control
curl -s -o after_converse_body.json -w 'HTTP %{http_code}\n' localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_converse.jsonjq -c "$J" after_converse_body.jsonReplay cache-hit body
curl -s -o after_replay_body.json -w 'HTTP %{http_code}\n' -D after_replay_headers.txt localhost:38470/v1/chat/completions -H 'Authorization: Bearer sk-lit5030' -H 'Content-Type: application/json' -d @req_replay.jsonjq -c "$J" after_replay_body.jsongrep -i 'x-litellm-response-cost' after_replay_headers.txtObservations from the run, each noting whether this PR causes it:
cache_controlfor non-Claude models, left alone by this PRType
🐛 Bug Fix
Caveats (if any)
Low
bedrock/converse/now cost 25% of input instead of $0, so spend rises for teams already caching there. Keeping the $0 rate would keep under-reporting what AWS bills, so it staystool_configinjection points are dropped on Invoke, so tools are never cached there (Bedrock 400s on the point Before)cache_controlfor non-Claude models (pre-existing)cache_controlrides the same chat transform after the Responses bridge and was not run live with a cache point; the Responses leg proves the stream usage only/bedrock/model/nova-pro/invoke-with-response-streamwith Bedrock's UnknownOperationException before and afteropenai/gpt-oss-20bnon-serverless 400s across llm_translation_testing and local_testing, the budget reset and router fallback tests, the Sumo Logic NDJSON test) fail identically on every other branch pipeline today and touch no Bedrock code. The AssemblyAI passthrough test in proxy_pass_through_endpoint_tests failed on an AssemblyAI 500 that the sibling pipelines hit on the same suite, and it passed at 0dff64cFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
0dff64c passes /live-pr-risk