fix(model_prices): bedrock Grok 4.6 does not support prompt caching - #38699
Closed
FahimaGold wants to merge 2 commits into
Closed
fix(model_prices): bedrock Grok 4.6 does not support prompt caching#38699FahimaGold wants to merge 2 commits into
FahimaGold wants to merge 2 commits into
Conversation
us.xai.grok-4.6 and global.xai.grok-4.6 (litellm_provider bedrock_converse)
are marked supports_prompt_caching: true, but AWS Bedrock does not support
prompt caching for Grok. With litellm.enable_anthropic_prompt_caching on,
AnthropicCacheControlHook.get_default_injection_points sees the flag as true
and auto-injects cache_control breakpoints; Bedrock then rejects the entire
request:
You invoked an unsupported model or your request did not allow prompt
caching.
so every call to these models fails. Set supports_prompt_caching: false and
drop the now-meaningless cache_read_input_token_cost on both rows (added in
BerriAI#37517). Native xAI Grok (xai/grok-4.6) is unaffected. Data kept identical
between the root map and the bundled backup.
Adds tests/test_litellm/test_bedrock_grok_prompt_caching_metadata.py asserting
the flag is false, no cache-read price remains, and the two JSON files agree.
Contributor
Greptile SummaryThe PR corrects prompt-caching capability and pricing metadata for the two Bedrock Converse Grok 4.6 model identifiers.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains, and the previously reported test-organization issue has been fixed by moving the regression into the existing cache-control hook test module.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | Correctly disables prompt caching and removes obsolete cache-read pricing for both Bedrock Converse Grok 4.6 entries. |
| litellm/model_prices_and_context_window_backup.json | Mirrors the root model-map corrections for the bundled fallback metadata. |
| tests/test_litellm/integrations/test_anthropic_cache_control_hook.py | Places the regression test beside the affected hook behavior and covers both model identifiers using the local model-cost map. |
Reviews (2): Last reviewed commit: "test: move Bedrock Grok caching regressi..." | Re-trigger Greptile
Per CLAUDE.md, a bug fix extends the existing mapped test file rather than adding a new top-level one. Fold the regression into TestEnableAnthropicPromptCaching in test_anthropic_cache_control_hook.py as a behavioural check next to the databricks/non-anthropic siblings: with enable_anthropic_prompt_caching on, us./global.xai.grok-4.6 report supports_prompt_caching false and get_default_injection_points returns no breakpoints. Uses local_model_cost_map so it reads this branch's map, not the network copy. Drops the standalone test_bedrock_grok_prompt_caching_metadata.py.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
Author
|
@greptileai review |
6 tasks
Contributor
|
Superseded by the rolling registry PR #38693, which re-verified these Bedrock Grok 4.6 caching changes against the AWS docs. |
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
us.xai.grok-4.6andglobal.xai.grok-4.6(litellm_provider: bedrock_converse) are markedsupports_prompt_caching: truein the model cost map. AWS Bedrock does not support prompt caching for Grok.With
litellm.enable_anthropic_prompt_cachingenabled,AnthropicCacheControlHook.get_default_injection_pointstreats the flag as authoritative and auto-injectscache_controlbreakpoints for these models. Bedrock then rejects the entire request:The result is a 100% failure rate for Bedrock-hosted Grok 4.6 whenever auto prompt-caching is on. Reproduced live against Bedrock across regions.
The wrong flag came in with #37517 (which added these rows). The offline bundled map lags remote, which is why it isn't always visible in local runs.
Fix
For both
us.xai.grok-4.6andglobal.xai.grok-4.6:supports_prompt_caching: true→falsecache_read_input_token_cost(a cache-read price is meaningless once caching is unsupported, and leaving it keeps the row self-contradictory)Applied identically to
model_prices_and_context_window.jsonand the bundledlitellm/model_prices_and_context_window_backup.json. Native xAI Grok (xai/grok-4.6) is a separate entry and is untouched.Note for maintainers
bedrock_mantle/xai.grok-4.6is also markedsupports_prompt_caching: true(added in the same #37517). It's the same model on the same platform, so it's very likely affected too — but I only reproduced thebedrock_conversepath, so I've left it out of this PR. Happy to include it if you can confirm Bedrock caching is unsupported on the Mantle route as well.Tests
tests/test_litellm/test_bedrock_grok_prompt_caching_metadata.py— data-level (reads the JSON files directly, so it isn't affected by remote-map refresh): assertssupports_prompt_cachingisfalse, nocache_read_input_token_costremains, and the root map and bundled backup agree for these rows. Fails on the pre-fix map, passes with the fix.