Skip to content

fix(model_prices): bedrock Grok 4.6 does not support prompt caching - #38699

Closed
FahimaGold wants to merge 2 commits into
BerriAI:litellm_internal_stagingfrom
FahimaGold:fix/bedrock-grok-no-prompt-caching
Closed

fix(model_prices): bedrock Grok 4.6 does not support prompt caching#38699
FahimaGold wants to merge 2 commits into
BerriAI:litellm_internal_stagingfrom
FahimaGold:fix/bedrock-grok-no-prompt-caching

Conversation

@FahimaGold

Copy link
Copy Markdown
Contributor

Problem

us.xai.grok-4.6 and global.xai.grok-4.6 (litellm_provider: bedrock_converse) are marked supports_prompt_caching: true in the model cost map. AWS Bedrock does not support prompt caching for Grok.

With litellm.enable_anthropic_prompt_caching enabled, AnthropicCacheControlHook.get_default_injection_points treats the flag as authoritative and auto-injects cache_control breakpoints for these models. Bedrock then rejects the entire request:

BedrockException — "You invoked an unsupported model or your request did not allow prompt caching."

The result is a 100% failure rate for Bedrock-hosted Grok 4.6 whenever auto prompt-caching is on. Reproduced live against Bedrock across regions.

The wrong flag came in with #37517 (which added these rows). The offline bundled map lags remote, which is why it isn't always visible in local runs.

Fix

For both us.xai.grok-4.6 and global.xai.grok-4.6:

  • supports_prompt_caching: truefalse
  • remove cache_read_input_token_cost (a cache-read price is meaningless once caching is unsupported, and leaving it keeps the row self-contradictory)

Applied identically to model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json. Native xAI Grok (xai/grok-4.6) is a separate entry and is untouched.

Note for maintainers

bedrock_mantle/xai.grok-4.6 is also marked supports_prompt_caching: true (added in the same #37517). It's the same model on the same platform, so it's very likely affected too — but I only reproduced the bedrock_converse path, so I've left it out of this PR. Happy to include it if you can confirm Bedrock caching is unsupported on the Mantle route as well.

Tests

tests/test_litellm/test_bedrock_grok_prompt_caching_metadata.py — data-level (reads the JSON files directly, so it isn't affected by remote-map refresh): asserts supports_prompt_caching is false, no cache_read_input_token_cost remains, and the root map and bundled backup agree for these rows. Fails on the pre-fix map, passes with the fix.

us.xai.grok-4.6 and global.xai.grok-4.6 (litellm_provider bedrock_converse)
are marked supports_prompt_caching: true, but AWS Bedrock does not support
prompt caching for Grok. With litellm.enable_anthropic_prompt_caching on,
AnthropicCacheControlHook.get_default_injection_points sees the flag as true
and auto-injects cache_control breakpoints; Bedrock then rejects the entire
request:

    You invoked an unsupported model or your request did not allow prompt
    caching.

so every call to these models fails. Set supports_prompt_caching: false and
drop the now-meaningless cache_read_input_token_cost on both rows (added in
BerriAI#37517). Native xAI Grok (xai/grok-4.6) is unaffected. Data kept identical
between the root map and the bundled backup.

Adds tests/test_litellm/test_bedrock_grok_prompt_caching_metadata.py asserting
the flag is false, no cache-read price remains, and the two JSON files agree.
@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR corrects prompt-caching capability and pricing metadata for the two Bedrock Converse Grok 4.6 model identifiers.

  • Disables prompt caching and removes cache-read pricing in both model maps.
  • Relocates the regression coverage into the existing Anthropic cache-control hook test suite.
  • Verifies that both affected models report caching as unsupported and receive no automatic cache-control injection.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains, and the previously reported test-organization issue has been fixed by moving the regression into the existing cache-control hook test module.

Important Files Changed

Filename Overview
model_prices_and_context_window.json Correctly disables prompt caching and removes obsolete cache-read pricing for both Bedrock Converse Grok 4.6 entries.
litellm/model_prices_and_context_window_backup.json Mirrors the root model-map corrections for the bundled fallback metadata.
tests/test_litellm/integrations/test_anthropic_cache_control_hook.py Places the regression test beside the affected hook behavior and covers both model identifiers using the local model-cost map.

Reviews (2): Last reviewed commit: "test: move Bedrock Grok caching regressi..." | Re-trigger Greptile

Comment thread tests/test_litellm/test_bedrock_grok_prompt_caching_metadata.py Outdated
Per CLAUDE.md, a bug fix extends the existing mapped test file rather than
adding a new top-level one. Fold the regression into
TestEnableAnthropicPromptCaching in test_anthropic_cache_control_hook.py as a
behavioural check next to the databricks/non-anthropic siblings: with
enable_anthropic_prompt_caching on, us./global.xai.grok-4.6 report
supports_prompt_caching false and get_default_injection_points returns no
breakpoints. Uses local_model_cost_map so it reads this branch's map, not the
network copy. Drops the standalone test_bedrock_grok_prompt_caching_metadata.py.
@FahimaGold FahimaGold changed the title fix(model_prices): Bedrock Grok 4.6 does not support prompt caching fix(model_prices): bedrock Grok 4.6 does not support prompt caching Aug 28, 2026
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FahimaGold

Copy link
Copy Markdown
Contributor Author

@greptileai review

@codspeed-hq

codspeed-hq Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing FahimaGold:fix/bedrock-grok-no-prompt-caching (fbd65d9) with litellm_internal_staging (b724ebc)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (63dd18c) during the generation of this report, so b724ebc was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Superseded by the rolling registry PR #38693, which re-verified these Bedrock Grok 4.6 caching changes against the AWS docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant