Add supported providers to prompt caching doc - #26124
Conversation
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Greptile SummaryThis PR adds a structured supported-providers list to the prompt caching docs and marks all current-generation xAI models ( Confidence Score: 5/5Safe to merge — all changes are documentation and JSON metadata with no runtime logic impact. All P0/P1 concerns from prior reviews (missing grok-4 flags) are resolved. The sole remaining comment is a P2 doc-clarity suggestion about specifying which xAI model generation supports caching, which does not block merge. docs/my-website/docs/tutorials/prompt_caching.md — minor wording could be tightened to clarify grok-3+ generation scope
|
| Filename | Overview |
|---|---|
| docs/my-website/docs/tutorials/prompt_caching.md | Added two provider lists (cache_control and provider-managed); xAI correctly placed as provider-managed, but the blanket "xai/" claim doesn't match the fact that grok-2/grok-beta models don't carry the supports_prompt_caching flag |
| docs/my-website/docs/completion/prompt_caching.md | Added xAI to the supported-providers list; straightforward and accurate |
| model_prices_and_context_window.json | Added supports_prompt_caching: true to all current-generation xAI models (grok-3, grok-4, grok-4.20, grok-code, grok-4-1 families); older models (grok-2, grok-beta) intentionally excluded, consistent with their lack of cache_read_input_token_cost |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[LiteLLM Prompt Caching] --> B{Caching Mechanism}
B --> C[cache_control Marker Required]
B --> D[Provider Managed - Automatic]
C --> C1[anthropic/]
C --> C2[bedrock/]
C --> C3[vertex_ai/]
C --> C4[gemini/]
C --> C5[azure_ai/]
C --> C6[openrouter/]
C --> C7[databricks/]
C --> C8[dashscope/]
C --> C9[minimax/]
C --> C10[zai/]
D --> D1[openai/]
D --> D2[deepseek/]
D --> D3["xai/ NEW - grok-3, grok-4, grok-4.20, grok-code families"]
Reviews (5): Last reviewed commit: "Add prompt caching flag to grok-4, grok-..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| - OpenAI (`openai/`) | ||
| - DeepSeek (`deepseek/`) | ||
| - xAI (`xai/`) | ||
| - Z.ai / GLM (`zai/`) |
There was a problem hiding this comment.
zai/ overrides remove_cache_control_flag_from_messages_and_tools to preserve cache_control markers rather than strip them, which means it behaves exactly like DashScope and MiniMax — i.e., it requires an explicit cache_control marker. Listing it under "Provider Managed (automatic, no marker needed)" will mislead users who expect caching to work without adding a marker.
| - Z.ai / GLM (`zai/`) | |
| - Z.ai / GLM (`zai/`) |
It should be moved up to the cache_control marker section alongside DashScope and MiniMax.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
^ tested for the grok-4x models |
|
lgtm |
4f823ce
into
litellm_internal_staging
* Add supported providers to prompt caching doc * Move Z.ai / GLM to cache_control marker list * Mark xAI models as supporting prompt caching * Narrow xAI prompt caching flag to models with documented cache pricing * Add prompt caching flag to grok-4, grok-4-0709, grok-4-latest --------- Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
4 participants
Relevant issues
Updates the doc to add a list of supported providers for prompt caching, where previously that was unclear. Also, changes some JSON values for xAI models since I went and tested it, and they support it on their end.
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
A screenshot of the updated doc page. Also, I tested a few of the providers and read through the respective code / provider docs to double-check these were accurate.
Type
📖 Documentation
Changes
Doc page under docs/tutorials/prompt_caching