-
-
Notifications
You must be signed in to change notification settings - Fork 11.1k
fix(model_prices): annotate retired xAI grok-2-era models and mark grok-4.20-multi-agent Responses-only #38526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
ben7am1n
wants to merge
12
commits into
BerriAI:litellm_internal_staging
from
ben7am1n:fix/xai-retired-models
Closed
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
10798ca
Merge pull request #36286 from BerriAI/litellm_internal_staging
yuneng-berri 6a919ae
Merge pull request #36304 from BerriAI/litellm_internal_staging
yuneng-berri 0e9cd98
Merge pull request #36560 from BerriAI/litellm_internal_staging
yuneng-berri bd0d135
Merge pull request #36725 from BerriAI/litellm_internal_staging
yuneng-berri bc6e7df
Merge pull request #37042 from BerriAI/litellm_internal_staging
yuneng-berri 007bd43
Merge pull request #37400 from BerriAI/litellm_internal_staging
yuneng-berri ef1cde4
fix: add moonshot/kimi-k3 to the cost map
mateo-berri 24555ac
Merge pull request #37753 from BerriAI/litellm_hotfix_kimi_k3_cost_map
mateo-berri 418c7c6
Merge pull request #37721 from BerriAI/litellm_internal_staging
yuneng-berri 947dbbf
Merge pull request #37913 from BerriAI/litellm_internal_staging
yuneng-berri 6e569ee
Merge pull request #38293 from BerriAI/litellm_internal_staging
yuneng-berri ca78d54
fix(model_prices): annotate retired xAI grok-2-era models and mark gr…
ben7am1n File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| import json | ||
| from pathlib import Path | ||
|
|
||
| # https://github.com/BerriAI/litellm/issues/38179 | ||
| # grok-2-era slugs: xAI deprecated them effective 2026-02-28 (same batch date as | ||
| # the pre-existing xai/grok-2-vision-1212 annotation) and they now hard-fail with | ||
| # "Model not found — retired by xAI". | ||
| RETIRED_GROK2_MODELS = [ | ||
| "xai/grok-2", | ||
| "xai/grok-2-1212", | ||
| "xai/grok-2-latest", | ||
| "xai/grok-2-vision", | ||
| "xai/grok-2-vision-latest", | ||
| "xai/grok-beta", | ||
| "xai/grok-vision-beta", | ||
| ] | ||
| GROK2_DEPRECATION_DATE = "2026-02-28" | ||
|
|
||
| # Per https://docs.x.ai/developers/model-capabilities/text/multi-agent (Limitations): | ||
| # "The multi-agent model does not work with the OpenAI Chat Completions API." | ||
| RESPONSES_ONLY_MODELS = [ | ||
| "xai/grok-4.20-multi-agent-0309", | ||
| "xai/grok-4.20-multi-agent-beta-0309", | ||
| ] | ||
|
|
||
| # Slugs still served by xAI (https://docs.x.ai/developers/models) that must not | ||
| # be marked deprecated. | ||
| ACTIVE_GROK_MODELS = [ | ||
| "xai/grok-4.5", | ||
| "xai/grok-4.6", | ||
| "xai/grok-4.20-0309-reasoning", | ||
| ] | ||
|
|
||
|
|
||
| def _load_model_cost(path: Path) -> dict: | ||
| with open(path) as f: | ||
| return json.load(f) | ||
|
|
||
|
|
||
| def test_retired_grok2_models_are_annotated(): | ||
| json_path = Path(__file__).parents[2] / "model_prices_and_context_window.json" | ||
| model_cost = _load_model_cost(json_path) | ||
|
|
||
| for model in RETIRED_GROK2_MODELS: | ||
| info = model_cost.get(model) | ||
| assert info is not None, f"{model} not found in model_prices_and_context_window.json" | ||
| assert ( | ||
| info.get("deprecation_date") == GROK2_DEPRECATION_DATE | ||
| ), f"{model} should carry deprecation_date {GROK2_DEPRECATION_DATE}" | ||
|
|
||
|
|
||
| def test_grok_4_20_multi_agent_is_responses_only(): | ||
| json_path = Path(__file__).parents[2] / "model_prices_and_context_window.json" | ||
| model_cost = _load_model_cost(json_path) | ||
|
|
||
| for model in RESPONSES_ONLY_MODELS: | ||
| info = model_cost.get(model) | ||
| assert info is not None, f"{model} not found in model_prices_and_context_window.json" | ||
| assert ( | ||
| info.get("mode") == "responses" | ||
| ), f"{model} only works with the xAI Responses API, not Chat Completions" | ||
|
|
||
|
|
||
| def test_active_grok_models_are_not_marked_deprecated(): | ||
| json_path = Path(__file__).parents[2] / "model_prices_and_context_window.json" | ||
| model_cost = _load_model_cost(json_path) | ||
|
|
||
| for model in ACTIVE_GROK_MODELS: | ||
| info = model_cost.get(model) | ||
| assert info is not None, f"{model} not found in model_prices_and_context_window.json" | ||
| assert ( | ||
| "deprecation_date" not in info | ||
| ), f"{model} is still served by xAI and must not be marked deprecated" | ||
|
|
||
|
|
||
| def test_xai_grok_metadata_backup_matches_main(): | ||
| repo_root = Path(__file__).parents[2] | ||
| main_path = repo_root / "model_prices_and_context_window.json" | ||
| backup_path = repo_root / "litellm" / "model_prices_and_context_window_backup.json" | ||
|
|
||
| main_cost = _load_model_cost(main_path) | ||
| backup_cost = _load_model_cost(backup_path) | ||
|
|
||
| for model in RETIRED_GROK2_MODELS + RESPONSES_ONLY_MODELS: | ||
| assert backup_cost.get(model) == main_cost.get(model), f"{model} differs between main and backup" | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This bug fix creates a separate test module whose autouse fixture replaces
litellm.model_cost, although every assertion reads the JSON files directly. This bypasses the mapped-test convention and adds global-state and cache-isolation risk without contributing to the regression coverageContext Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!