Repository navigation
chore(cost-map): remove models past their deprecation date - #42435
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… web search removal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
|
|
Comments Outside DiffThese findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ated_cost_map_entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # litellm/model_prices_and_context_window_backup.json # model_prices_and_context_window.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ated_cost_map_entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # tests/test_litellm/proxy/spend_tracking/test_budget_reservation.py
…ated_cost_map_entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # litellm/model_prices_and_context_window_backup.json # model_prices_and_context_window.json
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c51b772. Configure here.
…dels (#623) At 21:19Z on 2026-09-22 litellm removed models past their deprecation date from the model map on its GitHub main (BerriAI/litellm#42435). litellm fetches that file at import, so four tests that built a provider for groq/qwen/qwen3-32b or groq/llama-3.3-70b-versatile with no context window began raising UnknownContextWindow. No Zak Code change was involved: main's last CI ran at 19:40Z, before the upstream commit, and was green. Each test now declares the window its model had in litellm's bundled map (131000 and 128000), the idiom the neighbouring tests already use for self-hosted aliases. The Groq names stay: they are witnesses for "a metered cloud", and classification reads the provider prefix, not metadata. Measured: before, 4 failed / 37 passed in the two files against the live map; after, 41 passed against the live map and 41 passed with LITELLM_LOCAL_MODEL_COST_MAP=True; full suite 4555 passed, 0 failed. Not fixed here: pinning the bundled map for the whole suite fails 26 other tests (gpt-5.6, Gemini 3), so the suite's verdict still follows the live map. Co-authored-by: fleet-agent <fleet-agent@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t a live gemini row (#42615) The test priced gemini/gemini-3.1-flash-image-preview, which #42435 removed from the cost map as deprecated, so the calculator had no per-token rates to keep and the hardcoded expected value no longer matched. Price the live gemini/gemini-3.1-flash-image row instead and derive the expected cost from that row in litellm.model_cost, so a rate change on it cannot break the test while a calculator that drops the map's token rates still fails it. Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
) * test: point CircleCI-only suites at models still in the cost map #42435 removed cost map entries past their deprecation date and #42437 added litellm_uisettings to the config-synced tables, but both only updated tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation, logging_callback_tests, litellm_utils_tests, unit) kept using the removed models or the old table list and went red on main. Each test keeps its assertions and swaps the removed model for a current one with the same provider and capabilities. The fireworks tests pick a vision model from the cost map because #34941 set supports_vision false on minimax-m3, and the vertex image provider test injects the image model set because #42435 removed every vertex_ai-image-models entry. * test(vertex_ai): register the image model through add_known_models in the provider test
#42435 removed the OpenAI dall-e-3 cost-map row that test_dalle_3_azure_cost_tracking pinned, and Azure retired DALL-E 3 inference on 2026-03-04, so the test now mirrors a real Azure gpt-image-2 response and derives the expected cost from the cost map. Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: repair seven regressions caught by CircleCI on main - vertex_ai: stop treating fine-tuned endpoint ids (numeric or vertex_ai/gemini/<id>) and gemma models as Gemini 3+, which injected temperature=1.0 and Gemini 3 thinking config into their requests (#42465) - cost: price Azure DALL-E 3 from its azure/<quality>/<size>/dall-e-3 rows; it only worked through the OpenAI rows that #42435 removed - bedrock: stream bedrock/invoke/moonshot through an OpenAI-shaped chunk decoder; the generic decoder dropped every chunk, which the supports_response_schema flag from #42338 un-skipped in CI - proxy: keep the public model_group on pre-routing rejections so the Usage page groups them under the model name, not the deployment (#41077) - cost map: mirror the base rows' capability flags onto Bedrock regional and cross-region copies (#42254 and later syncs) - whitelist the new regional Bedrock rows from #42543 and #42588 for the converse routing check, following the existing regional-row convention * fix(model-prices): mirror capability flags onto ap-southeast-3 bedrock rows * refactor(bedrock): tighten types on the moonshot stream decoder and its tests
) * fix(registry): let reviewed retirements lower the shrink high-water Upstream BerriAI/litellm#42435 removed models past their deprecation dates, taking xai from 78 to 58 entries, and the per-provider high-water guard rejected every daily refresh since. Entries that our trusted committed data already marks with a past deprecation_date and that upstream drops are now subtracted from the high-water baseline before the 75% floor check, and the written baseline uses that lowered value. Any other disappearance still counts as unexplained loss. Fixes #70 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: refresh model registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
TLDR
Problem this solves:
deprecation_dateon or before 2026-09-22How it solves it:
deprecation_dateis on or before 2026-09-22 from both cost map filesUser Flow
Before: a developer asks the gateway about a model its provider shut down and gets an answer as if it still exists
gpt-4-0314deprecation_datein the past"model": "gpt-4-0314"and the provider returns a 404 model not foundAfter: the gateway no longer advertises the shut down model
gpt-4-0314is absent from the list; models with a futuredeprecation_dateare still there"model": "gpt-4-0314"and litellm reports the model is not mapped instead of forwarding itRelevant issues
Affected release
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
The change is a pure deletion from a data file, so the proof is the selection itself rather than a live call
Before (merge base)
past-dated entries present
python3 -c "import json;d=json.load(open('model_prices_and_context_window.json'));print(sum(1 for v in d.values() if isinstance(v,dict) and v.get('deprecation_date','9999')<='2026-09-22'))"337After (PR tip)
past-dated entries present
0the two cost map files stay aligned
diff <(python3 -c "import json;print('\n'.join(sorted(json.load(open('model_prices_and_context_window.json')))))") <(python3 -c "import json;print('\n'.join(sorted(json.load(open('litellm/model_prices_and_context_window_backup.json')))))")The JSON diff is deletion only: every
+line in the two cost map files is an identical-line shifted by the removed blocks (trailing comma churn on the last key of a run).make checkpassed at the tipType
Refactoring
Caveats (if any)
Medium
Low
dall-e-2inlitellm/__init__.py,command-rinlitellm/utils.py). They are not touched here since nothing fails because of themLink to Devin session: https://app.devin.ai/sessions/0a48cb98f12b4822bbe2ffcd2e51942a
Open in Devin Desktop: https://app.devin.ai/desktop/session/0a48cb98f12b4822bbe2ffcd2e51942a?variant=devin
Note
Medium Risk
Large-scale deletion from the canonical pricing map changes behavior for any caller still using the 337 removed model IDs (no price, not advertised), and removes several unit tests rather than repointing them.
Overview
Purges 337 model entries from the bundled cost maps (
model_prices_and_context_window.jsonand the backup copy) whosedeprecation_dateis on or before 2026-09-22, so the gateway stops advertising pricing and metadata for provider-shut-down models. Models with a future deprecation date are unchanged.GET /model/infoand related resolution no longer list keys likegpt-4-0314;POST /v1/chat/completions(and similar) against those names fail earlier with a model not mapped / cost-map error instead of forwarding to the provider and getting a 404.The PR also drops unit tests that depended on the removed keys (including some that only used them as fixtures for web-search, image, routing, and savings tests). It does not update leftover default literals in source (e.g.
dall-e-2,command-r) that still reference removed identifiers.Reviewed by Cursor Bugbot for commit c51b772. Bugbot is set up for automated code reviews on this repo. Configure here.