Skip to content

chore(cost-map): remove models past their deprecation date - #42435

Merged
kerry-berri merged 7 commits into
mainfrom
litellm_remove_deprecated_cost_map_entries
Sep 22, 2026
Merged

kerry-berri merged 7 commits into
mainfrom
litellm_remove_deprecated_cost_map_entries

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • 337 cost map entries carry a deprecation_date on or before 2026-09-22
  • Those models no longer serve traffic but still resolve pricing and metadata
  • Unit tests pinned on them mask that the entries are dead

How it solves it:

  • Removes every entry whose deprecation_date is on or before 2026-09-22 from both cost map files
  • Removes the unit tests that only pass because those entries exist
  • Leaves entries deprecating after 2026-09-22 untouched

User Flow

Before: a developer asks the gateway about a model its provider shut down and gets an answer as if it still exists

  1. They send GET https://litellm-domain/model/info with a key that can see gpt-4-0314
  2. The response lists the model with prices and a deprecation_date in the past
  3. They send POST https://litellm-domain/v1/chat/completions with "model": "gpt-4-0314" and the provider returns a 404 model not found

After: the gateway no longer advertises the shut down model

  1. They send GET https://litellm-domain/model/info with the same key
  2. gpt-4-0314 is absent from the list; models with a future deprecation_date are still there
  3. They send POST https://litellm-domain/v1/chat/completions with "model": "gpt-4-0314" and litellm reports the model is not mapped instead of forwarding it

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The change is a pure deletion from a data file, so the proof is the selection itself rather than a live call

Before (merge base)

past-dated entries present

  1. python3 -c "import json;d=json.load(open('model_prices_and_context_window.json'));print(sum(1 for v in d.values() if isinstance(v,dict) and v.get('deprecation_date','9999')<='2026-09-22'))"
  2. 337

After (PR tip)

past-dated entries present

  1. same command
  2. 0

the two cost map files stay aligned

  1. diff <(python3 -c "import json;print('\n'.join(sorted(json.load(open('model_prices_and_context_window.json')))))") <(python3 -c "import json;print('\n'.join(sorted(json.load(open('litellm/model_prices_and_context_window_backup.json')))))")
  2. no output

The JSON diff is deletion only: every + line in the two cost map files is an identical - line shifted by the removed blocks (trailing comma churn on the last key of a run). make check passed at the tip

Type

Refactoring

Caveats (if any)

Medium

  • Anything still routing to one of the 337 removed keys now gets a "model not mapped" cost error instead of a price. The full list is in the deleted JSON blocks
  • Some deleted tests used a removed model only as a fixture for generic behavior (Anthropic and Gemini web search cost, Gemini image cost, passthrough logging cost, savings baseline pricing, router moderation routing, DALL-E routing). Repointing those at supported models would restore the coverage; the decision for this PR is to remove them, and the same applies to the two Gemini image cost tests that landed on main mid review and were pinned on a removed model

Low

  • Source code still references a few removed identifiers as default literals (for example dall-e-2 in litellm/__init__.py, command-r in litellm/utils.py). They are not touched here since nothing fails because of them
  • Live and network test directories could not be run in the sandbox, only the unit test tree

Link to Devin session: https://app.devin.ai/sessions/0a48cb98f12b4822bbe2ffcd2e51942a
Open in Devin Desktop: https://app.devin.ai/desktop/session/0a48cb98f12b4822bbe2ffcd2e51942a?variant=devin


Note

Medium Risk
Large-scale deletion from the canonical pricing map changes behavior for any caller still using the 337 removed model IDs (no price, not advertised), and removes several unit tests rather than repointing them.

Overview
Purges 337 model entries from the bundled cost maps (model_prices_and_context_window.json and the backup copy) whose deprecation_date is on or before 2026-09-22, so the gateway stops advertising pricing and metadata for provider-shut-down models. Models with a future deprecation date are unchanged.

GET /model/info and related resolution no longer list keys like gpt-4-0314; POST /v1/chat/completions (and similar) against those names fail earlier with a model not mapped / cost-map error instead of forwarding to the provider and getting a 404.

The PR also drops unit tests that depended on the removed keys (including some that only used them as fixtures for web-search, image, routing, and savings tests). It does not update leftover default literals in source (e.g. dall-e-2, command-r) that still reference removed identifiers.

Reviewed by Cursor Bugbot for commit c51b772. Bugbot is set up for automated code reviews on this repo. Configure here.

kerry and others added 2 commits September 22, 2026 06:12
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… web search removal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot requested a review from a team September 22, 2026 06:16
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codspeed

codspeed Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_remove_deprecated_cost_map_entries (c51b772) with main (b73696a)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (cd69342) during the generation of this report, so b73696a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@greptile-apps

greptile-apps Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

No new blocking defect was identified in the changes made since the previous review.

Summary

Removes model-pricing and metadata entries whose deprecation dates are on or before 2026-09-22 from both cost maps.

  • Keeps the primary and backup cost-map files aligned.
  • Removes or narrows tests that depended on the deleted model entries.
  • Incorporates newer cost-map metadata while retaining entries with future deprecation dates.

Reviews (4) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."

@greptile-apps

greptile-apps Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

  • P1 Default Model Loses Mapping model_prices_and_context_window.json:15782 ▶

    Removing the dall-e-2 map entry breaks existing callers that omit the image model: litellm/images/main.py still assigns dall-e-2 by default, DEFAULT_IMAGE_ENDPOINT_MODEL remains dall-e-2, and Azure cost calculation normalizes an empty image model to the same identifier. These normal paths now reach metadata or cost lookup with a deleted key. This violates the repository directive to avoid backwards-incompatible changes without a user-controlled flag; update the defaults or retain a compatible mapping.

  • P1 Active Models Are Removed model_prices_and_context_window.json:77584 ▶

    This pruning removes openrouter/nex-agi/nex-n2.5-mini and openrouter/nex-agi/nex-n2.5-pro even though neither entry has a deprecation_date, causing valid models to disappear from pricing and metadata lookup. It also strips supports_anthropic_compaction from thirteen surviving Claude entries and removes the future 2026-09-25 dates from two OpenRouter free variants. Please apply the cutoff structurally so that only entries dated on or before it are changed.

  • P2 Generic Regression Coverage Removed tests/test_litellm/litellm_core_utils/llm_cost_calc/test_tool_call_cost_tracking.py:234 ▶

    These deletions remove tests for behavior that remains supported rather than only removing assertions tied to deprecated models. The raw Anthropic response tests verify per-request web-search charges when normalized Usage is missing or drops server_tool_use; the Gemini test file likewise loses token-detail image-cost coverage. No equivalent tests remain, and these cases can use an active model or local pricing data. This violates the repository requirement that modifications to existing tests must not weaken coverage, so the coverage must be restored before merging.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

kerry and others added 2 commits September 22, 2026 19:14
…ated_cost_map_entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	model_prices_and_context_window.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

kerry and others added 2 commits September 22, 2026 19:23
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ated_cost_map_entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	tests/test_litellm/proxy/spend_tracking/test_budget_reservation.py

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…ated_cost_map_entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	model_prices_and_context_window.json

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit c51b772. Configure here.

@kerry-berri
kerry-berri merged commit 075536e into main Sep 22, 2026
98 checks passed
zkysar1 added a commit to zkysar1/Zak-Code that referenced this pull request Sep 22, 2026
…dels (#623)

At 21:19Z on 2026-09-22 litellm removed models past their deprecation date
from the model map on its GitHub main (BerriAI/litellm#42435). litellm
fetches that file at import, so four tests that built a provider for
groq/qwen/qwen3-32b or groq/llama-3.3-70b-versatile with no context window
began raising UnknownContextWindow. No Zak Code change was involved: main's
last CI ran at 19:40Z, before the upstream commit, and was green.

Each test now declares the window its model had in litellm's bundled map
(131000 and 128000), the idiom the neighbouring tests already use for
self-hosted aliases. The Groq names stay: they are witnesses for "a metered
cloud", and classification reads the provider prefix, not metadata.

Measured: before, 4 failed / 37 passed in the two files against the live map;
after, 41 passed against the live map and 41 passed with
LITELLM_LOCAL_MODEL_COST_MAP=True; full suite 4555 passed, 0 failed.

Not fixed here: pinning the bundled map for the whole suite fails 26 other
tests (gpt-5.6, Gemini 3), so the suite's verdict still follows the live map.

Co-authored-by: fleet-agent <fleet-agent@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mateo-berri added a commit that referenced this pull request Sep 22, 2026
…t a live gemini row (#42615)

The test priced gemini/gemini-3.1-flash-image-preview, which #42435 removed
from the cost map as deprecated, so the calculator had no per-token rates to
keep and the hardcoded expected value no longer matched. Price the live
gemini/gemini-3.1-flash-image row instead and derive the expected cost from
that row in litellm.model_cost, so a rate change on it cannot break the test
while a calculator that drops the map's token rates still fails it.

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
mateo-berri pushed a commit that referenced this pull request Sep 23, 2026
)

* test: point CircleCI-only suites at models still in the cost map

#42435 removed cost map entries past their deprecation date and #42437 added
litellm_uisettings to the config-synced tables, but both only updated
tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation,
logging_callback_tests, litellm_utils_tests, unit) kept using the removed
models or the old table list and went red on main.

Each test keeps its assertions and swaps the removed model for a current one
with the same provider and capabilities. The fireworks tests pick a vision
model from the cost map because #34941 set supports_vision false on
minimax-m3, and the vertex image provider test injects the image model set
because #42435 removed every vertex_ai-image-models entry.

* test(vertex_ai): register the image model through add_known_models in the provider test
mateo-berri added a commit that referenced this pull request Sep 23, 2026
#42435 removed the OpenAI dall-e-3 cost-map row that
test_dalle_3_azure_cost_tracking pinned, and Azure retired DALL-E 3
inference on 2026-03-04, so the test now mirrors a real Azure
gpt-image-2 response and derives the expected cost from the cost map.

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
yuneng-berri added a commit that referenced this pull request Sep 23, 2026
* fix: repair seven regressions caught by CircleCI on main

- vertex_ai: stop treating fine-tuned endpoint ids (numeric or
  vertex_ai/gemini/<id>) and gemma models as Gemini 3+, which injected
  temperature=1.0 and Gemini 3 thinking config into their requests (#42465)
- cost: price Azure DALL-E 3 from its azure/<quality>/<size>/dall-e-3 rows;
  it only worked through the OpenAI rows that #42435 removed
- bedrock: stream bedrock/invoke/moonshot through an OpenAI-shaped chunk
  decoder; the generic decoder dropped every chunk, which the
  supports_response_schema flag from #42338 un-skipped in CI
- proxy: keep the public model_group on pre-routing rejections so the Usage
  page groups them under the model name, not the deployment (#41077)
- cost map: mirror the base rows' capability flags onto Bedrock regional and
  cross-region copies (#42254 and later syncs)
- whitelist the new regional Bedrock rows from #42543 and #42588 for the
  converse routing check, following the existing regional-row convention

* fix(model-prices): mirror capability flags onto ap-southeast-3 bedrock rows

* refactor(bedrock): tighten types on the moonshot stream decoder and its tests
avivsinai added a commit to avivsinai/litellm-rust that referenced this pull request Oct 9, 2026
)

* fix(registry): let reviewed retirements lower the shrink high-water

Upstream BerriAI/litellm#42435 removed models past their deprecation dates,
taking xai from 78 to 58 entries, and the per-provider high-water guard
rejected every daily refresh since. Entries that our trusted committed data
already marks with a past deprecation_date and that upstream drops are now
subtracted from the high-water baseline before the 75% floor check, and the
written baseline uses that lowered value. Any other disappearance still
counts as unexplained loss.

Fixes #70

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: refresh model registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant