fix(model_cost): resync backup cost map with canonical and guard against drift - #34527
fix(model_cost): resync backup cost map with canonical and guard against drift#34527onatozmenn wants to merge 1 commit into
Conversation
…nst drift
The two shipped cost maps had drifted. `model_prices_and_context_window.json` is
the canonical map; `litellm/model_prices_and_context_window_backup.json` is bundled
into the package and is what gets used when the remote fetch fails or when
LITELLM_LOCAL_MODEL_COST_MAP=True. Users were silently getting different model
metadata depending on which one loaded:
- gpt-5.4-mini / gpt-5.4-nano (and dated aliases): max_input_tokens 1050000 vs
272000. The sibling gpt-5.4 entry uses 1050000, so the backup was carrying a
stale gpt-5-generation value.
- gpt-5-pro / gpt-5-pro-2025-10-06: max_input_tokens 400000 vs 128000 and
max_output_tokens 128000 vs 272000. The backup values are transposed - its
input window was smaller than the base gpt-5 entry's 272000.
- github_copilot/claude-{haiku,opus,sonnet}-4.5: supported_endpoints was missing
/v1/messages in the canonical map. Here the backup was the correct side;
0b0fd6a (BerriAI#31802) only updated the backup.
- snowflake/claude-sonnet-4-6: supports_adaptive_thinking missing from the backup;
98ced0a only updated the canonical map.
- Five entries absent from the backup entirely: gemini-3-pro-image,
gemini-3.1-flash-image, openrouter/z-ai/glm-5.1, zai/glm-4.7-flash, zai/glm-5.1.
Drift ran in both directions, so this resolves each divergence against the commit
that introduced it rather than blindly copying one file over the other. The
/v1/messages endpoints are added to the canonical map first; everything else is
already correct there, so the backup is then regenerated from it via the existing
ci_cd/check_files_match.py.
ci_cd/check_files_match.py was meant to prevent exactly this, but it is referenced
nowhere: it used to run as a pre-commit hook and 51af6fe removed
.pre-commit-config.yaml when hooks moved to `make pre-commit`, without carrying it
over. The existing per-model checks (e.g. test_gpt_5_6_backup_matches_main) only
assert the specific models each was written for, so nothing caught this.
Adds tests/test_litellm/test_model_cost_map_sync.py, which asserts the invariant
globally in CI and reports the actual divergences on failure.
Signed-off-by: onatozmenn <onatozmen44@gmail.com>
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummarySynchronizes the bundled model cost map with the canonical map and adds regression tests that enforce semantic and byte-for-byte equality.
Confidence Score: 5/5The PR appears safe to merge, with the two shipped cost maps synchronized and guarded against future drift. The changed maps are byte-identical, their metadata updates align the canonical and fallback loading paths, and the new tests enforce both semantic and formatting consistency without introducing network or runtime dependencies.
|
| Filename | Overview |
|---|---|
| litellm/model_prices_and_context_window_backup.json | Resynchronizes fallback model metadata and entries with the canonical map; no actionable issue found. |
| model_prices_and_context_window.json | Adds the Anthropic Messages endpoint to three GitHub Copilot Claude model definitions, matching the bundled map. |
| tests/test_litellm/test_model_cost_map_sync.py | Adds correctly rooted semantic and byte-level equality checks that reflect the existing map synchronization contract. |
Reviews (1): Last reviewed commit: "fix(model_cost): resync backup cost map ..." | Re-trigger Greptile
|
Closing this one, since the data half has gone stale. When I opened it, the backup map needed 171 added and 46 removed lines to line up with the canonical file. That snapshot is two and a half weeks old now. Comparing the two files on current The part I'd still argue for is |
2 participants
TLDR
Problem this solves:
How it solves it:
Relevant issues
None open for this — found while auditing the cost maps.
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)On the CI checkbox: the only red check is
osv-scan, and it fails independently of this PR. It scansuv.lockandui/litellm-dashboard/package-lock.json; this branch touches neither, and both are byte-identical to the base branch. Running the same osv-scanner v2.3.8 command locally reports 5 advisories against the already-pinnedgitpython3.1.52 (GHSA-3rp5-jjmw-4wv2, GHSA-6p8h-3wgx-97gf, GHSA-fjr4-x663-mwxc, GHSA-r9mr-m37c-5fr3) andpostcss8.5.13 (GHSA-r28c-9q8g-f849). It was still green on PRs that ran a couple of hours before this one, so the advisories look freshly published.guard-fork-dependencies.ymlblocks fork PRs from touching lockfiles, so the bump (or anosv-scanner.tomlentry) has to happen in the canonical repo.Screenshots / Proof of Fix
LITELLM_LOCAL_MODEL_COST_MAP=Truemakes litellm load the bundled backup instead of the canonical map, so the drift is observable as real library behaviour. This readslitellm.model_costat runtime and compares it against the canonical file in the repo — no mocks, no test framework.Before, at base commit
f6a1050cbf:After, at
94897ad8a8:The remaining 2983 vs 2984 gap is
fallback_generalizations, a config key rather than a model, whichlitellm.model_costexcludes by design.Supporting test runs:
pytest tests/test_litellm/test_model_cost_map_sync.py— 2 passed. Both fail atf6a1050cbf, so they are load-bearing.pytest tests/test_litellm/llms/openai/test_gpt5_transformation.py tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py tests/test_litellm/completion_extras/test_litellm_responses_transformation_transformation.py tests/test_litellm/test_gpt_5_6_model_metadata.py tests/test_litellm/test_gpt_5_5_model_metadata.py tests/test_litellm/test_deepseek_model_metadata.py— 235 passed.python ci_cd/check_files_match.py— exits 0.Type
🐛 Bug Fix
Changes
model_prices_and_context_window.jsonis the canonical map.litellm/model_prices_and_context_window_backup.jsonis bundled into the package and is what gets used when the remote fetch fails orLITELLM_LOCAL_MODEL_COST_MAP=True. They had drifted, so model metadata depended on which one happened to load.The divergences, and which side each was resolved to:
gpt-5.4-mini/-nano(+dated)max_input_tokensgpt-5.4uses 1050000; the backup's 272000 is a stale gpt-5-generation valuegpt-5-pro(+dated)max_input_tokens,max_output_tokens,max_tokensgpt-5's 272000github_copilot/claude-{haiku,opus,sonnet}-4.5supported_endpoints0b0fd6a4d1(#31802) added/v1/messagesto the backup onlysnowflake/claude-sonnet-4-6supports_adaptive_thinking98ced0ae43added the flag to the canonical map onlyThe drift runs in both directions — 124 commits touched only the canonical map and 124 touched only the backup — so this doesn't just copy one file over the other. The
/v1/messagesendpoints are added to the canonical map first, since that's the one place the backup was ahead; everything else is already correct there, so the backup is then regenerated from it with the existingci_cd/check_files_match.py.Because every value here already exists in the canonical map, callers on the normal remote-fetch path see no change at all. What changes is that the bundled-backup path now agrees with them.
Why it went unnoticed:
ci_cd/check_files_match.pyexists to enforce exactly this, but nothing references it —git grep check_files_matchreturns nothing. It ran as a pre-commit hook until51af6fedb3removed.pre-commit-config.yamlwhen hooks moved tomake pre-commit, and it wasn't carried over. The existing per-model checks liketest_gpt_5_6_backup_matches_mainonly assert the specific models each was written for, so none of them could catch drift elsewhere.tests/test_litellm/test_model_cost_map_sync.pyasserts the invariant globally instead, and prints the actual divergences on failure rather than a bare byte-compare.One note for reviewers: I resolved
gpt-5-proand thegpt-5.4minis from family conventions in the map itself, since neither value had a commit message explaining it. If you have the authoritative numbers, those two rows are the ones worth a second look.Final Attestation
Investigated and prepared with AI assistance.