test: drop cost-map mirror tests

[ryan-crabbe-berri]

TLDR

Problem this solves:

• 39 tests just restate numbers from the cost map
• Only a JSON edit can break them, never a code change
• Each one is a second place you must edit
• They also drift against the network-fetched map

How it solves it:

• Delete the 39 that are pure data mirrors
• Keep every test that exercises resolution, hydration or billing code
• Drop the constants and one file left with nothing

User Flow

No user flow changes. This is a test-only PR: no runtime code, no config, and no registry values are touched, so nothing an end user sends or sees is different before and after

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

There is no live proxy surface here, so the proof is that every deleted assertion was already a copy of a value the repo ships. The check below reads model_prices_and_context_window.json and prints what four of the deleted tests asserted next to what the registry actually carries

Before (b97bc10)

1. Pick four of the deleted tests and read the value each one pinned:

$ python3 -c "
import json
m = json.load(open('model_prices_and_context_window.json'))
for key, field in [('gemini-3.7-flash','input_cost_per_token'),
                   ('cloudflare/@cf/meta/llama-3.1-8b-instruct-fp8','rpm'),
                   ('mistral/mistral-ocr-2512','ocr_cost_per_page'),
                   ('xai/grok-4.6','input_cost_per_token')]:
    print(f'{key}.{field} = {m[key][field]}')
"
gemini-3.7-flash.input_cost_per_token = 7.5e-07
cloudflare/@cf/meta/llama-3.1-8b-instruct-fp8.rpm = 300
mistral/mistral-ocr-2512.ocr_cost_per_page = 0.002
xai/grok-4.6.input_cost_per_token = 2e-06

2. Compare against what the tests asserted, straight out of the diff:

$ git show b97bc10ec9:tests/test_litellm/llms/mistral/ocr/test_mistral_ocr_cost.py | sed -n '80,83p'
def test_ocr3_model_info_price(local_model_cost_map) -> None:
    info = litellm.get_model_info(model=OCR3_MODEL, custom_llm_provider="mistral")
    assert info["ocr_cost_per_page"] == OCR3_COST_PER_PAGE

OCR3_MODEL is mistral/mistral-ocr-2512 and OCR3_COST_PER_PAGE is 0.002, the same key and the same number the registry ships. The lookup name is the literal JSON key, so no resolution runs and there is no code between the file and the assert. The test directly above it in the same file already opens the JSON and asserts that field, so the deleted one was also a duplicate

3. The same holds for the other 38: each reads an exact registry key through model_cost[key] or get_model_info("<that same key>") and asserts a literal it already contains

After (96c02d5)

1. The registry still ships every one of those values, unchanged:

$ python3 -c "
import json
m = json.load(open('model_prices_and_context_window.json'))
for key, field in [('gemini-3.7-flash','input_cost_per_token'),
                   ('cloudflare/@cf/meta/llama-3.1-8b-instruct-fp8','rpm'),
                   ('mistral/mistral-ocr-2512','ocr_cost_per_page'),
                   ('xai/grok-4.6','input_cost_per_token')]:
    print(f'{key}.{field} = {m[key][field]}')
"
gemini-3.7-flash.input_cost_per_token = 7.5e-07
cloudflare/@cf/meta/llama-3.1-8b-instruct-fp8.rpm = 300
mistral/mistral-ocr-2512.ocr_cost_per_page = 0.002
xai/grok-4.6.input_cost_per_token = 2e-06

2. Nothing that reads those values changed, so pricing a request is identical before and after

Type

🧹 Refactoring
✅ Test

Caveats

Low

• One file, the Fireworks Kimi metadata test, is gone entirely
• Its only test was a mirror, leaving just a fixture
• A bad bulk edit to those JSON values now fails later, at the cost-calculation tests instead of at a dedicated pin

---

Note

Low Risk
Deletes redundant tests only; production pricing and model metadata behavior are unchanged, with slightly less guard against silent JSON drift on removed pins.

Overview
Test-only cleanup: removes roughly 39 assertions (and one whole file) that only re-read litellm.model_cost / get_model_info and compare against literals already in model_prices_and_context_window.json.

What goes away: launch-pricing pins (Gemini 3.x, Grok 4.6, Azure GPT-5.5/5.6, Baseten/Wandb/OpenRouter entries), alias parity checks (gpt-5.6 vs sol), reasoning-effort flag mirrors, Databricks cache-field sweeps, OCR get_model_info duplicates, Fireworks Kimi max-output pins, Cloudflare RPM, gpt-image-2 metadata, Anthropic re-export prompt_cache_min_tokens map scans, and similar.

What stays: tests that still drive billing math (generic_cost_per_token, completion_cost), routing/hydration behavior, JSON file reads where they’re the primary check, and parametrize blocks that assert computed costs—not static map copies.

No runtime, config, or registry JSON changes in this diff.

^{Reviewed by Cursor Bugbot for commit 96c02d5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[ryan-crabbe-berri]

bugbot run

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This test-only PR removes cost-map assertions, constants, and one test file

• Most deleted assertions directly duplicated registry values while runtime billing and routing tests remain
• Some deleted regression and cross-entry invariant tests provided independent protection against token-limit and pricing drift

Confidence Score: 4/5

The runtime is unchanged, but the weakened regression coverage should be restored or replaced before merging to satisfy the repository requirement

Most removals are harmless mirrors, but deleting independent Fireworks limit and GPT-5.6 alias-parity regressions permits known metadata failures to recur undetected

Files Needing Attention: tests/test_litellm/llms/fireworks_ai/test_fireworks_ai_kimi_model_metadata.py, tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py

Important Files Changed

Filename	Overview
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py	Removes direct pricing assertions, including an independent GPT-5.6 alias-to-Sol parity regression
tests/test_litellm/llms/fireworks_ai/test_fireworks_ai_kimi_model_metadata.py	Deletes Fireworks Kimi limit coverage, leaving K2.5 without an equivalent exact output-limit assertion
tests/test_litellm/llms/databricks/test_databricks_cost_calculator.py	Removes provider-wide cache-rate invariants while retaining representative cache billing tests
tests/test_litellm/router_utils/test_reasoning_effort_capability.py	Removes provider-matrix reasoning metadata assertions while retaining generic resolver and selected hydration coverage
tests/test_litellm/test_utils.py	Removes several model-info and cross-platform metadata regressions, with partial replacement coverage remaining
tests/test_litellm/llms/mistral/ocr/test_mistral_ocr_cost.py	Removes a redundant model-info price assertion while retaining registry and end-to-end OCR billing tests

Comments Outside Diff (1)

1. tests/test_litellm/llms/fireworks_ai/test_fireworks_ai_kimi_model_metadata.py

   Regression coverage removed

   Deleting these limit and pricing parity regressions can hide unsupported token limits or pricing drift, violating the coverage requirement; restore protection before merging

   Rule Used: What: Flag any modifications to existing tests and verify they don't weaken test coverage or mask regressions. Why: Developers may alter tests to make failing code pass rather than fix the actual bug, hiding regressions. Good: ``` // Test updated t... (source)

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "test: drop cost-map mirror tests" | Re-trigger Greptile}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 96c02d5. Configure here.}

[mateo-berri]

Closing as superseded: #41443 and #41763 already removed these tests, apart from a handful those PRs kept on purpose as invariant or behavior checks