test: route the cost-calc tests through the local-cost-map fixture - #37835
test: route the cost-calc tests through the local-cost-map fixture#37835yuneng-berri wants to merge 1 commit into
Conversation
45 tests in test_llm_cost_calc_utils.py set LITELLM_LOCAL_MODEL_COST_MAP and swapped litellm.model_cost inline and never put either back, so anything that ran after them in the same process read the checked-in snapshot instead of the live map. The file already had a _local_model_cost_map fixture doing this correctly for the data-residency tests; those 45 now request it too. The fixture itself moves from a hand-rolled try/finally to monkeypatch, which also restores when setup raises rather than only when the test body finishes. Dropping the sys.path.insert takes the last reader of os and sys with it. It resolved "../../.." against the working directory, so it only ever pointed at the repo root when pytest happened to be invoked from this test's own folder.
Greptile SummaryThe PR routes cost-calculation tests through the existing monkeypatch-based local-cost-map fixture so environment and global pricing state are restored after each test.
Confidence Score: 5/5The PR appears safe to merge, with the test-state cleanup correctly scoped and no actionable regressions identified. The fixture installs a fresh local cost map for each requesting test and relies on pytest monkeypatch teardown to restore both the environment and the original module attribute; the removed path insertion was not load-bearing under supported test invocation.
|
| Filename | Overview |
|---|---|
| tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | Consolidates local cost-map setup into a monkeypatch fixture that restores environment and module state without weakening the existing assertions. |
| test-quality-budget.json | Ratchets the path-mutation, environment-mutation, and module-global-mutation budgets to reflect violations removed by the test refactor. |
Reviews (1): Last reviewed commit: "test: route the cost-calc tests through ..." | Re-trigger Greptile
|
Superseded by #37815, which already covers this file and more |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
TLDR
Problem this solves:
LITELLM_LOCAL_MODEL_COST_MAPset session-widelitellm.model_costswapped to the local snapshotHow it solves it:
_local_model_cost_mapfixturemonkeypatch, so setup failures restore toosys.path.insertthat resolved against the working directoryUser Flow
Before: a proxy admin reports a cost discrepancy, and the maintainer checking it locally reads a stale price
litellm.model_costand gets the checked-in snapshot, not the live mapAfter: the same run reads the live map, so the answer matches the proxy
LITELLM_LOCAL_MODEL_COST_MAPandlitellm.model_costback when each one finishesRelevant issues
Linear ticket
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
The leak is only visible from outside the pytest process, so the probe runs the file in-process and reads the environment back afterwards.
LITELLM_LOCAL_MODEL_COST_MAPis not part of the conftest snapshot, so nothing else restores it.Before (ff02d5c)
python leakprobe.py174 passed, thenLITELLM_LOCAL_MODEL_COST_MAP after: 'True'After (7c1c800)
python leakprobe.py174 passed, thenLITELLM_LOCAL_MODEL_COST_MAP after: NoneType
🧹 Refactoring
✅ Test
Caveats (if any)
litellm.model_costrestores via the conftest snapshot either wayFinal Attestation