test(cost-calc): stop 182 global writes leaking out of the cost-calc suites - #37815
Merged
yuneng-berri merged 3 commits intoAug 22, 2026
Merged
Conversation
Contributor
Greptile SummaryThe PR replaces hand-written environment and global-state restoration in the cost-calculation suites with pytest monkeypatch fixtures.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| test-quality-budget.json | Lowers the TQ003, TQ004, and TQ005 ceilings to match the removed test anti-patterns. |
| tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | Centralizes local model-cost-map setup and cleanup in a monkeypatch-backed fixture. |
| tests/test_litellm/test_cost_calculator.py | Replaces direct environment and LiteLLM configuration mutations with automatically restored monkeypatch operations. |
Reviews (2): Last reviewed commit: "test(cost-calc): stop 182 global writes ..." | Re-trigger Greptile
yuneng-berri
force-pushed
the
litellm_cost_calc_global_writes
branch
from
August 21, 2026 10:34
247b09d to
eee8cb9
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
yuneng-berri
added a commit
that referenced
this pull request
Aug 21, 2026
…em fully Both files are also in #37815, which converts the module-global writes as well as the env writes and folds them into one fixture. Two PRs rewriting the same lines differently is a conflict nobody benefits from resolving, so this one drops back to staging on those two and keeps the other 39. TQ004 clears 200 here instead of 275; the rest moves with #37815.
6 tasks
…suites Across test_cost_calculator.py and llm_cost_calc/test_llm_cost_calc_utils.py, 58 tests opened by setting LITELLM_LOCAL_MODEL_COST_MAP in os.environ and replacing litellm.model_cost, and none of them put the env var back. The second file already had a _local_model_cost_map fixture doing it by hand with a try/finally, so both idioms sat in the same file. Keep that fixture, give it monkeypatch, and have every one of those tests ask for it. The margin and discount tests drop their hand-rolled copy-then-restore in favour of monkeypatch.setattr, which also puts the global back when an assertion fails part way through. Both files also drop a sys.path.insert whose argument resolves outside the repo, so it was never what made the imports work. TQ003 1077 -> 1075, TQ004 768 -> 693, TQ005 2836 -> 2731, and the budget ceilings come down with them.
yuneng-berri
force-pushed
the
litellm_cost_calc_global_writes
branch
from
August 21, 2026 10:59
eee8cb9 to
2c4038a
Compare
Contributor
Author
This was referenced Aug 21, 2026
ryan-crabbe-berri
approved these changes
Aug 22, 2026
yucheng-berri
approved these changes
Aug 22, 2026
yuneng-berri
added a commit
that referenced
this pull request
Aug 22, 2026
#37806) * test: use monkeypatch.setenv for env writes in tests/test_litellm `os.environ["X"] = v` inside a test leaks the value into every test that runs after it in the same worker, so ordering decides the result. 262 of those writes across 40 files now go through pytest's `monkeypatch` fixture, which restores the previous value at teardown. The rewrite skips any test that a mock.patch-family decorator wraps, any test with defaulted positional parameters, any test whose own name is called directly elsewhere, and rebinds nothing inside nested defs, because in each of those cases appending a fixture parameter changes what pytest or mock binds. Ratchets the TQ004 ceiling from 768 to 506. * fix(test): delete the key through monkeypatch instead of popping it first Five tests popped a key straight out of `os.environ`, ran, then restored it with `monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone, so it recorded "absent" as the value to go back to and deleted the key at teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`, `UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first one ran without it. `monkeypatch.delenv(..., raising=False)` removes the key and restores whatever was there, so the try/finally the manual restore needed goes with it. * chore(test): leave the two cost-calc files to the PR that rewrites them fully Both files are also in #37815, which converts the module-global writes as well as the env writes and folds them into one fixture. Two PRs rewriting the same lines differently is a conflict nobody benefits from resolving, so this one drops back to staging on those two and keeps the other 39. TQ004 clears 200 here instead of 275; the rest moves with #37815.
…itellm_cost_calc_global_writes # Conflicts: # test-quality-budget.json
The local_cost_map fixture set LITELLM_LOCAL_MODEL_COST_MAP but never reloaded litellm.model_cost, and reading the variable is not what loads the map. So the three streaming-cost tests billed against whatever map the process happened to be holding, and their hardcoded prices only held when something else had already swapped in the checked-in one. This branch stops the cost-calc tests leaking that map, which left test_main billing at the ambient prices instead. The fixture now loads the map it names, so the prices these tests assert hold on their own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
User Flow
No end-user behavior changes. Both suites report the same counts, 89 and 174,
and a test that runs after them in the same worker now starts from the
environment it would have had on its own.
Relevant issues
Linear ticket
Pre-Submission checklist
Screenshots / Proof of Fix
Shared setup: run a suite, then read the process back in the same interpreter.
What is left over is what the next test in that worker inherits.
Before (ff02d5c)
the leak the suites leave behind
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.pywhat the budget rules see
uv run python scripts/check_test_quality.py tests/test_litellm/test_cost_calculator.py tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | grep -o 'TQ00[0-9]' | sort | uniq -cAfter (eee8cb9)
the leak the suites leave behind
what the budget rules see
uv run python scripts/test_quality_gate.pyType
✅ Test
🧹 Refactoring
Caveats (if any)
Final Attestation