test(pricing): pin the realtime mode assertion to the bundled cost map

[yuneng-berri]

Relevant issues

Flake introduced by #33728. Same fix as #33761, which is stuck on an unsigned CLA check; this copy is mergeable

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Root cause, live against the hosted map on 2026-07-17 (commit independent; this is what a deployed instance with default settings sees today):

$ curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | jq '."gpt-realtime-mini".mode'
"chat"

$ python -c "import litellm; print(litellm.get_model_info('gpt-realtime-mini')['mode'])"
chat

Before, at a402069 (litellm_internal_staging tip): the fetch from main succeeds on this machine, so the test fails every time

$ python -m pytest tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode -q
tests/test_litellm/test_gpt_realtime_mode.py:72: AssertionError
FAILED tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode
1 failed in 0.22s

After, at e360f96: the whole file passes and the target test is deterministic across fresh processes

$ python -m pytest tests/test_litellm/test_gpt_realtime_mode.py -q
29 passed in 0.28s

$ for i in 1 2 3 4 5; do python -m pytest tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode -q | tail -1; done
1 passed in 0.12s
1 passed in 0.17s
1 passed in 0.16s
1 passed in 0.12s
1 passed in 0.11s

The cache_clear matters: a remote-backed get_model_info call earlier in the same pytest worker caches the stale answer, and the pin alone would not flip it

$ python -c "
import litellm, os
print('cache poisoned by a remote-backed call:', litellm.get_model_info('gpt-realtime-mini')['mode'])
os.environ['LITELLM_LOCAL_MODEL_COST_MAP'] = 'True'
litellm.model_cost = litellm.get_model_cost_map(url='')
litellm.get_model_info.cache_clear()
print('after pin + cache_clear:', litellm.get_model_info('gpt-realtime-mini')['mode'])
"
cache poisoned by a remote-backed call: chat
after pin + cache_clear: realtime

Type

✅ Test

Changes

test_get_model_info_reports_realtime_mode resolved gpt-realtime-mini through litellm.get_model_info, which reads whatever cost map litellm loaded at import. By default that map is fetched from raw.githubusercontent.com/BerriAI/litellm/main, where these models are still mode "chat"; the #33728 retag lives in this branch's json and bundled backup. So the assertion failed whenever the fetch succeeded and passed whenever the runner was rate limited and litellm fell back to the bundled backup, which is why Unit Tests: MCP, Secrets, Containers & Misc flapped on roughly half of runs on unrelated PRs (for example runs 29622577355, 29622877031, 29621137528)

The test now forces LITELLM_LOCAL_MODEL_COST_MAP and rebinds litellm.model_cost to the bundled map, the same pattern tests/test_litellm/test_cost_calculator.py already uses, and clears the get_model_info lru cache before asserting so a remote-backed entry cached earlier in the same worker cannot leak through. It clears the cache again on the way out so no locally-backed entry outlives the test. The three sibling tests already read the repo json directly and were never affected, and no other test under tests/test_litellm resolves these models through the hosted map, so this closes the whole exposure, including for the CircleCI unit jobs once those run again

Beyond CI, the underlying skew is visible to end users: a deployed instance on the default hosted cost map still reports mode "chat" for the realtime-only gpt models (first proof block above) and will until the pricing json promotion reaches main. That promotion is the complementary fix; this PR only makes the unit suite independent of when it lands

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR fixes a flaky unit test (test_get_model_info_reports_realtime_mode) that failed whenever the test runner could fetch the hosted cost map from raw.githubusercontent.com/main, where the realtime-mode retag has not yet landed. The fix follows the same pattern already used in test_cost_calculator.py.

• Sets LITELLM_LOCAL_MODEL_COST_MAP=True and rebinds litellm.model_cost to get_model_cost_map(url=\"\") (the bundled backup) via monkeypatch, so teardown automatically restores both.
• Clears the get_model_info lru cache before the assertion (to evict any remote-backed entry cached earlier in the same worker) and again in a finally block (to evict the locally-backed entry before the next test).

Confidence Score: 5/5

Safe to merge — the change is confined to one test function and does not touch production code.

The fix correctly addresses the root cause (lru cache poisoning from a remote-backed call earlier in the same pytest worker) using the established monkeypatch + cache_clear pattern. Monkeypatch teardown restores the env var and model_cost; the finally block clears the cache before teardown runs, so no locally-backed entry can leak into subsequent tests. The three sibling tests are untouched and continue to cover the repo JSON directly. Test coverage is not weakened.

No files require special attention.

Important Files Changed

Filename	Overview
tests/test_litellm/test_gpt_realtime_mode.py	Pins test_get_model_info_reports_realtime_mode to the bundled cost map via monkeypatch + cache_clear to eliminate remote-map flakiness; sibling tests are unchanged.

_{Reviews (1): Last reviewed commit: "test(pricing): pin the realtime mode ass..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_/recursing-cohen-0b597b (e360f96) with litellm_internal_staging (40e914c)}