Skip to content

test(pricing): pin the gpt-realtime-mini mode assertion to the local cost map - #33761

Draft
cursor[bot] wants to merge 1 commit into
litellm_internal_stagingfrom
cursor/ci-autofix-automation-89ac
Draft

test(pricing): pin the gpt-realtime-mini mode assertion to the local cost map#33761
cursor[bot] wants to merge 1 commit into
litellm_internal_stagingfrom
cursor/ci-autofix-automation-89ac

Conversation

@cursor

@cursor cursor Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Before, on litellm_internal_staging at 7015bd2:

$ python3 -c "import litellm; print(litellm.get_model_info('gpt-realtime-mini')['mode'])"
chat

The assertion litellm.get_model_info('gpt-realtime-mini')['mode'] == 'realtime' fails because the default cost-map URL points at main, where the mode is still chat:

$ curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | jq '."gpt-realtime-mini".mode'
"chat"

After, on this branch:

$ python3 -m pytest tests/test_litellm/test_gpt_realtime_mode.py -v
...
tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode PASSED
...
29 passed in 0.40s

Failure logs on litellm_internal_staging: https://github.com/BerriAI/litellm/actions/runs/29605562029

Type

✅ Test

Changes

test_get_model_info_reports_realtime_mode in tests/test_litellm/test_gpt_realtime_mode.py called litellm.get_model_info, which by default fetches model_prices_and_context_window.json from the main branch on GitHub. #33728 landed the mode=realtime retag for gpt-realtime-mini on litellm_internal_staging but it has not reached main yet, so every CI run pulled the stale chat value and the assertion flipped. That failure was hitting the Unit Tests: MCP, Secrets, Containers & Misc job on every open PR that rebased on top of the broken commit.

Point the assertion at the bundled backup JSON (which does carry the retag) by forcing the local cost map on and rebinding litellm.model_cost, matching the pattern already used by the realtime transcription tests in tests/test_litellm/test_cost_calculator.py. Clear the get_model_info lru_cache first so a prior remote-backed call in the same worker cannot leak through. The three sibling tests in this file already read from the bundled file via _load_cost_map and were not affected.

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
Open in Web View Automation 

…ini mode

test_get_model_info_reports_realtime_mode called get_model_info, which
by default fetches model_prices_and_context_window.json from GitHub's
main branch. The mode=realtime change from #33728 lives on the internal
staging branch but has not landed on main yet, so CI fetched the stale
'chat' mode and the assertion flipped, breaking the
Unit Tests: MCP, Secrets, Containers & Misc job on every open PR.

Point the assertion at the bundled backup JSON by forcing the local
cost map flag on and rebinding model_cost, matching the pattern used
by the realtime transcription tests in test_cost_calculator.py. Clear
the get_model_info lru_cache so a prior remote-backed call in the same
worker can not leak through.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing cursor/ci-autofix-automation-89ac (a8e243e) with litellm_internal_staging (7015bd2)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants