test(pricing): pin the gpt-realtime-mini mode assertion to the local cost map - #33761
Draft
cursor[bot] wants to merge 1 commit into
Draft
test(pricing): pin the gpt-realtime-mini mode assertion to the local cost map#33761cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
…ini mode test_get_model_info_reports_realtime_mode called get_model_info, which by default fetches model_prices_and_context_window.json from GitHub's main branch. The mode=realtime change from #33728 lives on the internal staging branch but has not landed on main yet, so CI fetched the stale 'chat' mode and the assertion flipped, breaking the Unit Tests: MCP, Secrets, Containers & Misc job on every open PR. Point the assertion at the bundled backup JSON by forcing the local cost map flag on and rebinding model_cost, matching the pattern used by the realtime transcription tests in test_cost_calculator.py. Clear the get_model_info lru_cache so a prior remote-backed call in the same worker can not leak through. Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Screenshots / Proof of Fix
Before, on
litellm_internal_stagingat 7015bd2:The assertion
litellm.get_model_info('gpt-realtime-mini')['mode'] == 'realtime'fails because the default cost-map URL points atmain, where the mode is stillchat:After, on this branch:
Failure logs on
litellm_internal_staging: https://github.com/BerriAI/litellm/actions/runs/29605562029Type
✅ Test
Changes
test_get_model_info_reports_realtime_modeintests/test_litellm/test_gpt_realtime_mode.pycalledlitellm.get_model_info, which by default fetchesmodel_prices_and_context_window.jsonfrom themainbranch on GitHub. #33728 landed themode=realtimeretag forgpt-realtime-minionlitellm_internal_stagingbut it has not reachedmainyet, so every CI run pulled the stalechatvalue and the assertion flipped. That failure was hitting theUnit Tests: MCP, Secrets, Containers & Miscjob on every open PR that rebased on top of the broken commit.Point the assertion at the bundled backup JSON (which does carry the retag) by forcing the local cost map on and rebinding
litellm.model_cost, matching the pattern already used by the realtime transcription tests intests/test_litellm/test_cost_calculator.py. Clear theget_model_infolru_cachefirst so a prior remote-backed call in the same worker cannot leak through. The three sibling tests in this file already read from the bundled file via_load_cost_mapand were not affected.Final Attestation