Repository navigation
Conversation
…twins Every chatgpt/* row in the cost map lacked the reasoning annotations its openai twin carries, so /model/info reported twelve reasoning models as non-reasoning. The generalization fallback cannot rescue them: _get_model_info_from_generalization returns None when any candidate is an exact cost-map key, and chatgpt/<model> is one, so the unannotated row wins over the annotated twin it proxies. The flags are ours to assert rather than a vendor claim - ChatGPTConfig subclasses OpenAIConfig and ChatGPTResponsesAPIConfig subclasses OpenAIResponsesAPIConfig, so a chatgpt row's reasoning behaviour is whatever the openai row describes. Two rows are left alone because they have no openai twin to copy from: chatgpt/gpt-5.3-codex-spark and chatgpt/gpt-5.3-instant. The new test pins the twin invariant rather than the twelve values, so the next chatgpt row added cannot repeat this. It is scoped to openai twins: the azure registry already diverges here and is not this repository's to pin.
v0ropaev
requested review from
kerry-berri,
mateo-berri and
ryan-crabbe-berri
as code owners
September 24, 2026 08:36
Contributor
|
| mismatched = [] | ||
| for name, bare in twins: | ||
| for key in REASONING_ANNOTATION_KEYS: | ||
| if key in prices[bare] and prices[name].get(key) != prices[bare][key]: |
Contributor
There was a problem hiding this comment.
Extra flags escape the guard The test checks only keys on the OpenAI twin, so an extra ChatGPT effort flag passes and can falsely advertise support
Suggested change
| if key in prices[bare] and prices[name].get(key) != prices[bare][key]: | |
| if (key in prices[name], prices[name].get(key)) != (key in prices[bare], prices[bare].get(key)): |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
Contributor
|
Folded into the rolling registry PR #42951 at field level, with your test and a Co-authored-by trailer. Closing this one as superseded, thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #41394.
All fourteen
chatgpt/*rows in the cost map carry zero reasoning annotations, while the openai rows they proxy carry the full set. Twelve of them have an openai twin to copy from; this fills those twelve in.Why the fallback does not rescue them
_get_model_info_from_generalization(litellm/utils.py:5660) refuses to generalize when any candidate is an exact cost-map key:chatgpt/gpt-5.6-solis an exact key, so the empty row wins over the annotated twin it proxies. Measured throughget_model_info, withLITELLM_LOCAL_MODEL_COST_MAP=Trueso the local map is the one under test:mainchatgpt/gpt-5.6-solsupports_reasoning=None,xhigh=NoneTrue,Truechatgpt/gpt-5.4supports_reasoning=None,xhigh=NoneTrue,Truechatgpt/gpt-5.3-instantNone,NoneNone,None(no twin — left alone)Why these values are ours to assert
Not a vendor claim read off a docs page.
ChatGPTConfigsubclassesOpenAIConfig(litellm/llms/chatgpt/chat/transformation.py:16) andChatGPTResponsesAPIConfigsubclassesOpenAIResponsesAPIConfig(litellm/llms/chatgpt/responses/transformation.py:39), which is wherereasoningreachesallowed_keysandreasoning.encrypted_contentis forced intoinclude. A chatgpt row's reasoning behaviour is whatever the openai row describes, by inheritance in this repository's own code.Each of the twelve rows takes exactly the reasoning keys its twin has, values included — so
chatgpt/gpt-5.6-solgetssupports_minimal_reasoning_effort: false, not a blankettrue.Two rows are deliberately untouched:
chatgpt/gpt-5.3-codex-sparkandchatgpt/gpt-5.3-instanthave no bare twin at all, so there is nothing to copy and anything I put there would be a guess.The test
test_openai_reasoning_family_entries_carry_supports_reasoningalready guards this class of hole, but it filters onentry.get("litellm_provider") == "openai"(test_model_prices_schema.py:258), so chatgpt-provider rows fall straight through it — which is how fourteen of them got here.The new test pins the twin invariant rather than the twelve values, so the next
chatgpt/*row added cannot repeat this. Onmain's data it fails with every mismatch named:It is scoped to openai twins on purpose. The azure registry already diverges here on
main—azure/gpt-5.4hassupports_minimal_reasoning_effort: truewhere the baregpt-5.4hasfalse, andazure/gpt-5.2,azure/gpt-5.3-codexandazure/gpt-5.4-procarrysupports_reasoningwithout the effort flags their openai twins have. A wider invariant would fail on pre-existing data and would assert something this repository does not control.Verification
Reverting only the two JSON files to
mainwhile keeping the test: 1 failed, 61 passed.Both cost-map files are updated, so the backup-drift guard stays quiet, and
python ci_cd/generate_model_prices_schema.pyis a no-op — all five reasoning keys were already in the schema.The JSON was edited textually rather than round-tripped through
json.dump, which would have reformatted floats (0.00005→5e-05) across the whole 2.8 MB file. The diff is 50 added lines and 12 changed ones, and those twelve are only the trailing commas.