fix: de-dup jp.anthropic.claude-sonnet-4-6 and fail CI on duplicate pricing keys - #27
Merged
blackflame007 merged 1 commit intoJul 28, 2026
Conversation
…ricing keys litellm/model_prices_and_context_window_backup.json carried "jp.anthropic.claude- sonnet-4-6" twice, inherited from upstream at the base commit (pre-existing, not introduced by the sync or by the gemini-omni de-dup). json.load keeps the LAST occurrence, so the entry a reader finds first at line 2353 was dead text and the one at line 2385 was what actually loaded. The two copies were not identical. No shared field disagreed, but the winning copy was an older shape missing five fields the losing copy had: cache_creation_input_token_cost_above_1hr (6.6e-06) prompt_cache_min_tokens (1024) supports_adaptive_thinking supports_output_config supports_parallel_tool_use_config and carrying two the losing copy did not: supports_minimal_reasoning_effort and tool_use_system_prompt_tokens (346). So the duplicate was not inert. The losing first copy is byte-identical to the entry in model_prices_and_context_window.json and matches the shape of every sibling region (anthropic./global./us./eu./au. claude-sonnet-4-6), none of which carry supports_minimal_reasoning_effort or tool_use_system_prompt_tokens. The winning copy diverged from the canonical map and dropped the 1-hour prompt-cache write tier, so anything loading the backup map priced jp. 1hr cache writes at the 5-minute rate and undercounted that spend by ~60%. Drop the stale second copy and keep the first; the backup entry now matches the canonical map exactly. model_prices_and_context_window.json was NOT affected -- it has a single, correct entry. A full scan at every nesting depth confirms neither file has any other duplicate key. The durable half: nothing could have caught this. The validate-model-prices-json job ran only `jq empty`, and jq -- like json.load -- accepts a repeated key and silently keeps the last. That is how two of these have now reached us from upstream. scripts/check_model_prices_duplicate_keys.py re-parses both maps with an object_pairs_hook that inspects the raw key/value pairs before they collapse into a dict, so a repeat is reported instead of swallowed, at any nesting level. It is wired into the same job, which now also validates the backup map (previously unchecked). Verified the guard fails before it passes: it flags this duplicate at lines 2353/2385 on the pre-fix file, flags the gemini-omni duplicate on the pre-de-dup commit, flags a nested duplicate field, and exits 0 on the fixed tree. jq empty passes all three of those. tests/test_litellm/test_model_prices_no_duplicate_keys.py pins both halves: the maps stay duplicate-free, the backup and canonical jp. entries stay in sync with the 1hr tier present, and the guard itself is asserted to FAIL on deliberately duplicated fixtures so it cannot rot into a no-op.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9543c5f6e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+69
to
+73
| needle = f'"{key}":' | ||
| lines = tuple( | ||
| number | ||
| for number, line in enumerate(raw.splitlines(), start=1) | ||
| if needle in line |
There was a problem hiding this comment.
When a commonly repeated nested field such as mode is duplicated in one model, this global text scan reports every "mode": line in the entire price map rather than the two occurrences in the offending object. On the current canonical map that produces roughly 20 KB of unrelated line numbers and does not identify which model contains the duplicate, making the new CI failure difficult to act on; retain the object path or source positions while parsing instead of rescanning globally by key name.
Useful? React with 👍 / 👎.
blackflame007
merged commit Jul 28, 2026
9b19685
into
litellm_internal_staging
71 of 74 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes NOL-90.
The duplicate
litellm/model_prices_and_context_window_backup.jsoncarried"jp.anthropic.claude-sonnet-4-6"twice — lines 2353 and 2385 — inherited from upstream at the base commit. Pre-existing: not introduced by the sync (NOL-74) or by the gemini-omni de-dup (NOL-79).json.loadkeeps the last occurrence, so the entry a reader finds first (2353) was dead text and the one at 2385 was what actually loaded.They were not identical, and the wrong one was winning
No shared field disagreed — every price value was the same in both. The difference was in which fields existed.
cache_creation_input_token_cost_above_1hr6.6e-06prompt_cache_min_tokens1024supports_adaptive_thinkingtruesupports_output_configtruesupports_parallel_tool_use_configtruesupports_minimal_reasoning_efforttruetool_use_system_prompt_tokens346Kept the first (2353), dropped the second. Two independent reasons it is the intended entry:
jp.anthropic.claude-sonnet-4-6entry inmodel_prices_and_context_window.json, the canonical map.anthropic.,global.,us.,eu.,au.claude-sonnet-4-6all carrysupports_adaptive_thinking, the 1hr tier,prompt_cache_min_tokens: 1024,supports_output_configandsupports_parallel_tool_use_config, and none of them carrysupports_minimal_reasoning_effortortool_use_system_prompt_tokens.So this duplicate was not inert. The copy that won had diverged from the canonical map and dropped the 1-hour prompt-cache write tier, meaning anything loading the backup map priced
jp.1hr cache writes at the 5-minute rate — a ~60% undercount on that spend.tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.pyalready asserts6.6e-06for this model but reads the root map, which is why it stayed green.Sibling file
model_prices_and_context_window.jsonwas not affected — single, correct entry. A full scan at every nesting depth confirms neither file has any other duplicate key.The durable fix
Nothing could have caught this.
validate-model-prices-jsonran onlyjq empty, and jq — likejson.load— accepts a repeated key and silently keeps the last. That is how two of these have now reached us from upstream.scripts/check_model_prices_duplicate_keys.pyre-parses both maps with anobject_pairs_hookthat inspects the raw key/value pairs before they collapse into a dict, so a repeat is reported instead of swallowed, at any nesting level. Wired into the same job, which now also validates the backup map (previously unchecked at all).Proof it fails before it passes
A guard only ever observed passing is not a guard:
jq emptyjp.anthropic.claude-sonnet-4-62x at lines 2353, 238545e798ce73gemini/gemini-omni-flash-previewand the jp. one"mode"twice)modeat lines 3, 4It would have caught NOL-79 too.
tests/test_litellm/test_model_prices_no_duplicate_keys.pypins both halves — the maps stay duplicate-free, the backup and canonicaljp.entries stay in sync with the 1hr tier present, and the guard itself is asserted to fail on deliberately duplicated fixtures so it cannot rot into a no-op. Collected by the existingmiscunit job (tests/test_litellm/test_*.py).Tests
test_bedrock_anthropic_1hr_cache_pricing.py,test_claude_sonnet_4_6_config.py,test_anthropic_sonnet_1hr_cache_pricing.py,test_bedrock_usgov_haiku_1hr_cache.py: 50 passedtests/test_litellm/test_utils.py: 254 passedtests/test_litellm/litellm_core_utils/llm_cost_calc/: 149 passedmake pre-commit: exit 0zizmoron the edited workflow: no findingsNotes for review
fal_ai/kling/openrouter-videoentries touched.test-server-root-pathis expected red here — known, tracked in NOL-88.