fix(model-costs): apply GPT-5.6 Sol promotional pricing cut - #37880
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Mubashir Osmani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Greptile SummaryThe PR applies GPT-5.6 Sol promotional pricing across the canonical and backup cost maps and updates regression coverage.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | Updates the affected OpenAI GPT-5.6 family pricing fields consistently across tiers, caching modes, and context thresholds. |
| litellm/model_prices_and_context_window_backup.json | Mirrors the canonical pricing-map changes without detected divergence. |
| tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | Updates tier-specific expectations and adds regression coverage requiring the bare GPT-5.6 alias to match Sol pricing. |
| tests/test_litellm/test_cost_calculator.py | Updates the end-to-end cache-aware completion-cost expectation to the promotional rates. |
Reviews (3): Last reviewed commit: "test(model-costs): use the local_model_c..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
OpenAI's model page for gpt-5.6 serves the GPT-5.6 Sol page and states that the gpt-5.6 alias routes requests to GPT-5.6 Sol, so the alias bills at Sol's rates. The registry entry was left on the pre-cut rates while gpt-5.6-sol took the cut, overbilling gpt-5.6 callers by 25 percent on input and 50 percent on output. All 23 cost fields on gpt-5.6 now match gpt-5.6-sol, and a regression test pins the two entries together so they cannot drift again.
The docstring claimed azure pricing mirrors the openai family, which stopped being true when gpt-5.6-sol took its promotional cut and azure did not. Azure publishes no sol rate of its own today, so the entries stay where they are.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 71400e1. Configure here.
…test The new test set LITELLM_LOCAL_MODEL_COST_MAP and reassigned litellm.model_cost by hand, leaking both into every test that ran after it and skipping the get_model_info cache clear. The conftest fixture already does this properly and restores the original map on the way out.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 1a55418. Configure here.
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer calling GPT-5.6 Sol through the gateway, by any of its three names, sees spend well above what OpenAI actually invoices
{"model": "gpt-5.6-sol", ...}and get back 13 prompt tokens, 4 completion tokensx-litellm-response-cost: 0.000185, priced at $5.00 in and $30.00 out per 1M"service_tier": "fast"comes back atx-litellm-response-cost: 0.00037, priced at $10.00 in and $60.00 out{"model": "gpt-5.6", ...}, which OpenAI routes to the same Sol model, and get the same inflated0.000185and0.00037After: the same calls are billed at the promotional rates OpenAI publishes, whichever name the caller used
{"model": "gpt-5.6-sol", ...}for the same token countsx-litellm-response-cost: 0.000132, priced at $4.00 in and $20.00 out per 1M"service_tier": "fast"variant comes back atx-litellm-response-cost: 0.000264, priced at $8.00 in and $40.00 out{"model": "gpt-5.6", ...}call now matches at0.000132and0.000264instead of billing the pre-cut rateRelevant issues
Linear ticket
LIT-5985
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -vScreenshots / Proof of Fix
Rates come from https://developers.openai.com/api/docs/pricing (Standard, Batch, Flex and Fast mode tables), which also notes the promo runs at least through November 21, 2026. Sol short context is now $4.00 in, $0.40 cached, $5.00 cache write, $20.00 out, and long context is $8.00 / $0.80 / $10.00 / $30.00
Why the bare gpt-5.6 alias is in scope
OpenAI's model docs state the
gpt-5.6alias routes requests to GPT-5.6 Sol, and a live call confirms it: sending{"model": "gpt-5.6"}straight to OpenAI comes back with"model": "gpt-5.6-sol"in the response body. The alias entry held all 23 cost fields at the pre-cut Sol values, so anyone calling the short name was overbilled by 25% on input and 50% on output while the Sol entry itself was correct. The newtest_gpt_5_6_alias_prices_match_solpins the two entries together so they cannot drift apart againShared setup, a live proxy against real OpenAI with both deployments:
Each leg ran in its own worktree with its own venv, its own random free port and
LITELLM_LOCAL_MODEL_COST_MAP=True, so each proxy loaded the cost map committed at that leg's own commit. Before ran atbb99f5774e, this PR's merge base against litellm_internal_staging, and After at the tip. Real OpenAI, no mocks. Because the change touches shared cost code, every leg covers all three unified endpoints rather than just the one the User Flow narratesThe call, with
"service_tier":"fast"added to the body for the fast legs:Before (bb99f57, the merge base)
Expected is the pre-cut rate, $5.00 in and $30.00 out per 1M, doubled for fast. Split headers matched independently: the Sol standard chat leg returned
x-litellm-response-cost-input: 6.500000000000001e-05andx-litellm-response-cost-output: 0.00012./v1/model/infoat this commit reportedinput_cost_per_token5e-06 andoutput_cost_per_token3e-05 for both namesAfter (1a55418, the tip)
Expected is the promotional rate, $4.00 in and $20.00 out per 1M, doubled for fast. Split headers matched here too: 5.2e-05 and 8e-05 on the Sol standard chat leg. The two model names now agree on every endpoint, which is the whole point of the alias half of the fix. The QA legs ran at
71400e1029;1a55418ea2only swaps the alias test onto the conftestlocal_model_cost_mapfixture and cannot change billingScope check on the map itself
Diffing both cost maps across
bb99f5774e..1a55418ea2shows exactly 3 of 3165 entries changed (gpt-5.6,gpt-5.6-sol,daybreak-blue-latest), no entries added or removed, and no key added or removed on any entry, so nothing reading the map can break structurally. Every delta is a decrease: x0.8 on every input-side field and x0.6667 on every output-side field. All the family's internal rate ratios still hold afterwards (cache read 0.1x input, cache write 1.25x, batch and flex 0.5x, priority 2x, long-context input 2x and output 1.5x)Observations from the run
service_tier: "priority", not"fast"_prioritycost keys, no_fastkey existsusage.cost: nulldespite a correct cost headerdaybreak-blue-latesthas no flex, priority or batch keys at allgpt-5.6alias lackssupports_computer_usethat Sol carriesThe last two are preexisting and untouched by this PR, split out as LIT-5986 and LIT-5992
Type
🐛 Bug Fix
Caveats (if any)
_above_272k_tokens_priorityfields are missing from twelve entries. That is a preexisting gap unrelated to the promo, so it is split out as LIT-5986max_input_tokens(daybreak-blue-latest at 1050000, the three bedrock_mantle gpt-5.6 entries at 1000000, all of which should be 922000). Also preexisting and unrelated to pricing, split out as LIT-5987gpt-5.6alias is missingsupports_computer_use, which both gpt-5.6-sol and daybreak-blue-latest carry. Same alias-drift shape as the pricing bug, but it is a capability claim rather than a rate, so it wants its own check against OpenAI's docs. Split out as LIT-5992Link to Devin session: https://app.devin.ai/sessions/49ef693b0c6647599ce2f2dac8c9ab9b
Note
Medium Risk
Changes billed token rates for a live model family, so incorrect numbers would under/overcharge spend tracking. Scope is limited to three OpenAI cost-map entries plus tests; Azure/Bedrock are unchanged.
Overview
Lowers OpenAI GPT-5.6 Sol billing to the published promo rates so spend matches the invoice instead of overcharging ~25% on input and ~50% on output.
Updates
gpt-5.6-sol, thegpt-5.6alias (OpenAI routes it to Sol), anddaybreak-blue-latestacross standard, batch, flex, priority/fast, cache, and >272K keys. Azure and Bedrock Sol entries are left at their own published rates.Adds
test_gpt_5_6_alias_prices_match_solso the alias cannot drift from Sol on cost fields again, and retargets existing cost tests to the new rates.Reviewed by Cursor Bugbot for commit 1a55418. Bugbot is set up for automated code reviews on this repo. Configure here.