fix(cost): bill reasoning tokens at the service tier output rate - #35925
Conversation
A tier request against a model that publishes tier output pricing but no tier reasoning key (every current Gemini flash entry) billed reasoning tokens at the standard output_cost_per_reasoning_token, undercounting priority and fast traffic where thinking tokens dominate completions generic_cost_per_token now resolves the reasoning rate with explicit precedence: an explicit output_cost_per_reasoning_token_<tier> key wins, then the tier-resolved output rate when the model prices that tier, then the standard reasoning key, then the output base cost. The two tier reasoning keys are wired through ModelInfo so providers can publish real tiered reasoning prices when they exist
Greptile SummaryThis PR makes reasoning-token billing follow service-tier output pricing while preserving explicit tier-specific reasoning rates.
Confidence Score: 5/5The PR appears safe to merge, with the tier-aware billing behavior consistently wired through pricing resolution, model information, and generated types. The changed calculator preserves existing token accounting, applies explicit tier reasoning rates first, inherits tier output pricing only when that tier is published, and retains standard reasoning pricing for models without tier rates.
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/llm_cost_calc/utils.py | Adds tier-aware reasoning-token rate selection while leaving token-count accounting unchanged. |
| litellm/types/utils.py | Adds optional flex and priority reasoning-token pricing fields to model information and custom-pricing types. |
| litellm/utils.py | Propagates the new tier-specific reasoning prices through model-information construction. |
| tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | Adds focused regression tests for tier fallback, explicit override, null values, and standard-price preservation. |
| ui/litellm-dashboard/src/lib/http/schema.d.ts | Updates generated API declarations with the two new optional reasoning-price fields. |
Reviews (1): Last reviewed commit: "fix(cost): bill reasoning tokens at the ..." | Re-trigger Greptile
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| return tier_reasoning_cost | ||
| tier_output_key: Final = _get_service_tier_cost_key("output_cost_per_token", service_tier) | ||
| if tier_output_key != "output_cost_per_token" and model_info.get(tier_output_key) is not None: | ||
| return completion_base_cost |
There was a problem hiding this comment.
Medium: Tiered reasoning charges bypass budget reservations
This settles reasoning tokens at the tier output rate, while _max_cost_for_cost_info() reserves output using only output_cost_per_token and output_cost_per_reasoning_token and ignores the request's service tier. For example, a priority gemini-3.5-flash request can be reserved at 9e-6 per output token but settle at 1.62e-5, allowing a user to submit a reasoning-heavy request that is admitted below the remaining budget and then exceeds it. Update budget reservation to resolve the requested tier's output and reasoning keys and reserve every output token at the highest applicable rate.
There was a problem hiding this comment.
Reservation already ignored service_tier for input and output tokens; making it tier-aware is a separate estimator change covering every token type at once
PR overviewThis pull request updates cost calculation so reasoning tokens are billed using the output rate associated with the request’s service tier. The tier-aware settlement calculation is not yet matched by the pre-request budget reservation logic. A user can submit a reasoning-heavy request that passes the budget check at a lower rate but settles above the remaining budget, causing a controlled spending limit to be exceeded. This issue remains open, with no fixes addressed yet. Open issues (1)
Fixed/addressed: 0 · PR risk: 6/10 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…itellm_tier_aware_reasoning_token_cost # Conflicts: # litellm/types/utils.py
Seven live e2e tests covering cost-tracking regressions that currently ship unnoticed: cache-write tokens billed at the cache-creation rate (BerriAI#34046), per-component cost_breakdown on the spend row (BerriAI#31686), cache reads billed at the cache-read discount on streamed calls (BerriAI#34812), cache tokens surviving the anthropic-messages to Responses bridge (BerriAI#34957), priority-tier rates applied to input, output and reasoning (BerriAI#35923, BerriAI#35925), the per-component response cost headers summing to the total (BerriAI#36965), and cost injected into the final usage frame of an /openai passthrough stream (BerriAI#36503). Every test registers its own deployment with a distinct custom rate per component, so a component billed at the wrong rate cannot pass. The shared helpers in cost_rows.py encode the one thing the two surfaces disagree on: the spend row's input_cost is gross of cache while the response's cost-input header is net of it.
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: priority requests to thinking models bill reasoning tokens, the bulk of every completion, at the cheaper standard rate
POST https://litellm-domain/v1/chat/completionstogemini-3.5-flashwith"service_tier": "priority"and"reasoning_effort": "medium"under their virtual keyx-litellm-response-costheader prices the one visible text token at the priority rate and every reasoning token at the standard ratePOST https://litellm-domain/v1/messagesunderbills its reasoning tokens identicallyAfter: every output token settles at the requested tier's rate
POST https://litellm-domain/v1/chat/completionstogemini-3.5-flashwith"service_tier": "priority"and"reasoning_effort": "medium"under their virtual keyx-litellm-response-costheader prices every completion token, reasoning included, at the priority ratePOST https://litellm-domain/v1/messagesbills the same wayRelevant issues
Linear ticket
Resolves LIT-2095
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both legs hit the real Gemini API through a local proxy booted with
LITELLM_LOCAL_MODEL_COST_MAP=Trueand this config. gemini/gemini-3.5-flash prices standard input/output at 1.5e-06/9e-06 (reasoning key equals the output rate) and priority at 2.7e-06/1.62e-05/v1/chat/completions, priority tier with thinking:
Before (bcce83a, base branch): usage 26 prompt, 161 completion of which 160 reasoning. Text billed priority, reasoning billed standard
After (1b30b1b, this PR): usage 26 prompt, 136 completion of which 135 reasoning. Every output token at the priority rate
Standard-tier control on the fixed proxy (same request without service_tier): usage 26/157, cost 0.001452 = 26 x 1.5e-06 + 157 x 9e-06, so standard billing is untouched
/v1/messages, same prompt with
"service_tier":"priority","max_tokens":1024:/v1/responses, same prompt with
"service_tier":"priority","reasoning":{"effort":"medium"}: both legs bill fully standard rates (before 26/128 -> 0.001191, after 26/114 -> 0.001065, each exactly prompt x 1.5e-06 + output x 9e-06). service_tier never reaches the cost lookup on that path today, for any token type, so this PR neither applies nor regresses there; that plumbing gap is a separate pre-existing issueType
🐛 Bug Fix
Changes
generic_cost_per_tokenpriced reasoning tokens through a hardcodedoutput_cost_per_reasoning_tokenlookup that ignoredservice_tier, while input, output, and cache tokens all resolve tier-suffixed keys. Every current Gemini flash entry publishes tier output pricing plus a reasoning key equal to its output rate, so priority and fast requests undercounted exactly the reasoning tokens, which dominate thinking-model completionsThe new
_resolve_reasoning_token_costresolves the rate with explicit precedence: a publishedoutput_cost_per_reasoning_token_<tier>key wins, then the tier-resolved output base cost when the model prices that tier at all, then the standard reasoning key, then the output base cost. The tier lookup guards on the raw value rather than_get_cost_per_unit, because that helper silently falls back from a missing tier key to the standard key, which is the exact undercount being fixed;get_model_infoalso dumps unpublished fields as explicit None, and a None tier key must not shadow the tier output rate (covered by a dedicated regression test)output_cost_per_reasoning_token_flexandoutput_cost_per_reasoning_token_priorityare wired throughModelInfo,ModelInfoBase, and_get_model_info_helperso providers that ever publish distinct tiered reasoning prices get them applied without further code changes; the dashboard schema.d.ts regeneration follows from thatFour regression tests in
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py: the real-map gemini-3.5-flash priority/fast case, the explicit tier reasoning key winning, the present-but-None tier key falling back to the tier output rate, and a tier request against a model with no tier pricing keeping the standard reasoning rate. The first three fail on the base branchFinal Attestation