fix(ptu): stop per-token billing on a PTU-configured deployment - #36829
Conversation
Greptile SummaryThe PR prevents PTU deployments from accruing usage-based charges in addition to reserved-capacity costs and preserves budget enforcement for those deployments.
Confidence Score: 4/5The PR is not yet safe to merge because removing PTU can discard an explicitly supplied zero rate and restore unintended public-map billing. The current release logic treats caller-authored zeros and PTU-generated zeros identically, removes both from persisted pricing blobs, and therefore leaves an operator-requested free rate vulnerable to public pricing fallback. Files Needing Attention: litellm/proxy/management_endpoints/model_management_endpoints.py
|
| Filename | Overview |
|---|---|
| litellm/proxy/management_endpoints/model_management_endpoints.py | Adds PTU pricing validation, zeroing, and release behavior, but the release path still cannot preserve an explicit zero supplied while PTU is removed. |
| litellm/proxy/auth/auth_checks.py | Keeps PTU deployments subject to budget checks despite their intentionally zero per-token prices. |
| litellm/proxy/spend_tracking/ptu_flat_cost_rollup.py | Reports closed PTU attribution windows through bounded, Slack-safe alerts. |
| ui/litellm-dashboard/src/components/add_model/advanced_settings.tsx | Adds inline PTU incompatibility validation to visible usage-cost fields, including per-second pricing. |
| ui/litellm-dashboard/src/components/model_info_view.tsx | Updates PTU edit-form pricing display and validation behavior. |
| ui/litellm-dashboard/src/utils/ptuValidation.ts | Centralizes validation that prevents nonzero visible usage rates alongside PTU configuration. |
| tests/test_litellm/proxy/management_endpoints/test_ptu_model_settings.py | Broadly covers PTU zero pricing, budget enforcement, creation, updates, and release, but does not cover an explicit zero supplied during PTU removal. |
Reviews (9): Last reviewed commit: "fix(ptu): stop per-token billing on a PT..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| """ | ||
| if not is_ptu_cost_attribution_enabled(): | ||
| return _NO_PRICING_OVERRIDE | ||
| if model_info.get("ptu_count") is None or model_info.get("cost_per_ptu_per_hour") is None: |
There was a problem hiding this comment.
High: PTU effective windows allow budget bypass
This condition ignores ptu_effective_from and ptu_effective_to, although the flat-cost rollup charges only during that window. A team administrator can configure an expired or future window and continue calling the deployment with zero token rates, so budget reservation and spend tracking record no cost. Preserve normal per-token pricing outside the active PTU window, or prevent the deployment from serving requests there.
PR overviewThis PR changes PTU-configured deployments to use flat PTU billing instead of per-token usage billing. The affected model-management logic determines when token pricing is suppressed for PTU deployments. Two security issues remain open in the PTU billing logic, despite one issue having been addressed. Administrators can configure inactive PTU billing windows or a zero hourly PTU rate so requests accrue neither token-based nor flat-rate spend, allowing continued provider resource consumption without budget enforcement. The PTU activation conditions should be tightened before merging. Open issues (2)
Fixed/addressed: 1 · PR risk: 7/10 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if not is_ptu_cost_attribution_enabled(): | ||
| return _NO_PRICING_OVERRIDE | ||
| if model_info.get("ptu_count") is None or model_info.get("cost_per_ptu_per_hour") is None: | ||
| return _NO_PRICING_OVERRIDE |
There was a problem hiding this comment.
🔴 Reserved-capacity deployments serve traffic completely free when their reservation is not active
Per-token prices are set to zero for a reserved-capacity deployment (_ptu_zeroed_pricing at litellm/proxy/management_endpoints/model_management_endpoints.py:385-386) without regard to whether the reservation is actually charging, so during any period when the capacity charge does not apply the deployment's traffic costs nothing at all.
Impact: Requests through such a deployment are recorded as $0 spend, so that usage is never billed and never counts toward team or key budgets.
How the zeroed price outlives the flat capacity charge
The flat capacity charge is only accrued for the overlap of a UTC day with [ptu_effective_from, ptu_effective_to) (litellm/proxy/spend_tracking/ptu_flat_cost_rollup.py:1-12, window parsing at litellm/proxy/spend_tracking/ptu_flat_cost_rollup.py:146-159), and the whole rollup is skipped when the feature flag is off (litellm/proxy/spend_tracking/ptu_flat_cost_rollup.py:522).
The zeroed pricing, by contrast, is written into both model_info and litellm_params permanently and is only released when a patch explicitly nulls ptu_count/cost_per_ptu_per_hour (_ptu_pricing_delta at litellm/proxy/management_endpoints/model_management_endpoints.py:427-434). So:
- a deployment configured with a future
ptu_effective_frombills nothing (no flat cost yet, zero per-token) until the window opens; - once
ptu_effective_topasses, the flat cost stops but the zeros remain, so all later traffic is free; - turning
ENABLE_PTU_COST_ATTRIBUTIONoff stops the rollup while the stored zeros stay, so the documented "disabling pauses PTU" behaviour silently means free serving.
A fix likely needs either to release/repair the zero pricing when the window closes (or the flag is disabled), or to have cost calculation treat a reserved-capacity deployment's zero price as applying only inside the effective window.
Was this helpful? React with 👍 or 👎 to provide feedback.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
c1a90f6 to
a3d484d
Compare
|
@greptileai review latest head |
|
bugbot run |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
a3d484d to
c39b686
Compare
| """ | ||
| if not is_ptu_cost_attribution_enabled(): | ||
| return _NO_PRICING_OVERRIDE | ||
| if model_info.get("ptu_count") is None or model_info.get("cost_per_ptu_per_hour") is None: |
There was a problem hiding this comment.
Medium: Zero-rate PTU configuration bypasses budget enforcement
This treats cost_per_ptu_per_hour=0 as active PTU pricing and zeroes all usage prices. Because a zero-value flat charge does not add spend, subsequent requests can consume provider resources without affecting the budget. Require a strictly positive PTU rate before suppressing usage-based pricing, or preserve normal pricing when the flat rate is zero.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
@greptileai review latest head |
|
bugbot run |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
c39b686 to
2f2036d
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
@greptileai review latest head |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2f2036d. Configure here.
796fadc to
44f793a
Compare
|
@greptileai review latest head |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 44f793a. Configure here.
|
@greptileai releasing the zeros is the ticket's acceptance criterion; keeping them bills nothing and waives budgets, proven live. Per-second only lacks a form field. Please re-review 44f793a |
44f793a to
3f1dc2f
Compare
A deployment with PTU flat-cost attribution also billed every request per
token, so a team paid for reserved capacity and again for the traffic that
capacity serves. Nothing set the per-token price and an unset price falls
back to the public cost map, which made the double charge the default.
/model/new and /model/{id}/update now store zero for every pricing field the
cost map could otherwise fill, refuse a price the caller supplies alongside
PTU config with a 400 naming the field, zero a price already on the row
rather than rejecting later edits of unrelated fields, and drop the zeros
again when the PTU config goes.
A PTU deployment is no longer read as a free model by the budget checks,
which would have waived every budget for it.
3f1dc2f to
a9620f1
Compare
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3f1dc2f. Configure here.
|
@greptileai releasing the zeros is the ticket's acceptance criterion; keeping them bills nothing and waives budgets, proven live. Per-second only lacks a form field. Please re-review a9620f1 |
| return _NO_PRICING_OVERRIDE, frozenset( | ||
| field | ||
| for field in _CUSTOM_PRICING_FIELDS.union(_PTU_ZEROED_PRICING_FIELDS) | ||
| if _is_zero_price(model_info.get(field)) or _is_zero_price(litellm_params.get(field)) | ||
| ) |
There was a problem hiding this comment.
🟡 A new per-token price set in the same edit that turns off reserved-capacity billing is silently thrown away
A price the caller sends in the same edit that removes the reserved-capacity settings is deleted from the deployment (the release set built at litellm/proxy/management_endpoints/model_management_endpoints.py:455-459) instead of being kept, so the deployment silently falls back to the public price list rather than the price the operator just chose.
Impact: An operator who moves a deployment off reserved capacity and sets a new per-token rate in one save ends up with the rate they entered missing and the model billed at the vendor's list price.
Why the release also drops a freshly supplied rate
On such a patch _ptu_zeroed_pricing returns empty (the merged model_info no longer carries ptu_count/cost_per_ptu_per_hour), so _ptu_pricing_delta takes the release branch. The release predicate is _is_zero_price(model_info.get(field)) or _is_zero_price(litellm_params.get(field)). update_db_model merges the patch's litellm_params into merged_litellm_params only (litellm/proxy/management_endpoints/model_management_endpoints.py:509-515); it never copies those values into merged_model_info, so merged_model_info["input_cost_per_token"] is still the 0.0 this feature wrote earlier. The or therefore fires, and the loop at litellm/proxy/management_endpoints/model_management_endpoints.py:554-556 pops the field from both blobs, removing the non-zero rate the caller supplied in this very request. Repro: PATCH with model_info={ptu_count: null, cost_per_ptu_per_hour: null} plus litellm_params={input_cost_per_token: 2.5e-06} on a PTU row; the stored row comes back with no input_cost_per_token at all.
Prompt for agents
In _ptu_pricing_delta (litellm/proxy/management_endpoints/model_management_endpoints.py), the release set returned when a patch clears the PTU pair includes any pricing field that reads as zero in either the merged model_info or the merged litellm_params. Because update_db_model merges the patch's litellm_params into merged_litellm_params only (model_info keeps the previously stored zero), a patch that clears PTU config and supplies a brand-new non-zero rate in the same request has that rate popped out of both blobs, so the deployment silently reverts to the public cost map. The release should exclude any field the patch itself supplies with a non-zero value (i.e. compute `supplied` and skip fields where `_is_nonzero_price(supplied.get(field))`), and ideally consider only fields whose merged litellm_params value is zero rather than OR-ing with the stale model_info copy. Add a regression test in tests/test_litellm/proxy/management_endpoints/test_ptu_model_settings.py covering "clear PTU and set a new per-token price in one PATCH".
Was this helpful? React with 👍 or 👎 to provide feedback.
|
@greptileai a per-second rate is operator-authored, so the refusal is correct, and it stays editable in the LiteLLM Params box. Please re-review a9620f1 |
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a proxy admin buys reserved Azure capacity for a team, configures it on the deployment, and the team is still charged for every request it sends through that capacity
input_cost_per_tokenreading 3e-07 andoutput_cost_per_tokenreading 2.5e-06, values they never enteredAfter: the same deployment charges the capacity cost only, and an attempt to price it per token fails immediately with an explanation
Relevant issues
Linear ticket
Resolves LIT-5508
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Two proxies on one Postgres, same config, same deployment shape, real Gemini calls. Before at
add095b494(the merge base), after atc39b686e91.Before,
add095b494. The deployment carries no price of its own, so the public cost map supplies one:After,
c39b686e91. Same deployment shape, same ten requests, same token count:A price sent alongside PTU config is refused, and the message names the field:
Removing the PTU config hands per-token billing back:
A row mispriced through a path this PR does not cover heals on its next save rather than blocking unrelated edits:
The dashboard round-trips the whole
model_infoblob that/model/infohands it, and that blob carries cost-maprates the operator never set. Only the price a caller authors on
litellm_paramsis read as an attempt to charge,so putting an existing deployment onto PTU from the form works and still stores zeros:
Clearing PTU from the same form releases the zeros, so per-token billing comes back:
Deployments without PTU config are untouched, create, reprice and clear, on both sides:
A closed PTU window raises an operator alert rather than changing what is billed. Reserved capacity is billed by
the provider until the deployment is deleted, so per-token pricing there would invent a charge that does not exist:
Budget enforcement still applies to a PTU deployment. A team over its budget, calling the deployment directly:
All of the above is the recording below, driven end to end at
a9620f1a47The dashboard, same commit. A deployment on reserved capacity prices at $0.00/1M
Typing a cost while PTU Count is set fails inline instead of as a server error, and the other
rates show the 0 they are stored at rather than reading as unset
Putting an existing deployment on capacity from Edit Settings now works, where the rates the form
seeded from the cost map used to block the save. It saves, and the deployment prices at $0.00/1M
The same rule on the Add Model form
Type
🐛 Bug Fix
Caveats (if any)
Final Attestation