fix(volcengine): support tiered Doubao pricing - #34902
Conversation
|
|
4bfa7bc to
a13b4cf
Compare
Greptile SummaryAdds request-wide tiered cost calculation for Volcengine models.
Confidence Score: 5/5The PR appears safe to merge; no concrete changed-code failure was identified. The new dispatcher preserves custom-pricing precedence, the calculator applies the repository’s established tier boundaries and flat-pricing fallback, and the dashboard transformation correctly formats the token-tier shapes introduced by this change.
|
| Filename | Overview |
|---|---|
| litellm/cost_calculator.py | Adds the Volcengine provider dispatch after custom-pricing overrides and before generic pricing fallback. |
| litellm/llms/volcengine/cost_calculator.py | Implements request-wide tier selection, cached-token accounting, short-output rates, and flat-pricing fallback without an identified defect. |
| model_prices_and_context_window.json | Adds synchronized Seed 1.8 pricing metadata and retains the existing Seed 2.0 tier definitions. |
| litellm/model_prices_and_context_window_backup.json | Mirrors the canonical pricing-map changes for the bundled backup map. |
| tests/test_litellm/llms/volcengine/test_cost_calculator.py | Covers tier boundaries, request-wide rates, cached prompts, short outputs, and top-level dispatch. |
| ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/utils/modelDataTransformer.ts | Derives formatted minimum–maximum token-price ranges from tier metadata while preserving flat-price behavior. |
| ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/ModelsTableColumns.tsx | Labels tiered ranges appropriately while retaining the existing currency rendering. |
| ui/litellm-dashboard/src/components/model_dashboard/types.ts | Extends model cost fields to support formatted range strings and identifies tiered rows. |
Reviews (1): Last reviewed commit: 4bfa7bc | Re-trigger Greptile
a13b4cf to
04a6e02
Compare
| { | ||
| "input_cost_per_token": 1.1e-07, | ||
| "output_cost_per_token": 2.8e-07, | ||
| "output_cost_per_token_above_200_tokens": 1.1e-06, |
There was a problem hiding this comment.
Low: Higher output rate bypasses the pre-call budget bound
An authenticated user can request more than 200 output tokens while near their budget limit because _max_cost_for_cost_info() only considers output_cost_per_token and output_cost_per_reasoning_token. For this tier it reserves at 2.8e-07, while post-call accounting charges the entire output at 1.1e-06; concurrent requests can therefore incur substantially more provider spend than the atomic budget gate permits. Update the budget reservation calculation to include this conditional rate when estimating maximum output cost.
PR overviewThis pull request adds support for tiered pricing for Volcengine Doubao models, including a higher conditional output-token rate. One issue remains open in the pre-call budget calculation: it reserves output costs using the lower base rate even when the higher tier may apply. An authenticated user near their budget limit could use sufficiently large or concurrent requests to cause provider spending beyond the enforced reservation, though exploitation depends on those budget and concurrency conditions. Open issues (1)
Fixed/addressed: 0 · PR risk: 4/10 |
Co-authored-by: Cursor <cursoragent@cursor.com>
04a6e02 to
4427f56
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
$0.00Context
The current code contains tiered price data for the Seed 2.0 models, but the cost dispatcher has no Volcengine branch, so requests fall through to flat pricing and are tracked as zero. This reintroduces the routing intent from #30357 while using Volcengine's request-wide, input-length-selected tier semantics instead of graduated bracket math.
Testing
22 passed— Volcengine calculator and model-cost-map unit tests on thelitellm_internal_stagingbase