fix(batches): price a retrieved batch from its deployment's model and rates - #37077
Conversation
… rates Retrieving a completed batch computed its cost with no model identity: neither the deployment's model nor its configured pricing reached the batch cost calculation. For bedrock that left the cost model falling back to the provider's own response model (e.g. "claude-sonnet-4-6"), which does not resolve under a bedrock provider, so the lookup missed and cost silently became $0 while usage stayed correct. Dropping the deployment's model info separately discarded any rates configured on that deployment, billing a zero-cost deployment at the public rate instead. Both are the same omission at the call site, so both are fixed by passing the logging object's own model and the pricing the router registered for the deployment.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…s it The router registers a model_info entry for every deployment, priced or not, and get_model_info fills absent costs with 0. Resolving deployment pricing through it therefore reported a free deployment for any ordinary one, which priced its batches at $0 while usage stayed correct: the same silent under-count this branch set out to remove, widened from bedrock to every provider. Caught by a live batch run, where four vertex batches that price correctly today came back at $0. The raw registration is now what decides: pricing is used only when the deployment actually declares one of the batch cost fields, so ordinary deployments fall back to the global cost map exactly as before. The earlier test missed this by using a deployment id that was never registered, where get_model_info does raise; a real deployment is always registered.
self.model can carry the router's model_group alias, which no cost map resolves, so a bedrock batch still priced at $0 after the model name started being passed. The deployment's own litellm_params model is used when present. Verified against the local (image-bound) cost map that dev and prod both force: alias 'claude-opus-4-5' prices $0.000000 while 'bedrock/global.anthropic.claude-opus-4-5-20251101-v1:0' prices $0.017000
On a batch retrieve both self.model and litellm_params[model] come back None, so the cost model fell through to the provider's own response model (an Anthropic id like claude-opus-4-5-20251101) which does not resolve under a bedrock provider, leaving bedrock batches at $0 with correct usage. model_call_details carries the deployment's provider-qualified model (bedrock/global.anthropic.claude-opus-4-5-20251101-v1:0), confirmed by instrumenting a live retrieve, so it is preferred with the previous two sources kept as fallbacks.
Greptile SummaryThe PR threads deployment model identity and pricing into completed-batch accounting and preserves explicit zero batch rates. However, its partial-pricing merge can bypass deployment-specific standard rates
Confidence Score: 4/5The PR is not yet safe to merge because partially configured deployments can have retrieved batches charged at public rates instead of their configured rates Filling an absent batch-rate field from published pricing prevents batch_cost_calculator from deriving the charge from the deployment-specific standard rate Files Needing Attention: litellm/litellm_core_utils/litellm_logging.py
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/litellm_logging.py | Adds deployment pricing resolution, but filling absent batch fields from public pricing can override configured standard-rate fallback |
| litellm/cost_calculator.py | Correctly distinguishes explicit zero batch rates from absent rates |
| litellm/batches/batch_utils.py | Threads deployment model identity and ModelInfo into completed-batch aggregation |
| tests/test_litellm/litellm_core_utils/test_litellm_logging.py | Covers deployment lookup and partial standard pricing, but does not assert the resulting batch charge |
| tests/test_litellm/batches/test_batch_utils.py | Adds focused regression coverage for Bedrock model identity and zero-cost deployment pricing |
| tests/test_litellm/test_cost_calculator.py | Verifies explicit zero and absent batch-rate behavior |
Reviews (4): Last reviewed commit: "test(batches): cover the deployment with..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… rates Carries the four commits behind upstream PR BerriAI#37077 onto the release branch: - pass the deployment's model and its registered pricing into batch cost - resolve deployment pricing only when the deployment declares it, since the router registers an entry for every deployment and get_model_info fills absent costs with 0 - prefer model_call_details for the model, because self.model and litellm_params[model] are both None on a batch retrieve and self.model can otherwise hold the router's model_group alias Verified end to end against real batches for 23 deployments: bedrock anthropic went from $0 to exact ($0.0034 haiku, $0.0102 sonnet, $0.017 opus), the nine standard vertex deployments are byte-identical, and the nine zero-rate deployments went from being billed at public rates to $0 with real token counts. Two release-branch adaptations: the two AWS credential-forwarding tests are dropped (that upstream change is not on this branch), and the handler test resolves whichever success-handler entrypoint this version exposes. Also applies the outstanding ruff format fix to openai_files_endpoints/common_utils.py.
…rrections The earlier carry landed only the first of upstream PR BerriAI#37077's four commits: cherry-pick -n stopped at a test conflict and the remaining three were never applied, so the branch shipped without the corrections that PR needed. Adds them: - resolve deployment pricing only when the deployment declares it, since the router registers an entry for every deployment and get_model_info fills absent costs with 0, which would price ordinary deployments' batches at $0 - prefer model_call_details for the model, because self.model and litellm_params[model] can both be None on a batch retrieve and self.model can otherwise hold the router's model_group alias, which no cost map resolves Two release-branch adaptations: register_model here takes no persist_across_reloads argument, and the handler test resolves whichever success-handler entrypoint this version exposes.
…nset Substituting a deployment's pricing wholesale billed the token direction it did not configure at zero: get_model_info fills an absent cost with 0, and any non-None pricing field suppressed the global fallback. A deployment declaring only input_cost_per_token therefore billed output at nothing. Each of the four batch cost fields now falls back to the model's published rate when the deployment leaves it unset, so a one-sided override applies to the side it configures and only that side. Adds a parametrized regression over input-only, output-only, and both-zero, plus coverage for a deployment whose model has no published entry. Annotates the new test helpers per the repo's type-coverage rule and drops the narrative banner comment from the batch tests.
|
Both addressed. The undeclared side now falls back to the published rate, so a one-sided override applies only to the side it configures, and the test helpers carry concrete types. Reproduced the partial-pricing case first: a deployment declaring only input billed output at $0.000000 against a published $0.0075. It now reads input from the deployment and output from the cost map, covered by a parametrized regression over input-only, output-only, and both-zero. |
batch_cost_calculator gated the batch rate fields on truthiness, so a deployment that configures input_cost_per_token_batches or its output twin as 0.0 was read as having configured nothing and that token direction fell through to half the standard rate. Layering declared rates over published ones made this reachable: a deployment declaring only a zero batch rate previously kept a fabricated zero on the standard field, which happened to bill nothing. The two batch fields are now gated on presence. Verified no cost-map entry changes behavior: the only three carrying a zero batch rate are embeddings, whose standard output rate is also 0.0, so both paths yield the same zero. Adds a parametrized regression over an explicit zero, an explicit non-zero, and unset, plus coverage for the deployment id get_model_info cannot resolve, which were the lines Codecov flagged.
|
Agreed, and fixed at the source: batch_cost_calculator now gates the two batch rate fields on presence rather than truthiness, so a rate configured as 0.0 means free instead of unset. Before this a deployment declaring only a zero batch rate was charged half the standard rate for that direction, which layering declared rates over published ones had newly exposed. Checked the cost map for fallout: the only three entries carrying a zero batch rate are text-embedding-3-large, -3-small and ada-002-v2, whose standard output rate is also 0.0, so both the old and new paths yield the same zero. No published pricing changes behavior. Also covered the two branches Codecov flagged, so the uncovered lines in litellm_logging.py are gone. |
Codecov's remaining uncovered patch line was the early return taken when no model is available to look a published entry up by, which leaves a deployment's own declared rates standing alone. Measuring the patch lines against the coverage report now leaves none uncovered.
Filling each cost field independently let a published batch rate outrank a standard rate the deployment configured itself: a deployment declaring only input_cost_per_token had its batches billed at the model's published batch price rather than half its own rate. Measured on a model that publishes both, that billed $0.001500 where the deployment's own rate meant $0.000500. Declaring either rate for a direction now claims that whole direction, so nothing published can displace it, and a direction the deployment is silent on still inherits both published rates.
|
Confirmed and fixed in Ownership is now per token direction rather than per field: declaring either rate for a direction claims that whole direction, so nothing published can displace it, while a direction the deployment is silent on still inherits both published rates. Verified across standard-only, batch-zero-only, all-four-zero, and output-only Also covered the last uncovered patch line Codecov flagged, the early return when no model is resolvable at all |
|
The direction-ownership approach handles that edge case correctly. Looking at the merged result for a deployment with only
The test in The zero-rate path is also consistent: Coverage of the four declared-rate combinations (standard-only, batch-zero-only, all-four-zero, output-only) and the Codecov-flagged early return makes this complete. Implementation looks correct. |
|
@greptileai it seems like all open items are resolved but your review still sits at 4/5. Are there still open items that need to be addressed |
|
All four threads are resolved and the current diff addresses every issue that was open. The score was set during the first automated pass and simply hasn't been updated since. Looking at the code now:
There are no remaining open items. This is a 5/5 — it's correct, the edge cases are tested, and standard pricing is demonstrably untouched. Good to merge. |
fix(batches): price a retrieved batch from its deployment's model and rates (internal copy of #37077)
|
Continuing this in #37219 since the org-owned fork blocks maintainer pushes. All commits are cherry-picked there with authorship preserved. Thanks @marty-sullivan! |
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a team running Bedrock batch jobs sees every completed batch land in their spend logs with real token counts but $0.00 spend, so chargeback under-reports
purpose: batch,target_model_names: claude-sonnet-4-6) and get back a gateway file idinput_file_idand get back a batch id withstatus: validatingstatus: completedAfter: the same batch reports the spend those tokens actually cost, and a zero-cost deployment stays at zero
status: validatingstatus: completedRelevant issues
Follows #36876 and #37050, which made a batch cost row reach the spend logs at all. With rows now landing, their amounts became auditable, which is how this surfaced
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Shared setup. A proxy runs against real Bedrock and Vertex with 23 batch-enabled deployments: five Bedrock Anthropic models, nine standard Vertex models, and nine Vertex deployments configured to cost nothing. Batch polling is off so the retrieve path is the only accountant, which is the path this PR changes.
LITELLM_LOCAL_MODEL_COST_MAP=True, matching how these deployments run. One real batch per deployment was submitted and left to complete, then every batch was retrieved over HTTP and the resulting spend rows read back from the database. The same 23 batches are used on both sides, with the rows cleared and each batch's stored state rewound between runs so both runs price the same provider outputBefore (973329e)
Bedrock Anthropic
GET /v1/batches/{id}for each of the five, then read the rowsStandard Vertex
Zero-cost deployments
After (d86c0bd)
Bedrock Anthropic
GET /v1/batches/{id}for each of the five, then read the rowsStandard Vertex
Zero-cost deployments
Across all 23 batches every row carries token counts that reconcile against the provider's own output file, 23 of 23 price as expected, and no batch produced more than one row
Type
🐛 Bug Fix
Changes
Retrieving a completed batch computed its cost with no model identity: the call that aggregates a batch's output passed neither the deployment's model nor its pricing
For Bedrock that mattered because the cost model then fell back to the provider's own response model, a bare name like
claude-opus-4-5-20251101, which does not resolve under a bedrock provider prefix. The lookup missed, and a missed lookup returns zero rather than raising, so cost became $0 while usage stayed correctDropping the deployment's model info mattered independently: rates configured on a deployment are registered by the router under that deployment's id, and never consulting them meant a deployment configured to cost nothing was billed from the global map instead. Standard deployments were unaffected, since their configured rates and the global map agree, which is why this only showed up on Bedrock and on zero-rate deployments
Both are fixed at the one call site, by passing the deployment's model and the pricing registered for it. Two details are worth calling out because the obvious implementations of each are wrong:
The model comes from
model_call_details. On this pathself.modelandlitellm_params["model"]are both None, andself.modelcan otherwise hold the router'smodel_groupalias, which no cost map resolves; a live retrieve confirmedmodel_call_detailsis the only one of the three carrying the provider-qualified deployment modelDeployment pricing is decided from the raw registration rather than
get_model_info. The router registers an entry for every deployment whether or not it declares pricing, andget_model_infofills absent costs with 0, so asking it cannot tell "configured as free" apart from "no pricing configured" and would have priced every ordinary deployment's batches at $0.get_router_deployment_model_infotherefore returns pricing only when the deployment actually declares one of the batch cost fields, leaving the global map as the defaultOne behavior change worth naming: with a model name now reaching this path, a proxy that sets
disable_vertex_batch_output_transformationwill take the Vertex-specific accounting branch on retrieve, which it previously could not reach. That branch exists for exactly that flag, so this makes the flag work on the retrieve path rather than changing what it meansCaveats
Final Attestation