fix(batches): price a retrieved batch from its deployment's model and rates (internal copy of #37077)

[mateo-berri]

Internal copy of #37077 by @marty-sullivan (head 74ce9bb), whose org-owned fork blocks maintainer pushes. All 8 commits are cherry-picked with authorship preserved, plus follow-up commits from review and QA: a fix for an in-place mutation of get_model_info's lru-cached dict, a merge absorbing #37078 and #37205 from staging, and a fix making the enterprise cost poller price from the same deployment rates as the retrieve path

TLDR

Problem this solves:

• Retrieving a batch prices it with no model identity
• Bedrock batches silently cost $0 with correct tokens
• A deployment's own configured rates are ignored
• A zero-cost deployment gets billed the public rate
• The enterprise cost poller ignored deployment rates too

How it solves it:

• Price against the deployment's own provider-qualified model
• Honor rates the deployment actually declares
• Merge published rates onto a copy, never the cached entry
• The poller prices through the same shared helper

User Flow

Before: a team running Bedrock batch jobs sees completed batches under-billed: $0.00 on gateways without background batch cost tracking, and a deployment's own configured rates ignored on gateways with it

1. They upload a JSONL file with POST https://litellm-domain/v1/files (purpose: batch, target_model_names: claude-haiku-4-5) and get back a gateway file id
2. They send POST https://litellm-domain/v1/batches with that input_file_id and get back a batch id with status: validating
3. They poll GET https://litellm-domain/v1/batches/{batch_id} until it returns status: completed
4. They open https://litellm-domain/ui/?page=logs: without background batch cost tracking the batch's entry shows real tokens against $0.00 spend, and with it the entry is priced at the model's public batch rate
5. A team whose gateway administrator configured custom batch rates on a deployment is billed the public rate either way: 0.00242 instead of 0.0052 in the live run below

After: the same batch reports the spend those tokens actually cost, at the rates the deployment declares

1. They upload the same JSONL file with POST https://litellm-domain/v1/files and get back a gateway file id
2. They send the same POST https://litellm-domain/v1/batches and get back a batch id with status: validating
3. They poll GET https://litellm-domain/v1/batches/{batch_id} until it returns status: completed
4. https://litellm-domain/ui/?page=logs now shows exactly one spend entry for the batch with the same tokens priced at the model's published batch rate
5. The custom-rate deployment's batch is billed exactly the rates declared on that deployment: 0.0052 in the live run below

Relevant issues

Follows #36876 and #37050, which made a batch cost row reach the spend logs at all. With rows now landing, their amounts became auditable, which is how this surfaced

Linear ticket

Resolves LIT-5666

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Fresh end-to-end run against real Bedrock managed batches at the merged tip, with the enterprise cost poller active (PROXY_BATCH_POLLING_INTERVAL=30), which is the production-realistic accountant. Before at the current staging base 3f4810b, After at 5a11fe1 (the tip differs from it only by a test-only commit). A broader 23-deployment run of the inline retrieve path (Bedrock + Vertex + zero-cost Vertex) is preserved in #37077's description

Shared setup. Two deployments of bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0: haiku-batch-std with no custom pricing and haiku-batch-custom declaring input_cost_per_token_batches: 2.0e-06 and output_cost_per_token_batches: 4.0e-06. LITELLM_LOCAL_MODEL_COST_MAP=True, fresh database per side. One real 100-record batch per deployment:

curl -s http://localhost:{PORT}/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F target_model_names="haiku-batch-std" -F file=@in_std.jsonl
curl -s http://localhost:{PORT}/v1/batches -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"input_file_id":"<unified id>","endpoint":"/v1/chat/completions","completion_window":"24h"}'
curl -s http://localhost:{PORT}/v1/batches/<batch id> -H "Authorization: Bearer $KEY"   # poll to completed
psql -c 'SELECT model, call_type, spend, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs";'

Before (3f4810b)

1. Both real Bedrock jobs completed in ~10 minutes; the poller then wrote one aretrieve_batch row per batch

<std batch id>_batch_cost     us.anthropic.claude-haiku-4-5...  aretrieve_batch  0.00242  1400  600
<custom batch id>_batch_cost  us.anthropic.claude-haiku-4-5...  aretrieve_batch  0.00242  1400  600

2. Both rows price at the public batch rate. The custom deployment's declared rates are ignored: it books 0.00242 instead of 1400 x 2.0e-06 + 600 x 4.0e-06 = 0.0052
3. The retrieve path itself still cannot price at all: every client GET /v1/batches/{id} logs response_cost_failure with "Model is None" (19 occurrences across the two batches), so a proxy without the enterprise poller books $0. The total blackout this causes on the older base, where batches got no priced row at all, is the 973329e run in fix(batches): price a retrieved batch from its deployment's model and rates #37077's description

After (5a11fe1)

1. Same flow, same tokens; the poller writes exactly one row per batch

<std batch id>_batch_cost     us.anthropic.claude-haiku-4-5...  aretrieve_batch  0.00242               1400  600
<custom batch id>_batch_cost  us.anthropic.claude-haiku-4-5...  aretrieve_batch  0.005200000000000001  1400  600

2. The std row matches the published batch rate exactly: 1400 x 0.55e-06 + 600 x 2.75e-06 = 0.00242 (the 1.1x us. profile rates halved), relative error 1.8e-16
3. The custom row matches the deployment's declared batch rates exactly: 1400 x 2.0e-06 + 600 x 4.0e-06 = 0.0052, relative error 1.7e-16, where the published rates would have given 0.00242
4. Accounting ownership is clean: ~11 client GETs per batch wrote no extra rows (batch_ignore_default_logging on the retrieve path), and the spend transactions carry the LiteLLM Proxy/CheckBatchCost user agent

Type

🐛 Bug Fix

Changes

Retrieving a completed batch computed its cost with no model identity: the call that aggregates a batch's output passed neither the deployment's model nor its pricing

For Bedrock that mattered because the cost model then fell back to the provider's own response model, a bare name like claude-opus-4-5-20251101, which does not resolve under a bedrock provider prefix. The lookup missed, and a missed lookup returns zero rather than raising, so cost became $0 while usage stayed correct

Dropping the deployment's model info mattered independently: rates configured on a deployment are registered by the router under that deployment's id, and never consulting them meant a deployment configured to cost nothing was billed from the global map instead. Standard deployments were unaffected, since their configured rates and the global map agree, which is why this only showed up on Bedrock and on custom-rate deployments

Both are fixed at the one call site, by passing the deployment's model and the pricing registered for it. Two details are worth calling out because the obvious implementations of each are wrong:

The model comes from model_call_details. On this path self.model and litellm_params["model"] are both None, and self.model can otherwise hold the router's model_group alias, which no cost map resolves; a live retrieve confirmed model_call_details is the only one of the three carrying the provider-qualified deployment model

Deployment pricing is decided from the raw registration rather than get_model_info. The router registers an entry for every deployment whether or not it declares pricing, and get_model_info fills absent costs with 0, so asking it cannot tell "configured as free" apart from "no pricing configured" and would have priced every ordinary deployment's batches at $0. The shared helper therefore returns pricing only when the deployment actually declares one of the batch cost fields, leaving the global map as the default. Ownership is per token direction: the side a deployment leaves unset keeps the published rates, and a declared side is never displaced by them

On top of the original PR, this copy merges the published rates onto a copy of get_model_info's return value. That function serves every call from one lru-cached dict, so writing the merge into it poisoned every later get_model_info lookup of the deployment id with the published model's rates for the life of the process; the new regression test fails on the in-place version

This copy also extends the fix to the enterprise cost poller. #37050 hands accounting ownership to CheckBatchCost whenever it is active (poller_owns_accounting), but the poller priced from deployment_info.model_info.model_dump(), which never carries litellm_params pricing, so a custom-rate deployment's batches would have been billed at the public rate on exactly the path that now owns them. The deployment-pricing merge is extracted to a module-level deployment_pricing_model_info and the poller calls it with the same deployment id and provider-qualified model as the retrieve path; a regression test asserts the poller passes the declared batch rates through to calculate_batch_cost_and_usage

One behavior change worth naming: with a model name now reaching this path, a proxy that sets disable_vertex_batch_output_transformation will take the Vertex-specific accounting branch on retrieve, which it previously could not reach. That branch exists for exactly that flag, so this makes the flag work on the retrieve path rather than changing what it means

Caveats

• Bedrock InvokeModel usage shapes beyond Anthropic's remain unparsed
• They warn and bill $0; parsing them is follow-up work

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR corrects completed-batch accounting by carrying the deployment’s provider-qualified model and configured pricing into cost calculation.

• Merges published and deployment-specific rates without mutating cached model metadata.
• Preserves explicit zero batch rates and supports independently configured input and output pricing.
• Adds regression coverage for Bedrock pricing, custom deployment rates, polling, and cached model information.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported validation issue has been fixed, and no blocking failure remains.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/litellm_logging.py	Resolves deployment model identity and composes configured pricing with published rates on a copied model-info object.
litellm/cost_calculator.py	Distinguishes explicit zero batch rates from unset rates during input and output cost calculation.
litellm/batches/batch_utils.py	Threads deployment-level model information through completed-batch aggregation.
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Aligns poller pricing with inline retrieval by using router-registered deployment pricing.
tests/test_litellm/test_cost_calculator.py	Replaces the previously reported disabled type suppressions with typed literals and adds explicit-zero pricing coverage.
tests/test_litellm/litellm_core_utils/test_litellm_logging.py	Covers deployment pricing resolution, one-sided overrides, cache immutability, and retrieve-path propagation.

_{Reviews (2): Last reviewed commit: "test(cost): type the batch_cost_calculat..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e736b59. Configure here.}

[tin-berri]

LGTM — solid fix for the batch-retrieve $0 pricing bug (missing model identity meant Bedrock cost-map lookups silently missed) and for custom deployment rates being ignored in favor of the public rate. Good catches on top of the original: avoiding the in-place mutation that would've poisoned get_model_info's lru-cached dict, per-direction rate ownership so a declared side never gets displaced by published rates, and extending the same fix to the enterprise cost poller. Live Bedrock QA matches expected cost to ~1e-16 relative error on both the standard and custom-rate deployments.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_internal_copy_37077 (e736b59) with litellm_internal_staging (2bc87ec)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (2975265) during the generation of this report, so 2bc87ec was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩