fix(batches): price anthropic passthrough message batches correctly in batch cost job

[mateo-berri]

Relevant issues

Related: #24204, #11364

Linear ticket

Resolves LIT-4008

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live e2e before/after against the real Anthropic API through a local proxy, real batches billed by Anthropic, no mocks. Config is one model_list entry (model_name: claude-opus-4-6, model: anthropic/claude-opus-4-6) with PROXY_BATCH_POLLING_INTERVAL=60 so the CheckBatchCost job runs every minute. Same commands in both runs, each against a fresh database; only the checked-out commit differs

Before (commit 76eeaf2)

Create a message batch through the passthrough:

curl -s http://localhost:61304/anthropic/v1/messages/batches \
  -H "Authorization: Bearer sk-...xxxx" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"requests":[{"custom_id":"qa-before-1","params":{"model":"claude-opus-4-6","max_tokens":32,"messages":[{"role":"user","content":"Say hi"}]}},{"custom_id":"qa-before-2","params":{"model":"claude-opus-4-6","max_tokens":32,"messages":[{"role":"user","content":"Say bye"}]}}]}'

{"id":"msgbatch_01TQuNX53hJWH8GY9Dvk8CaZ","type":"message_batch","processing_status":"in_progress","request_counts":{"processing":2,"succeeded":0,"errored":0,"canceled":0,"expired":0}, ...}

Poll until it ends, then fetch the results to show both requests really completed:

curl -s http://localhost:61304/anthropic/v1/messages/batches/msgbatch_01TQuNX53hJWH8GY9Dvk8CaZ \
  -H "Authorization: Bearer sk-...xxxx" -H "anthropic-version: 2023-06-01"
# ... "processing_status":"ended","request_counts":{"processing":0,"succeeded":2, ...}

curl -s http://localhost:61304/anthropic/v1/messages/batches/msgbatch_01TQuNX53hJWH8GY9Dvk8CaZ/results \
  -H "Authorization: Bearer sk-...xxxx" -H "anthropic-version: 2023-06-01"

{"custom_id":"qa-before-1","result":{"type":"succeeded","message":{"model":"claude-opus-4-6", ..., "content":[{"type":"text","text":"Hi there! ..."}],"usage":{"input_tokens":9, ..., "output_tokens":25,"service_tier":"batch"}}}}
{"custom_id":"qa-before-2","result":{"type":"succeeded","message":{"model":"claude-opus-4-6", ..., "content":[{"type":"text","text":"Bye! ..."}],"usage":{"input_tokens":9, ..., "output_tokens":16,"service_tier":"batch"}}}}

The next CheckBatchCost cycle fetches the results from the OpenAI-shaped files endpoint; Anthropic rejects it and the error body is swallowed as file content (proxy log):

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/files/msgbatch_01TQuNX53hJWH8GY9Dvk8CaZ/content \
-H 'x-api-key: sk-ant-...xxxx' -H 'anthropic-version: 2023-06-01' -H 'anthropic-beta: files-api-2025-04-14' \
-d '{}'

batch_utils.py:291 - json_objects=[
    {
        "type": "error",
        "error": {
            "type": "invalid_request_error",
            "message": "File id must have `file_` prefix."
        },
        "request_id": "req_011CcmvZkS7vsYTbAztg4dVZ"
    }
]

The job records $0 and permanently marks the batch processed:

$ psql litellm_qa_lit4008_before -c 'SELECT call_type, model, spend, total_tokens, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE call_type = '"'"'aretrieve_batch'"'"';'
    call_type    |      model      | spend | total_tokens | prompt_tokens | completion_tokens
-----------------+-----------------+-------+--------------+---------------+-------------------
 aretrieve_batch | claude-opus-4-6 |     0 |            0 |             0 |                 0

$ psql litellm_qa_lit4008_before -c 'SELECT model_object_id, status, batch_processed FROM "LiteLLM_ManagedObjectTable";'
          model_object_id          |  status  | batch_processed
-----------------------------------+----------+-----------------
 msgbatch_01TQuNX53hJWH8GY9Dvk8CaZ | complete | t

18 real input tokens and 41 real output tokens billed by Anthropic, tracked as $0 forever

After (commit 65a34a0)

Same commands, fresh database:

curl -s http://localhost:64900/anthropic/v1/messages/batches \
  -H "Authorization: Bearer sk-...xxxx" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"requests":[{"custom_id":"qa-after-1","params":{"model":"claude-opus-4-6","max_tokens":32,"messages":[{"role":"user","content":"Say hi"}]}},{"custom_id":"qa-after-2","params":{"model":"claude-opus-4-6","max_tokens":32,"messages":[{"role":"user","content":"Say bye"}]}}]}'

{"id":"msgbatch_011RLULF5pVTevaJP7k9dyCJ","type":"message_batch","processing_status":"in_progress","request_counts":{"processing":2,"succeeded":0,"errored":0,"canceled":0,"expired":0}, ...}

curl -s http://localhost:64900/anthropic/v1/messages/batches/msgbatch_011RLULF5pVTevaJP7k9dyCJ \
  -H "Authorization: Bearer sk-...xxxx" -H "anthropic-version: 2023-06-01"
# ... "processing_status":"ended","request_counts":{"processing":0,"succeeded":2, ...}

curl -s http://localhost:64900/anthropic/v1/messages/batches/msgbatch_011RLULF5pVTevaJP7k9dyCJ/results \
  -H "Authorization: Bearer sk-...xxxx" -H "anthropic-version: 2023-06-01"

{"custom_id":"qa-after-1","result":{"type":"succeeded","message":{"model":"claude-opus-4-6", ..., "content":[{"type":"text","text":"Hi there! ..."}],"usage":{"input_tokens":9, ..., "output_tokens":25,"service_tier":"batch"}}}}
{"custom_id":"qa-after-2","result":{"type":"succeeded","message":{"model":"claude-opus-4-6", ..., "content":[{"type":"text","text":"Bye! ..."}],"usage":{"input_tokens":9, ..., "output_tokens":18,"service_tier":"batch"}}}}

The next CheckBatchCost cycle now fetches from the message batches results endpoint (proxy log; the pre-call debug template always prints "curl -X POST" but the handler issues a GET):

check_batch_cost.py:307 - Batch ID: msgbatch_011RLULF5pVTevaJP7k9dyCJ is complete, tracking cost and usage

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/messages/batches/msgbatch_011RLULF5pVTevaJP7k9dyCJ/results \
-H 'x-api-key: sk-ant-...xxxx' -H 'anthropic-version: 2023-06-01' -H 'anthropic-beta: files-api-2025-04-14' \
-d '{}'

spend_tracking_utils.py:443 - SpendTable: created payload - request_id: 02d0208138b1bbe065782bb99c59f745, model: claude-opus-4-6, spend: 0.0005825

and the spend row carries the real cost and token counts:

$ psql litellm_qa_lit4008_after2 -c 'SELECT call_type, model, spend, total_tokens, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE call_type = '"'"'aretrieve_batch'"'"';'
    call_type    |      model      |   spend   | total_tokens | prompt_tokens | completion_tokens
-----------------+-----------------+-----------+--------------+---------------+-------------------
 aretrieve_batch | claude-opus-4-6 | 0.0005825 |           61 |            18 |                43

$ psql litellm_qa_lit4008_after2 -c 'SELECT model_object_id, status, batch_processed FROM "LiteLLM_ManagedObjectTable";'
          model_object_id          |  status  | batch_processed
-----------------------------------+----------+-----------------
 msgbatch_011RLULF5pVTevaJP7k9dyCJ | complete | t

The dollar amount is exact: (18 input tokens x $5/M + 43 output tokens x $25/M) x 50% batch discount = $0.0005825. This batch uses no prompt caching, so the cache-detail aggregation added at this commit leaves prompt_tokens_details unset in the spend log's usage object and the token totals are unchanged

Type

🐛 Bug Fix

Changes

Anthropic Message Batches created through the proxy's /anthropic/v1/messages/batches passthrough never got cost attributed. When the CheckBatchCost background job saw a completed anthropic batch, it fetched the results with an OpenAI-shaped files call (POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id must have file_ prefix". The error body was silently wrapped as file content, parsed as a results file with zero successful rows, and the job wrote a $0 aretrieve_batch spend row and set batch_processed=true, making the $0 permanent

The fix lands at four layers. AnthropicFilesConfig.transform_file_content_request now routes msgbatch_ file ids to GET /v1/messages/batches/{id}/results, the endpoint where Anthropic actually serves batch results, so afile_content works for anthropic batch output files without any poller-side branching. BaseLLMHTTPHandler.retrieve_file_content (sync and async) now raises the provider error class on a non-2xx response instead of returning the error body as binary file content. litellm/batches/batch_utils.py now understands Anthropic's results JSONL shape: success is result.type == "succeeded" rather than response.status_code == 200, the response body is result.message, and usage is converted through the existing AnthropicConfig.calculate_usage helper so cache_creation_input_tokens and cache_read_input_tokens are counted. batch_cost_calculator now prices cache creation tokens at cache_creation_input_token_cost / 2 instead of folding them into the base input rate, keeping the 50% anthropic batch discount correct for base input, cache reads, cache writes, and output alike

CheckBatchCost also no longer treats a cost tracking failure as terminal. The completed-batch handling moved into _track_completed_batch_cost, and any results-fetch or cost-computation error now logs, records a cost_tracking_error metric, and leaves the row with batch_processed=false so the next poll retries, instead of the previous behavior where the swallowed error produced a permanently recorded $0

No *_batches pricing keys were added to the cost map on purpose: batch_cost_calculator's fallback already bills at half the regular token rate, which is exactly Anthropic's 50% batch discount, while the *_batches fast path ignores cache tokens entirely, so adding those keys would have priced cache-heavy batches worse

A follow-up commit also aggregates cache token details into the batch-level Usage: _get_batch_job_total_usage_from_file_content now sums per-row cache read and cache creation tokens and carries them on the aggregated usage (prompt_tokens_details.cached_tokens / cache_creation_tokens), so the aretrieve_batch spend log row exposes the cache split instead of a single prompt-token lump. Purely OpenAI-shaped batches without cache data keep prompt_tokens_details unset

Regression tests cover the msgbatch_ results-endpoint routing, the raise-on-HTTP-error behavior in retrieve_file_content, the anthropic JSONL success/usage/cost parsing (including exact 50% cache-aware pricing math), the cache-creation pricing in batch_cost_calculator, and that a failed cost tracking attempt leaves the managed object row unprocessed, and that the aggregated batch usage carries the summed cache read and cache creation token details

[greptile-apps]

Greptile Summary

Fixes Anthropic passthrough message batches being permanently recorded as $0 spend by correcting four compounding bugs in the CheckBatchCost background job. The core problem was that the job tried to fetch Anthropic batch results via the OpenAI Files API, which Anthropic rejected; the error body was silently wrapped as file content, parsed as an empty results file, and a $0 spend row was written with batch_processed=true — making the $0 permanent.

• Routing fix (AnthropicFilesConfig): msgbatch_ file IDs are now routed to GET /v1/messages/batches/{id}/results instead of the Files API; BaseLLMHTTPHandler now raises the provider error class on HTTP 4xx/5xx instead of silently returning the error body as binary content.
• Parsing fix (batch_utils): Anthropic batch results JSONL is now parsed correctly — success is result.type == \"succeeded\", the response body is result.message, and usage goes through AnthropicConfig.calculate_usage so cache tokens are counted properly.
• Pricing fix (batch_cost_calculator): Cache-creation tokens are now priced at cache_creation_input_token_cost / 2 rather than folded into the base input rate, keeping the 50% batch discount correct for all token types.
• Resilience fix (CheckBatchCost): Cost-tracking failures now log, record a cost_tracking_error metric, and leave batch_processed=false so the next poll retries, instead of permanently recording $0.

Confidence Score: 5/5

Safe to merge — the fix is narrowly scoped to Anthropic passthrough batch cost tracking, backed by live e2e evidence, and all changed code paths have dedicated unit tests.

Each of the four bug layers (endpoint routing, error propagation, JSONL parsing, cache token pricing) is independently verified by a targeted unit test, and the PR includes a live e2e run against the real Anthropic API showing the exact dollar amount matches the expected formula. The retry-on-failure change is safe because the only observable difference for previously passing batches is that a transient fetch error no longer permanently records $0.

No files require special attention.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Refactored completed-batch handling into _track_completed_batch_cost; errors now log, record a metric, and leave batch_processed=false for retry instead of permanently writing $0
litellm/llms/anthropic/files/transformation.py	Routes msgbatch_ file IDs to GET /v1/messages/batches/{id}/results instead of the Files API, which rejected them with a prefix error
litellm/llms/custom_httpx/llm_http_handler.py	Both sync and async retrieve_file_content now raise the provider error class on HTTP 4xx/5xx instead of silently returning the error body as binary file content
litellm/batches/batch_utils.py	Adds Anthropic-specific JSONL parsing (result.type == "succeeded", result.message) and aggregates cache read/creation tokens into the returned Usage; cost routing now correctly enters the batch_cost_calculator path for all Anthropic rows
litellm/cost_calculator.py	Fixed batch_cost_calculator to price cache-creation tokens at cache_creation_input_token_cost / 2 instead of folding them into the base input rate, keeping all token types at the 50% batch discount
tests/proxy_unit_tests/test_check_batch_cost.py	New test verifies that a failed cost-tracking attempt does not mark the job as processed
tests/test_litellm/batches/test_batch_utils.py	Comprehensive new tests cover Anthropic JSONL success/usage/cost parsing, cache-aware pricing math, and cache token aggregation in total usage
tests/test_litellm/llms/anthropic/files/test_anthropic_files_transformation.py	New tests confirm msgbatch_ IDs route to the batch results endpoint and that a custom api_base is respected
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	New tests verify that both sync and async retrieve_file_content raise the provider error class on HTTP 4xx responses
tests/test_litellm/test_cost_calculator.py	New tests verify cache-creation tokens are priced at the write rate in batch_cost_calculator, and the fallback to input_cost_per_token works when no write-rate key is present

_{Reviews (3): Last reviewed commit: "fix(batches): carry cache token details ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

The only concrete defect is in the new test test_calculate_batch_cost_and_usage_anthropic_end_to_end, which asserts models == ["claude-sonnet-4-5"] while the fixed code returns ["claude-sonnet-4-5-20250929"] extracted from the result file — this will fail in CI and must be corrected before the tests can validate the fix

_get_batch_models_from_file_content returns [model_name] when one is passed (the test passes model_name="claude-sonnet-4-5"), and the file's 76 tests all pass locally.

[mateo-berri]

@greptileai