fix(proxy): bill cancelled and failed batches that still produced an output file

[mateo-berri]

TLDR

Problem this solves:

• Cancelled/failed batches with an output file were retired unbilled
• Provider bills the completed requests, gateway records zero spend forever

How it solves it:

• Poller now bills any terminal batch that produced an output file
• Shared constant lists provider terminal batch statuses once
• Terminal batches whose advertised output file 404s retire first cycle

User Flow

Before: a developer cancels an in-progress batch, the provider still completes part of it and serves the results, and the gateway never records any spend for that work

1. They send POST https://litellm-domain/v1/files with purpose=batch and target_model_names to upload their requests JSONL
2. They send POST https://litellm-domain/v1/batches with the returned input_file_id and the batch starts at the provider
3. They send POST https://litellm-domain/v1/batches/{batch_id}/cancel and get HTTP 200; the provider finishes in-flight requests and finalizes the batch as cancelled with 142 of 2000 requests completed
4. GET https://litellm-domain/v1/batches/{batch_id} shows status cancelled, a populated output_file_id, and request_counts.completed 142
5. GET https://litellm-domain/v1/files/{output_file_id}/content returns HTTP 200 with 142 result lines carrying real token usage
6. GET https://litellm-domain/spend/logs?request_id={batch_id} shows only the $0 batch-creation row, and GET https://litellm-domain/key/info shows spend 0.0, permanently: the batch is never revisited

After: the same cancelled batch is billed for its completed work within one poll cycle

1. They send POST https://litellm-domain/v1/files with purpose=batch and target_model_names to upload their requests JSONL
2. They send POST https://litellm-domain/v1/batches with the returned input_file_id and the batch starts at the provider
3. They send POST https://litellm-domain/v1/batches/{batch_id}/cancel and get HTTP 200; the provider finishes in-flight requests and finalizes the batch as cancelled with partial completion
4. GET https://litellm-domain/v1/batches/{batch_id} shows status cancelled, a populated output_file_id, and non-zero request_counts.completed
5. GET https://litellm-domain/v1/files/{output_file_id}/content returns HTTP 200 with the completed result lines
6. Within one poll cycle GET https://litellm-domain/spend/logs?request_id={batch_id} gains a batch-retrieval row with the real cost and token counts of those completed requests, and GET https://litellm-domain/key/info shows the matching non-zero spend

Relevant issues

Fixes #37156

Linear ticket

Resolves LIT-5663

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy against the real OpenAI batch API, real $$$. Both legs ran the identical flow in their own worktree with a fresh Postgres DB, exported PROXY_BATCH_POLLING_INTERVAL=10 (the interval the reporter used), a random free port, and this config

model_list:
  - model_name: gpt-5.4-mini-batch
    litellm_params:
      model: openai/gpt-5.4-mini
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      mode: batch
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Note on statuses: the issue's headline expired case was already billed at the merge base (added by #35360), so the drop reproduces through cancelled, which hits the identical branch: OpenAI batches only expire after a real 24h window, while a cancel finalizes with a partial output file in minutes. A billed control leg during the repro (a completed 300-request batch at 3b6ae75) recorded aretrieve_batch spend 0.0048375, showing the pipeline prices this exact output-file accounting whenever the status is billable

The PR tip b34cbb1 is a base-sync merge touching only uv.lock, so 2f9d331 is the last behavioral commit. The core-fix commit 2198410 was also independently proven with the same flow (356/2000 completed, billed $0.0057405, tokens matching the output file)

Before (badd737, the merge base)

1. Create a key and upload a 2000-request JSONL for gpt-5.4-mini-batch:

   KEY=$(curl -s http://localhost:22679/key/generate -H "Authorization: Bearer $MASTER" -H 'Content-Type: application/json' -d '{"key_alias":"lit5663-qa-before","user_id":"lit5663-user","models":["gpt-5.4-mini-batch"]}' | jq -r .key)
   FILE=$(curl -s http://localhost:22679/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F 'target_model_names=gpt-5.4-mini-batch' -F file=@cancelleg2.jsonl | jq -r .id)
   BATCH=$(curl -s http://localhost:22679/v1/batches -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"input_file_id\":\"$FILE\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}" | jq -r .id)

2. Once the provider reports in_progress with completed >= 1 (here 33), cancel it: curl -s -X POST "http://localhost:22679/v1/batches/$BATCH/cancel" -H "Authorization: Bearer $KEY" returns HTTP 200 with status cancelling
3. Provider finalizes ~25 min later (batch_6a83619b03e481909e56da22a2670e17): "status": "cancelled", "output_file_id": "file-HzaqhkCafEGvQoTpB2QRrm", "request_counts": {"total": 2000, "completed": 142, "failed": 0}
4. curl -s "http://localhost:22679/v1/batches/$BATCH" -H "Authorization: Bearer $KEY" shows status cancelled with a populated (unified) output_file_id and counts 142/0/2000
5. GET /v1/files/{output_file_id}/content with the customer key returns HTTP 200 with 142 JSONL result lines totaling 2698 prompt + 568 completion = 3266 tokens of real billed work
6. curl -s "http://localhost:22679/spend/logs?request_id=$BATCH" -H "Authorization: Bearer $MASTER" shows only the $0 acreate_batch row (no aretrieve_batch row), and /key/info reports "spend": 0.0; both unchanged when re-checked 90 s later, after the proxy log's CheckBatchCost: marked job 93dfe2e2-eec4-47fd-8199-1cdb0669e4ad as cancelled in DB retired the batch for good

After (2f9d331, the PR's last behavioral commit)

1. Same key/file/batch creation as the before leg, port 43477:

   KEY=$(curl -s http://localhost:43477/key/generate -H "Authorization: Bearer $MASTER" -H 'Content-Type: application/json' -d '{"key_alias":"lit5663-qa-after2","user_id":"lit5663-user","models":["gpt-5.4-mini-batch"]}' | jq -r .key)
   FILE=$(curl -s http://localhost:43477/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F 'target_model_names=gpt-5.4-mini-batch' -F file=@cancelleg2.jsonl | jq -r .id)
   BATCH=$(curl -s http://localhost:43477/v1/batches -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"input_file_id\":\"$FILE\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}" | jq -r .id)

2. Cancel at in_progress with completed 19: curl -s -X POST "http://localhost:43477/v1/batches/$BATCH/cancel" -H "Authorization: Bearer $KEY" returns HTTP 200 with status cancelling
3. Provider finalizes 25 min later (batch_6a836348484c81909cd9ce50584baede): "status": "cancelled", "output_file_id": "file-8pYJexZwE5bM66N5TL9oAg", "request_counts": {"total": 2000, "completed": 437, "failed": 0}
4. curl -s "http://localhost:43477/v1/batches/$BATCH" -H "Authorization: Bearer $KEY" shows status cancelled with a populated (unified) output_file_id and counts 437/0/2000
5. GET /v1/files/{output_file_id}/content returns HTTP 200 with 437 JSONL result lines totaling 8303 prompt + 1748 completion = 10051 tokens
6. 11 seconds after provider finalization, curl -s "http://localhost:43477/spend/logs?request_id=${BATCH}_batch_cost" -H "Authorization: Bearer $MASTER" shows an aretrieve_batch row with "spend": 0.007046625 and tokens 8303/1748/10051, exactly matching the output file, and /key/info reports "spend": 0.007046625

Observations from the runs:

• Spend row request_id gains a _batch_cost suffix; leaves bug alone
• Poller ticks ~2x the configured interval; leaves bug alone
• OpenAI cancels finalize ~25 minutes later; leaves bug alone
• Proxy batch GETs can clobber tracking status; worsens pickup
• Before-leg poller computed cost yet wrote nothing; the bug

Type

🐛 Bug Fix

Caveats (if any)

Vertex AI advertises an output path for every batch, including terminal ones that never wrote it. Since 2f9d331 a terminal batch whose advertised output file 404s at the provider is retired unbilled on the first poll cycle instead of retrying until the staleness sweep. Ambiguous failures (the Vertex GCS downloader raises one error for 404, 403, and network blips alike) still retry and stay bounded by the 7-day sweep

A client that polls its batch through the proxy can still lose the spend: the status poll records the provider's terminal status on the gateway's tracking row, and the poller skips rows already recorded as cancelled or failed, so a poll landing between provider finalization and the next poller cycle (up to an hour apart by default) hides the batch from billing. That race predates this PR (at the merge base every ordering went unbilled, so this PR strictly improves and regresses nothing), needs its own fix (the poller's pickup set plus an upgrade guard against retroactively billing old terminal rows), and is filed with live proof as #37217

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes batch cost polling and spend attribution for terminal statuses; incorrect 404 detection could skip billing or retire jobs early, but behavior is scoped to CheckBatchCost with new unit tests.

Overview
Fixes CheckBatchCost so gateway spend matches provider batch work and terminal jobs do not poll forever.

Billing: Any provider-terminal batch (completed, complete, failed, expired, cancelled) with an output_file_id now goes through the same output fetch and aretrieve_batch cost path. Cancelled/failed batches that still produced partial results are billed instead of being finalized with zero spend.

Retirement: Terminal batches whose advertised output 404s at the provider (e.g. Vertex paths with no file) are finalized unbilled on the first cycle via new _is_output_file_gone_at_provider and _finalize_unbilled_terminal_job, instead of retrying until the staleness sweep.

Refactor: PROVIDER_TERMINAL_BATCH_STATUSES centralizes terminal status checks; unbillable terminal handling is consolidated into _finalize_unbilled_terminal_job (managed file IDs + batch_processed). Tests cover billing for expired/cancelled/failed with output and 404 output retirement.

^{Reviewed by Cursor Bugbot for commit b34cbb1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR bills failed and cancelled terminal batches when they produced output, while preserving unbilled finalization for terminal batches without output

• Adds a shared provider-terminal-status constant
• Routes terminal batches with output files through cost tracking
• Extracts terminal unbilled finalization and adds focused regression coverage

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Expands output-file billing to all provider terminal statuses and centralizes unbilled terminal-job finalization
tests/proxy_unit_tests/test_check_batch_cost.py	Updates terminal managed-file assertions and covers billing for expired, cancelled, and failed batches with output

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Terminal batches skipped by poller
  • Removed failed/expired/cancelled from the primary find_many not_in filter so the poller re-selects terminal rows written by retrieve_batch or cancel_batch and bills their output, relying on batch_processed=False to gate reprocessing.

Or push these changes by commenting:

@cursor push 1fbb43efed

Preview (1fbb43efed)

diff --git a/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py b/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
--- a/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
+++ b/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
@@ -786,19 +786,18 @@
         # every subsequent poll cycle.
         if self._has_batch_processed_column:
             try:
-                # Include "complete"/"completed" batches: the retrieve_batch
-                # endpoint may transition a batch to "complete" before
-                # CheckBatchCost runs.  The batch_processed=False filter
-                # already prevents reprocessing finished batches.
+                # Include every provider-terminal batch (complete/completed as well as
+                # failed/expired/cancelled): retrieve_batch or cancel_batch may write
+                # any of those statuses before CheckBatchCost runs, and a terminal batch
+                # with an output_file_id still needs to be billed. The batch_processed=False
+                # filter already prevents reprocessing finalized rows, so only stale_expired
+                # (retired by the fallback schema path) is excluded here.
                 jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
                     where={
                         "file_purpose": "batch",
                         "batch_processed": False,
                         "status": {
                             "not_in": [
-                                "failed",
-                                "expired",
-                                "cancelled",
                                 "stale_expired",
                             ]
                         },

diff --git a/tests/proxy_unit_tests/test_check_batch_cost.py b/tests/proxy_unit_tests/test_check_batch_cost.py
--- a/tests/proxy_unit_tests/test_check_batch_cost.py
+++ b/tests/proxy_unit_tests/test_check_batch_cost.py
@@ -174,13 +174,12 @@
         assert find_call[1]["order"] == {"created_at": "asc"}
         not_in = find_call[1]["where"]["status"]["not_in"]
         assert "stale_expired" in not_in
-        # "complete"/"completed" are intentionally NOT excluded from the
-        # primary query — the batch_processed=False filter is sufficient.
-        # This allows CheckBatchCost to pick up batches that were
-        # transitioned to "complete" by the retrieve_batch endpoint
-        # before CheckBatchCost had a chance to process them.
-        assert "complete" not in not_in
-        assert "completed" not in not_in
+        # No provider-terminal status is excluded from the primary query, the
+        # batch_processed=False filter is sufficient. This lets CheckBatchCost pick up
+        # batches that retrieve_batch or cancel_batch transitioned to complete/completed
+        # or failed/expired/cancelled before CheckBatchCost had a chance to bill them.
+        for terminal in ("complete", "completed", "failed", "expired", "cancelled"):
+            assert terminal not in not_in
         assert find_call[1]["where"]["batch_processed"] is False
         assert check_batch_cost_instance.batch_processed_support_confirmed is True

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b34cbb1. Configure here.}