fix: don't retire a completed batch from cost recovery while output_file_id is lagging

[syahra712]

TLDR

Problem this solves:

• A batch can report completed before its output file id appears
• A status check in that window wrote the batch off spend tracking forever
• The lost spend never reached logs or spend reports, with no retry

How it solves it:

• Retire a completed batch from cost recovery only when provably safe
• Safe means the output file arrived or zero request lines succeeded
• Unknown counts stay eligible so a later pass records the spend
• The background cost sweep now applies the same safe-to-retire check
• Regression tests pin the retire outcomes on both paths

User Flow

Before: a developer who checks their batch right as it finishes sees it complete and downloads results, but its spend never appears

1. They send POST https://litellm-domain/v1/batches with an uploaded input_file_id and get back "status": "validating" with a batch id
2. They poll GET https://litellm-domain/v1/batches/{batch_id} and see "status": "in_progress"
3. One poll lands just as the batch finishes: "status": "completed" with "output_file_id": null while request_counts shows completed lines
4. The next poll shows the same batch with output_file_id populated, and GET https://litellm-domain/v1/files/{output_file_id}/content downloads the results fine
5. They open https://litellm-domain/ui/?page=logs and the batch's spend never shows up, that day or ever

After: the same unlucky poll no longer costs the spend record, which appears once the output file id arrives

1. They send POST https://litellm-domain/v1/batches with an uploaded input_file_id and get back "status": "validating" with a batch id
2. They poll GET https://litellm-domain/v1/batches/{batch_id} and see "status": "in_progress"
3. One poll lands just as the batch finishes: "status": "completed" with "output_file_id": null while request_counts shows completed lines
4. The next poll shows the same batch with output_file_id populated, and GET https://litellm-domain/v1/files/{output_file_id}/content downloads the results fine
5. Shortly after, https://litellm-domain/ui/?page=logs shows the batch's real spend

Relevant issues

Fixes #37713

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

QA'd live at the PR tip af18f77db64ff47fd33f1c863d5bd19a1102e2d7 (after) against the merge base 490c9f9f3f (before). Each leg ran 2 proxy instances with --num_workers 2 (4 worker processes) sharing one Postgres, creating real OpenAI batches on gpt-5.4-mini that cost real money, with the proxy's api_base pointed at a local passthrough shim to https://api.openai.com that reproduces the reported provider lag deterministically: while a flag file exists, GET /v1/batches/{id} responses whose real upstream status is completed get output_file_id nulled, and everything else passes through untouched. Both cases confirmed the upstream batch was genuinely complete with a real output file id by calling api.openai.com directly outside the shim

Case A, the user-retrieve door

Both legs: upload a 2-line JSONL on instance A (POST /v1/files with purpose=batch and target_model_names), create the batch on instance B (POST /v1/batches), then poll alternating across both instances with the lag window on and background polling disabled until the unlucky poll lands (before leg shown, port 41283; the after leg observed the identical shape on its own ports)

curl -s "http://127.0.0.1:41283/v1/batches/$BATCH1" -H "Authorization: Bearer $LITELLM_MASTER_KEY"

{"status": "completed", "output_file_id": null, "request_counts": {"completed": 2, "failed": 0, "total": 2}, "model": "gpt-5.4-mini-2026-03-17"}

Spend at that point on both instances of both legs:

curl -s "http://127.0.0.1:<PORT>/spend/logs?request_id=${BATCH1}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"  ->  []

Then the lag flag is removed and both proxies restart with PROXY_BATCH_POLLING_ENABLED=true

Before (merge base 490c9f9): the spend never appears

curl -s "http://127.0.0.1:41283/spend/logs?request_id=${BATCH1}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"   # 2 min after reboot  ->  []
curl -s "http://127.0.0.1:52609/spend/logs?request_id=${BATCH1}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"   # 4 min after reboot  ->  []

Still [] on both instances 15 and 27 minutes later, and the batch GET keeps serving output_file_id: null forever

{"status": "completed", "output_file_id": null, "request_counts": {"completed": 2, "failed": 0, "total": 2}}

After (tip af18f77): exactly one spend row appears on both instances within 2 minutes

curl -s "http://127.0.0.1:42893/spend/logs?request_id=${BATCH1}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"

[{"request_id": "..._batch_cost", "spend": 2.625e-05, "model": "gpt-5.4-mini", "total_tokens": 30, "prompt_tokens": 22, "completion_tokens": 8, "call_type": "aretrieve_batch", "request_tags": ["User-Agent: LiteLLM Proxy", "User-Agent: LiteLLM Proxy/CheckBatchCost"]}]

The batch GET on the same instance now also shows a populated output_file_id, and the second instance (51237) returns the same single row: 1 row total across 2 instances x 2 workers, so no double billing

Case B, the background poller door

Both legs: proxies stay up with polling enabled (45s interval), lag flag back on, a fresh batch uploaded on B and created on A. The unlucky sighting reproduces ("status": "completed", "output_file_id": null, request_counts.completed: 2), the flag is held about 2 minutes past the first sighting so the poller runs multiple cycles against the lagging batch, then the flag is removed

Before (merge base): the poller wrote the batch off during the window, so removing the lag never brings the spend back

curl -s "http://127.0.0.1:41283/spend/logs?request_id=${BATCH2}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"   # +2 min   ->  []
curl -s "http://127.0.0.1:52609/spend/logs?request_id=${BATCH2}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"   # +5 min   ->  []

Still [] on both instances 16 minutes after the flag came off, with output_file_id still null on both. A control batch created on the same before rig with the lag flag off billed normally about 70 seconds after create (spend: 3.075e-05, tagged User-Agent: LiteLLM Proxy/CheckBatchCost), so the poller was alive at the merge base and the loss is attributable to the lag window alone

After (tip): exactly one spend row within two poll cycles of the lag clearing

curl -s "http://127.0.0.1:42893/spend/logs?request_id=${BATCH2}_batch_cost" -H "Authorization: Bearer $LITELLM_MASTER_KEY"

[{"request_id": "..._batch_cost", "spend": 3e-05, "model": "gpt-5.4-mini", "total_tokens": 30, "prompt_tokens": 20, "completion_tokens": 10, "call_type": "aretrieve_batch", "request_tags": ["User-Agent: LiteLLM Proxy", "User-Agent: LiteLLM Proxy/CheckBatchCost"]}]

Same single row on the second instance, and a stability pass one poll cycle later still shows exactly 1 row per batch per instance

Observations from the runs a reviewer cannot see in the diff, all pre-existing behavior this PR leaves untouched, except the last, which this PR adds:

• Terminal batch GETs serve the stored snapshot, not the provider
• Poller keys spend by unified id, retrieve by provider id
• Retrieve during validating writes a frozen zero-spend row
• A two-line OpenAI batch can complete upstream in eight seconds
• The new skip log line is info-level, invisible by default

Type

🐛 Bug Fix

Caveats (if any)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• af18f77 passes /live-pr-risk

---

Note

Medium Risk
Touches batch spend accounting (batch_processed retirement). The change is conservative (retry instead of drop spend) but incorrect gating could delay or skip billing.

Overview
Stops permanently retiring a completed batch from spend tracking when the provider reports status=completed but output_file_id is still missing.

Adds _completed_batch_safe_to_retire: retire only if the output file has arrived, or request_counts.completed == 0 (nothing to bill). Unknown counts stay eligible for the next poll.

The same gate is applied on the retrieve-path DB update (batch_processed=True) and on CheckBatchCost’s unbilled-terminal branch, so a lagging output id no longer drops the spend record forever. Regression tests cover both paths.

^{Reviewed by Cursor Bugbot for commit af18f77. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR completes the batch cost-recovery fix by retaining completed batches while their output file ID is lagging and retiring them only when output exists or no requests succeeded.

• Applies the same safe-retirement predicate to both database updates and the enterprise cost poller.
• Adds coverage for output-present, zero-success, successful-with-lagging-output, and unknown-count states.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported premature-retirement path is guarded in both relevant production paths, and the helper parameters are now typed.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	The terminal poller path now preserves completed jobs whose output file may still arrive, resolving the previously reported premature-retirement path.
litellm/proxy/openai_files_endpoints/common_utils.py	The typed helper centralizes the safe-retirement invariant and guards database updates from prematurely setting batch_processed.
tests/proxy_unit_tests/test_check_batch_cost.py	Poller tests distinguish zero-success terminal batches from lagging-output and unknown-count batches.
tests/test_litellm/proxy/openai_files_endpoint/test_files_common_utils.py	Helper-level tests cover all four retirement outcomes using typed fixture parameters.

_{Reviews (2): Last reviewed commit: "fix(check_batch_cost): leave a lagging-o..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing syahra712:fix/batch-processed-guard (af18f77) with litellm_internal_staging (490c9f9)}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit af18f77. Configure here.}

[mateo-berri]

LGTM. Thanks for the contribution!