fix(batches): stop uncostable batches from starving the cost poll page

[devin-ai-integration]

TLDR

Problem this solves:

• Uncostable batches held poll slots forever
• Once they filled the page, newer batches went uncosted
• Terminal rows escaped the staleness sweep entirely

How it solves it:

• Retire rows with no model id in their unified id
• Retire rows the provider 404s as gone
• Sweep completed rows never costed past the staleness cutoff

User Flow

Before: a customer submitting daily batches sees new batches stop showing any spend, with no error anywhere

1. Over time some of their batches become uncostable, either because the id the gateway handed back carries no routable model or because the provider has since dropped its record of the batch
2. Once enough of those pile up, they submit a fresh batch: POST https://litellm-domain/v1/batches with an input_file_id and get back 202 with a scrambled batch id and "status": "validating"
3. They fire and forget, since the gateway's background cost job has always billed completed batches on its own (polling GET https://litellm-domain/v1/batches/{id} after completion is the one escape hatch, because retrieving a completed batch bills it on the spot)
4. They open https://litellm-domain/ui/?page=logs and there is no spend row for that batch, and none appears an hour later, or the next day
5. The only way out is to go into the database by hand and mark the dead batches processed

After: the dead batches drop out on their own and the fresh batch gets costed like it always did

1. Same starting point, with the same pile of uncostable batches
2. They submit the same POST https://litellm-domain/v1/batches and get the same 202 with "status": "validating"
3. They fire and forget just the same
4. https://litellm-domain/ui/?page=logs shows that batch at real non-zero spend within the first poll cycle after it completes
5. No manual database surgery, and the dead batches stay out of the way for good

Relevant issues

Fixes #36640

Linear ticket

Resolves LIT-5462

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Full before and after pair at this PR's final head 3b09484344, before at the exact merge base 5f2986a1f3 (port 30397), after at the head (port 47613). Each leg ran in its own fresh worktree with its own venv and a fresh database, a live proxy against the real OpenAI batch API, and PROXY_BATCH_POLLING_INTERVAL=60 with MAX_OBJECTS_PER_POLL_CYCLE=2 so two dead rows are enough to fill the poll page. The model id is pinned so the seeded 404 row resolves to a live deployment

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      id: qa-fixed-gpt4omini
      mode: batch

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

litellm_settings:
  require_managed_files: true

Both legs ran the identical script. Seed two dead rows 10 and 9 days old at terminal status complete with batch_processed=false: one whose unified id decodes to litellm_proxy;llm_batch_id:poison-no-model (no model id), one encoding model_id:qa-fixed-gpt4omini;llm_batch_id:batch_deadbeef0000000000000000000000 (404 at OpenAI). Submit a real batch, fire and forget. Once OpenAI reports it complete, seed two more identical dead rows minutes old, too young for the 7 day sweep, and submit a second real batch. The batches are never retrieved through the proxy, because retrieval bills inline on both versions and would mask the background job; completion is confirmed against the provider directly

printf '%s\n' '{"custom_id":"r1","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"say hi in 3 words"}],"max_tokens":20}}' > batch_input.jsonl
FID=$(curl -s http://localhost:<port>/v1/files -H "Authorization: Bearer sk-1234" \
  -F purpose="batch" -F target_model_names="gpt-4o-mini" \
  -F file="@batch_input.jsonl;type=application/jsonl" | jq -r .id)
BID=$(curl -s http://localhost:<port>/v1/batches -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d "{\"input_file_id\":\"$FID\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}" | jq -r .id)
PROVIDER_BID=$(python3 -c "import base64,sys;s=sys.argv[1];print(next(p.split(':',1)[1] for p in base64.urlsafe_b64decode(s+'='*(-len(s)%4)).decode().split(';') if p.startswith('llm_batch_id:')))" "$BID")
curl -s "https://api.openai.com/v1/batches/$PROVIDER_BID" -H "Authorization: Bearer $OPENAI_API_KEY" | jq '{id,status,completed_at}'
curl -s "http://localhost:<port>/spend/logs" -H "Authorization: Bearer sk-1234" | jq 'sort_by(.startTime) | map({call_type, spend, total_tokens, startTime})'

Before, merge base 5f2986a1f3

Both creates returned 202 with a unified id and "status": "validating", and OpenAI finished each batch within about a minute of submission

batch 1: batch_6a7e6ba3c1c8819092dcb86945621c09  completed_at 2026-08-14T01:14:15Z
batch 2: batch_6a7e6c25ac9081908a22a39bf31247f8  completed_at 2026-08-14T01:16:00Z

The final /spend/logs read at 01:37:36Z, about 20 poll cycles after batch 1 completed and 19 after batch 2, still holds nothing but the four $0 create rows. Neither batch is ever costed, the starvation in one screen

[
  { "call_type": "acreate_file",  "spend": 0.0, "total_tokens": 0, "startTime": "2026-08-14T01:13:06.755000Z" },
  { "call_type": "acreate_batch", "spend": 0.0, "total_tokens": 0, "startTime": "2026-08-14T01:13:07.606000Z" },
  { "call_type": "acreate_file",  "spend": 0.0, "total_tokens": 0, "startTime": "2026-08-14T01:15:17.130000Z" },
  { "call_type": "acreate_batch", "spend": 0.0, "total_tokens": 0, "startTime": "2026-08-14T01:15:17.532000Z" }
]

After, head 3b09484344

Same creates, same fire and forget

batch 1: batch_6a7e6be9fbfc8190b855df266788f734  completed_at 2026-08-14T01:16:14Z
batch 2: batch_6a7e6c88d6b08190ac0d58326e39bec0  completed_at 2026-08-14T01:17:50Z

The /spend/logs read at 01:23:19Z shows both batches costed by the background job with zero client retrievals at any point. Batch 1's spend row lands 29 seconds after provider completion, within one 60s cycle. Batch 2's lands 97 seconds after: the first post-completion cycle's two slots went to the fresh poison rows, retired on sight, and the very next cycle costed the batch

[
  { "call_type": "acreate_file",    "spend": 0.0,         "total_tokens": 0,  "startTime": "2026-08-14T01:14:16.753000Z" },
  { "call_type": "acreate_batch",   "spend": 0.0,         "total_tokens": 0,  "startTime": "2026-08-14T01:14:17.704000Z" },
  { "call_type": "aretrieve_batch", "spend": 0.000002475, "total_tokens": 18, "startTime": "2026-08-14T01:16:43.388000Z" },
  { "call_type": "acreate_file",    "spend": 0.0,         "total_tokens": 0,  "startTime": "2026-08-14T01:16:56.255000Z" },
  { "call_type": "acreate_batch",   "spend": 0.0,         "total_tokens": 0,  "startTime": "2026-08-14T01:16:56.688000Z" },
  { "call_type": "aretrieve_batch", "spend": 0.000002475, "total_tokens": 18, "startTime": "2026-08-14T01:19:27.324000Z" }
]

Kept from the earlier live run at c11ebbed27: screenshots and a screen recording of the Admin UI logs page showing the costed batch under the LiteLLM Proxy/CheckBatchCost user agent are in this Slack thread and in a comment below. That run also probed the other direction: a row pointed at an unreachable deployment kept retrying across 9 cycles and a raw non-base64 unified id was never retired, so only provably dead rows are dropped. A 404 alone is still not trusted, since retirement needs the error to name the batch id and the batch's own deployment to still be in the router, so a renamed Azure deployment or a fallback that never saw the batch keeps retrying

Observations from this run:

• Fresh retirements delayed batch 2's costing one cycle (this PR, bounded)
• Pre-fix, dead 404 rows re-hit OpenAI every cycle (this PR removes)

Type

🐛 Bug Fix

Caveats (if any)

• Transient failures still retry, bounded only by the staleness cutoff
• Old schemas without batch_processed retire by status instead
• Retrieving a completed batch already billed it inline, on both versions

Link to Devin session: https://app.devin.ai/sessions/d5c8c2d2e44f4033a4500e9cf415b306
Requested by: @mateo-berri

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes batch spend attribution and when rows stop being polled; incorrect retirement could drop legitimate spend, though 404 handling is narrowed and heavily tested.

Overview
Fixes LIT-5462: uncostable managed batch rows could fill every slot in the paginated CheckBatchCost poll (MAX_OBJECTS_PER_POLL_CYCLE), so newer batches never got costed or showed spend.

During each poll, rows that can never succeed are retired via _retire_job (sets batch_processed=True, or stale_expired on schemas without that column): unified IDs that decode but have no model_id, and provider 404s that explicitly name the batch when the batch’s deployment is still on the router. Misrouted 404s (missing deployment, wrong Azure deployment name, transient errors) are not retired so spend can still be recovered after config fixes.

Stale cleanup adds a second sweep for old complete/completed rows still at batch_processed=False, since the existing non-terminal staleness pass never touched them. Terminal status lists are centralized in TERMINAL_MANAGED_OBJECT_STATUSES.

Regression coverage lives in TestPollPageStarvation in the proxy unit tests.

^{Reviewed by Cursor Bugbot for commit 3b09484. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

The PR prevents uncostable managed batches from permanently occupying the bounded cost-poll page.

• Retires decoded unified batch IDs that lack routing information.
• Retires provider-missing batches after guarded 404 responses.
• Marks old completed, never-costed rows as processed after the configured staleness cutoff.
• Adds regression coverage for retirement, retry, compatibility, and starvation behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Adds bounded cleanup and targeted retirement paths for managed batch rows that otherwise block newer cost-poll work.
tests/proxy_unit_tests/test_check_batch_cost.py	Adds focused tests for stale cleanup, unroutable IDs, guarded provider 404s, transient retries, legacy schemas, and page advancement.

_{Reviews (4): Last reviewed commit: "refactor(batches): decode unified ids th..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

[devin-ai-integration]

🟡 A batch can be permanently abandoned for cost tracking when a temporary or misconfigured lookup returns "not found"

Any "not found" response from the provider is treated as permanent and the batch is dropped from future cost polling (_is_batch_gone_at_provider at enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py:749-750), so a batch whose lookup only fails because of a fixable configuration mistake never gets its spend recorded, even after the mistake is corrected.
Impact: Spend for real, completed batches can silently disappear from usage reporting with no way to recover it other than manual database edits.

How a fixable misconfiguration produces a permanent 404 retirement

_is_batch_gone_at_provider (enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py:237-245) matches any NotFoundError/openai.NotFoundError, not specifically "this batch id does not exist". Two concrete non-permanent sources of a 404:

1. Azure returns HTTP 404 DeploymentNotFound when the deployment name in config is wrong or was renamed; litellm maps that to NotFoundError.
2. If the deployment id encoded in the unified batch id is no longer in the router and default fallbacks are configured, Router._common_checks_available_deployment (litellm/router.py:10812-10826) silently substitutes the fallback model, and the retrieve then hits a provider that has never heard of this batch id, returning 404.

In both cases _retire_job sets batch_processed = True (enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py:206-215), which permanently excludes the row from the poll query at enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py:695-710, so restoring the deployment/config later does not bring the batch back for costing. Narrowing the check (e.g. requiring the provider error body to reference the batch id, or requiring the resolved deployment id to actually still exist in the router before treating the 404 as permanent) would avoid retiring recoverable rows.

Prompt for agents

In enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py, check_batch_cost now permanently retires a managed-object row whenever llm_router.aretrieve_batch raises any NotFoundError (see _is_batch_gone_at_provider). A 404 is not always evidence that the provider dropped the batch: Azure returns 404 DeploymentNotFound for a wrong/renamed deployment, and if the deployment id embedded in the unified batch id is missing from the router while default fallbacks are configured, Router._common_checks_available_deployment substitutes a fallback deployment whose provider has never seen this batch id, also yielding a 404. In both cases the row is marked batch_processed=True and can never be costed again, even after the operator fixes the config. Consider tightening the permanence test, for example by requiring that the resolved model_id still exists in the router (llm_router.has_model_id) before treating a 404 as permanent, and/or by only retiring after the 404 repeats across several poll cycles.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Good catch, tightened in 8947008: retirement now also requires the 404 to name the batch id, so Azure's DeploymentNotFound keeps retrying alongside your deployment check.

[devin-ai-integration]

Re-verified live with real OpenAI batches: dead rows retire on first sight, the page advances, the real batch gets costed, and connection errors keep retrying.

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c11ebbe. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit da84142. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8947008. Configure here.}

[mateo-berri]

LGTM

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3b09484. Configure here.}