fix(proxy): fetch background responses through the router in CheckResponsesCost

[devin-ai-integration]

TLDR

Problem this solves:

• Background /v1/responses rows stay queued forever
• Poll job re-fetches them every cycle, never attributes cost
• incomplete responses were also treated as non-terminal

How it solves it:

• Fetch through llm_router so deployment credentials apply
• Treat incomplete as terminal, like failed/cancelled
• Log the skip at warning level so it is visible

User Flow

Before: the background response finishes upstream, but the gateway never records its cost

1. Run the gateway with a model whose provider API key is set only in the config file, not exported in the gateway's environment
2. POST https://litellm-domain/v1/responses with {"model": "nano-c08", "input": "Reply with exactly one word: ping", "background": true, "store": true} returns 200 with a response id and status "queued"
3. Leave the response alone; upstream it finishes within a minute
4. Open https://litellm-domain/ui/?page=logs (or GET https://litellm-domain/spend/logs): only a $0.00 creation entry ever appears for that response, with no billed entry no matter how long you wait

After: the same background response shows up as billed spend shortly after it finishes

1. Run the gateway with a model whose provider API key is set only in the config file, not exported in the gateway's environment
2. POST https://litellm-domain/v1/responses with {"model": "nano-c08", "input": "Reply with exactly one word: ping", "background": true, "store": true} returns 200 with a response id and status "queued"
3. Leave the response alone; upstream it finishes within a minute
4. Open https://litellm-domain/ui/?page=logs a minute or two later: a billed entry for the response appears with its real token counts and cost, for this run 13 input and 5 output tokens billed at $0.00000885

Relevant issues

Fixes #35131

Linear ticket

Resolves LIT-2361

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxies against real OpenAI spend, one per build, each on its own fresh postgres database. Before runs at 729bec6 (litellm_internal_staging lineage, fix absent; the branch tip 972c0d0 is also pre-fix and identical in the touched region), after runs at this PR's head 55c392b. Config pins nano-c08 (openai/gpt-5.4-nano) with api_key: os.environ/OPENAI_API_KEY_C08QA and proxy_batch_polling_interval: 30

To reproduce the credential shape deterministically, both proxy environments export a poisoned OPENAI_API_KEY=sk-invalid-c08-qa, so SDK-level env resolution always 401s and only the router deployment's config credential can authenticate. One background response per leg, then hands off: no HTTP call touches the response afterwards, since a GET /v1/responses/{id} runs the full logging path and would write its own billed row, masking the observation. Observation is psql and proxy logs only

$ curl -s http://localhost:18231/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model": "nano-c08", "input": "Reply with exactly one word: ping", "background": true, "store": true}'
{"id": "resp_U_ffcZVcqgoRj85Ero8DhS8e50Bvx2COgvIQhcgDvmu5-...", "status": "queued", ...}

Before, the poll job swallows the credential failure every cycle and the row never moves, eight-plus consecutive cycles at a fixed 38s interval:

03:15:07 - LiteLLM Proxy:INFO: check_responses_cost.py:147 - Skipping job resp_U_ffcZVcqgoRj85Ero8D... due to error:
  litellm.AuthenticationError: OpenAIException - "Incorrect API key provided: sk-inval*****8-qa"
03:15:45 - ... (same, cycle 2)
...
03:19:33 - ... (same, cycle 8)

qa_c08_before> SELECT status FROM "LiteLLM_ManagedObjectTable";   -> queued
qa_c08_before> SELECT count(*) FROM "LiteLLM_SpendLogs" WHERE spend > 0;   -> 0

After, the first poll cycle fetches through the router deployment (succeeding despite the poisoned env var, which only the deployment-credential path can do), marks the row terminal, and bills it:

03:15:19 - LiteLLM Proxy:INFO: check_responses_cost.py:179 - Response resp_5BaBrYAXhz04fbz9EOPpDM_... has terminal status completed, marking as complete
03:15:19 - LiteLLM Proxy:INFO: check_responses_cost.py:190 - Marked 1 response jobs as completed

qa_c08_after> SELECT status FROM "LiteLLM_ManagedObjectTable";   -> completed
qa_c08_after> SELECT call_type, spend, prompt_tokens, completion_tokens FROM "LiteLLM_SpendLogs" WHERE spend > 0;
 aget_responses | 8.85e-06 | 13 | 5

The billed amount is exact for gpt-5.4-nano: 13 input tokens at 2e-07 plus 5 output tokens at 1.25e-06 is 8.85e-06 USD. A foreground request on the same config independently billed an identical 8.85e-06 row as a control

Leg	Commit	Row after 8+ poll cycles	Billed row
Before	729bec6	still queued	none
After	55c392b	completed on cycle 1	8.85e-06

Type

🐛 Bug Fix

Changes

CheckResponsesCost polls LiteLLM_ManagedObjectTable for queued / in_progress background responses and called litellm.aget_responses(response_id=...) directly. That SDK entrypoint only resolves credentials from provider env vars, while every deployment-scoped credential (Azure api_base / api_version, a config-only or secret-manager api_key) lives on the router deployment. So the fetch raised, the exception branch logged at info and continued, and the row never left queued; no cost was ever attributed and the same rows were re-polled every cycle. GET /v1/responses/{id} works for the exact same response because the proxy sends it through llm_router.aget_responses, which decodes the model_id embedded in the response id and applies that deployment's litellm_params.

The row stores the encrypted response id, whose plaintext is the LiteLLM-encoded id carrying model_id, so after the existing decrypt step the id is routable:

model_id = ResponsesAPIRequestUtils.get_model_id_from_response_id(response_id)
if model_id is None:
    return await litellm.aget_responses(...)   # nothing to route on, unchanged behavior
return await self.llm_router.aget_responses(...)  # deployment credentials + fallbacks

incomplete is a terminal status in the Responses API but was missing from the terminal set, which is a second way a row could be polled forever, so the status check is now a single TERMINAL_RESPONSE_STATUSES membership test over completed, failed, cancelled, incomplete. The skip log moved from info to warning: a row that cannot be fetched is a real misconfiguration and was previously invisible on default log levels.

Tests in tests/proxy_unit_tests/test_check_responses_cost.py cover a deployment-scoped id (both the raw encoded form and the encrypted form actually stored in the table) going through the router with litellm.aget_responses patched to raise if it is reached, an id with no model_id still falling back to the SDK, and an incomplete response transitioning the row out of queued. All three fail on litellm_internal_staging.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/c5c0ca2f9cae4d32a3a02a02fb1104ed

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This follow-up completes the background Responses API polling fix.

• Routes deployment-scoped response retrieval through the existing Router so configured credentials are applied.
• Falls back to the SDK when the response ID has no deployment or its deployment is no longer configured.
• Treats incomplete responses as terminal and raises fetch-failure logging to warning level.
• Adds regression coverage for routed, encrypted, fallback, missing-deployment, and incomplete-response cases.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py	Adds deployment-aware response retrieval with an SDK fallback and expands terminal-status handling.
tests/proxy_unit_tests/test_check_responses_cost.py	Adds focused regression tests covering router selection, encrypted IDs, SDK fallback, removed deployments, and incomplete responses.

_{Reviews (3): Last reviewed commit: "merge litellm_internal_staging" | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

LGTM