feat(interactions): durable cross-pod settlement for background interaction billing

[devin-ai-integration]

TLDR

Problem this solves:

• Background interaction created on one replica, deleted on another: never billed
• Same instance with 2 workers: the delete bills only on the creating worker
• The creating process's poll dies with it, so a restart loses billing too
• A read of an interaction Gemini no longer knows answers 200 with every field null

How it solves it:

• The create writes a pending settlement row with its billing attribution
• Delete on any replica or worker claims that row once, then bills
• Proxy startup resumes the poll for every row nobody claimed
• The claim is one conditional update, so one spend row however many settlers race
• A delete elsewhere that cannot read the interaction fails instead of deleting it unbilled
• Provider credentials and request headers are never stored, and a settled row keeps only its outcome
• A read of a deleted or unknown interaction answers Gemini's 404 or 400 instead of 200 with null fields

User Flow

Before

The create is served by one proxy replica and the delete by another, and the interaction's usage never shows up in the key's spend; a read of an id Gemini no longer knows answers 200 with every field null

1. POST https://litellm-domain/v1beta/interactions with a virtual key and {"model":"gemini-3.8-flash","input":"Reply with the single word pong.","background":true} returns 200 with {"id":"v1_ChdJM2pB...","status":"in_progress","usage":null}
2. GET https://litellm-domain/v1beta/interactions/v1_ChdJM2pB... a few seconds later (the load balancer sends it to a different replica) returns 200 with "status":"completed" and "usage":{"total_tokens":78,...}
3. DELETE https://litellm-domain/v1beta/interactions/v1_ChdJM2pB... (also landing on the other replica) returns 200 with {"success":true,"id":"v1_ChdJM2pB..."}
4. GET https://litellm-domain/spend/logs?request_id=v1_ChdJM2pB... with the master key returns [], right after the delete and still a minute later
5. GET https://litellm-domain/key/info with the virtual key shows spend unchanged from before the create
6. GET https://litellm-domain/v1beta/interactions/v1_ChdJM2pB... after the delete returns 200 with "id":null,"status":null,"usage":null, and so does GET https://litellm-domain/v1beta/interactions/lpr-missing-11cf61ec01 with a mistyped id, so the client cannot tell a deleted interaction from one still running

After

The same create and delete on different replicas bills the interaction once under the key that created it, and a read of a deleted or mistyped id answers Gemini's error

1. POST https://litellm-domain/v1beta/interactions with a virtual key and {"model":"gemini-3.8-flash","input":"Reply with the single word pong.","background":true} returns 200 with {"id":"v1_ChduSGZB...","status":"in_progress","usage":null}
2. GET https://litellm-domain/v1beta/interactions/v1_ChduSGZB... a few seconds later (the load balancer sends it to a different replica) returns 200 with "status":"completed" and "usage":{"total_tokens":67,...}
3. DELETE https://litellm-domain/v1beta/interactions/v1_ChduSGZB... (also landing on the other replica) returns 200 with {"success":true,"id":"v1_ChduSGZB..."}
4. GET https://litellm-domain/spend/logs?request_id=v1_ChduSGZB... with the master key returns exactly one row: "model":"gemini/gemini-3.8-flash", "spend":0.00022725, "total_tokens":67
5. GET https://litellm-domain/key/info with the virtual key shows spend increased by 0.00022725
6. GET https://litellm-domain/v1beta/interactions/v1_ChduSGZB... after the delete returns 404 with litellm.NotFoundError: GeminiException and Gemini's Requested entity was not found. message, and GET https://litellm-domain/v1beta/interactions/lpr-missing-27d61d508e with a mistyped id returns 400 with Gemini's Invalid interaction name message

The same holds when the replica that served the create restarts before the interaction completes: once it is back, GET https://litellm-domain/spend/logs?request_id=<id> lists the row within one poll interval instead of never. A DELETE sent to a replica that cannot reach Gemini to read the interaction answers 500 instead of deleting it unbilled, and the creating replica's poll or the next delete bills it

Linear ticket

Resolves LIT-4442

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Last updated 2026-10-03 at 2cdc471. The product tip is 6afd57f; 57f2ede and 2cdc471 touch only tests/integration/, so the live legs and the /live-pr-risk run at 6afd57f and 57f2ede carry over to the tip, and the two cells those commits changed ran again at their commit

Every run below is the checked-in integration module tests/integration/spend/test_background_interaction_settlement.py plus one live cell against the real Gemini API, both legs on the same machine on 2026-10-02 (times UTC). Each leg owns a Postgres (port 54812, database audit41955_base or audit41955_head), a Redis (32815), a scripted Gemini upstream that records every call and the x-goog-api-key it carried (45569), and a 2-worker proxy booted from that leg's worktree (integration._support.proxy, base on 51315, head on 29294, Prisma db push at boot). Inside the module the rig fixture boots, from the same worktree, X = a 1-worker creator whose own poll never ticks inside a test (initial interval 300 s), Y = a 2-worker settler with a fast poll (1 s ticks, 8 s timeout), and single tests boot Z, P, or C replicas or SIGKILL Y's workers. The four deployments (in-progress, completed-at-once, failing-create, custom-priced) point at the module's upstream; the live cell's deployment carries no api_base and the real GEMINI_API_KEY, so its calls go to generativelanguage.googleapis.com and cost real money. Base is the merge base b28ce93; head is the PR's product tip 6afd57f, with the tests-only commits 57f2ede and 2cdc471 on top. Both legs ran under a loaded machine (headroom readings CRITICAL or TIGHT throughout), which matters for the one flaky cell explained under the matrix

Before (b28ce93)

Cross-replica delete on real Gemini (L1)

1. POST /v1beta/interactions on replica A with the virtual key and {"model":"integration-3c0025dd8da04be185d6af3727f2e64e","input":"Reply with the single word pong.","background":true} (that model alias is the scenario's deployment of gemini/gemini-3.8-flash) returns 200 with "id":"v1_ChdJM2pBYXB...", "status":"in_progress", "usage":null
2. GET /v1beta/interactions/v1_ChdJM2pBYXB... on replica B returns 200 with "status":"completed" and "usage":{"total_tokens":78,...}
3. DELETE /v1beta/interactions/v1_ChdJM2pBYXB... on replica B returns 200 with {"success":true,"id":"v1_ChdJM2pBYXBHbEZfYUYxTWtQcXVlS3dBMBIXSTNqQWFwR2xGX2FGMU1rUHF1ZUt3QTA"}
4. SELECT ... FROM "LiteLLM_SpendLogs" WHERE request_id = '<id>' 20 s later returns []; the settlement table does not exist on this side
5. GET /key/info with the virtual key shows "spend": 0.0

Read of a deleted or mistyped id on real Gemini

1. GET /v1beta/interactions/<the id deleted above> returns 200 with "id":null,"status":null,"usage":null after Gemini answered 404 Requested entity was not found. through the recorder
2. GET /v1beta/interactions/lpr-missing-11cf61ec01 returns 200 with the same null body after Gemini's 400 Invalid interaction name; GET /interactions/<id> behaves the same on both reads
3. DELETE /v1beta/interactions/<unknown id> returns 404 (control, unchanged on both sides)

Integration matrix

1. python tests/integration/run.py accounting --seed 4106601 --order-seed 0 --results audit/base-b28ce93d2b-run1 tests/integration/spend/test_background_interaction_settlement.py against the base proxy
2. 2026-10-02T19:30:44Z: 28 cells, 18 failed in 831.283 s; every red is the merge base's missing settlement (relation "LiteLLM_BackgroundInteractionSettlement" does not exist, State did not converge: [] on a cross-replica bill, a fail-closed delete answering 200), see the Base column of the matrix

After (2cdc471)

Cross-replica delete on real Gemini (L1)

1. POST /v1beta/interactions on replica A with the virtual key and {"model":"integration-f1e5fe1483644b16b291a1bcf4b471f5","input":"Reply with the single word pong.","background":true} (that model alias is the scenario's deployment of gemini/gemini-3.8-flash) returns 200 with "id":"v1_ChduSGZBYXI...", "status":"in_progress", "usage":null
2. GET /v1beta/interactions/v1_ChduSGZBYXI... on replica B returns 200 with "status":"completed" and "usage":{"total_tokens":67,...}
3. DELETE /v1beta/interactions/v1_ChduSGZBYXI... on replica B returns 200 with {"success":true,"id":"v1_ChduSGZBYXI3aU9iR0ktc0FQcTZhMzZRURIXbkhmQWFyN2lPYkdJLXNBUHE2YTM2UVE"}
4. SELECT ... FROM "LiteLLM_SpendLogs" WHERE request_id = '<id>' 20 s later returns exactly one row: "spend":0.00022725, "prompt_tokens":8, "completion_tokens":59, "call_type":"acreate_interaction"; the settlement row reads claimed_by: Mateos-MacBook-Pro.local:35570 (replica B's worker), outcome: billed, claimed and settled
5. GET /key/info with the virtual key shows "spend": 0.00022725

Read of a deleted or mistyped id on real Gemini

1. GET /v1beta/interactions/<the id deleted above> returns 404 litellm.NotFoundError: GeminiException carrying Gemini's Requested entity was not found.
2. GET /v1beta/interactions/lpr-missing-27d61d508e returns 400 carrying Gemini's Invalid interaction name; GET /interactions/<id> behaves the same on both reads
3. DELETE /v1beta/interactions/<unknown id> returns 404 (control, unchanged on both sides)

Integration matrix

1. python tests/integration/run.py accounting --seed 4106601 --order-seed 0 --results audit/head-6afd57f9de-run1 tests/integration/spend/test_background_interaction_settlement.py against the head proxy, then the same command a second time into -run2
2. 2026-10-02T19:30:44Z: 28 cells, 2 failed in 583.581 s; 2026-10-02T19:49:08Z: 28 cells, 1 failed in 392.156 s
3. The two tests-only commits each changed one cell, and that cell ran again on both legs at its commit with the same seeds, head twice and base once, per the rerun rule for a commit that touches no product file: H5 at 57f2ede (base 2026-10-02T20:06:03Z, 2 of 2 failed, 246.362 s; head 2026-10-02T20:07:14Z, 1 of 2 failed, 166.401 s, and 2026-10-02T20:11:22Z, 1 of 2 failed, 129.173 s); E5 at 2cdc471 (base 2026-10-02T20:21:07Z, 1 of 1 failed, 106.410 s; head 2026-10-02T20:19:01Z, 0 of 1 failed, 141.600 s, and 2026-10-02T20:22:58Z, 0 of 1 failed, 202.452 s). The other 26 rows keep their 6afd57f results, since no product file changed after it

/audit matrix

Cell ids follow the inventory: H = happy paths, S = sad paths, E = edge cases, U = unsettled, C = chaos, L = live. "Base expected" is what the merge base should do (red where the cell asserts a settlement row or a cross-replica bill, which main cannot produce). A row is VERIFIED when base matched that expectation and both head runs were green

Cell	Test	Base expected	Base	Head run 1	Head run 2	Verdict	Ran at
H1a	test_creator_poll_bills_a_completed_background_interaction_once	green	green	green	green	VERIFIED	6afd57f
H1b	test_creator_poll_records_its_settlement_durably	red	red (test_creator_poll_records_its_settlement_durably=FAIL)	green	green	VERIFIED	6afd57f
H2	test_delete_on_another_replica_bills_the_creators_interaction_once[/v1beta/interactions]	red	red (/v1beta/interactions=FAIL)	green	green	VERIFIED	6afd57f
H2b	test_delete_on_another_replica_bills_the_creators_interaction_once[/interactions]	red	red (/interactions=FAIL)	green	green	VERIFIED	6afd57f
H3	test_delete_of_a_failed_interaction_releases_without_a_spend_row	red	red (test_delete_of_a_failed_interaction_releases_without_a_spend_row=FAIL)	green	green	VERIFIED	6afd57f
H4	test_delete_of_a_requires_action_interaction_bills_its_usage	red	red (test_delete_of_a_requires_action_interaction_bills_its_usage=FAIL)	green	green	VERIFIED	6afd57f
H5	test_a_replica_booting_later_resumes_and_bills_unclaimed_interactions	red	red (test_a_replica_booting_later_resumes_and_bills_unclaimed_interactions=FAIL)	green	green	VERIFIED	57f2ede
H6	test_deletes_on_the_creating_proxy_bill_each_interaction_once	red	red (test_deletes_on_the_creating_proxy_bill_each_interaction_once=FAIL)	green	green	VERIFIED	6afd57f
H7	test_custom_deployment_pricing_bills_at_the_deployment_rate_on_another_replica	red	red (test_custom_deployment_pricing_bills_at_the_deployment_rate_on_another_replica=FAIL)	green	green	VERIFIED	6afd57f
H8	test_cancel_then_delete_on_another_replica_releases_without_a_spend_row	red	red (test_cancel_then_delete_on_another_replica_releases_without_a_spend_row=FAIL)	green	green	VERIFIED	6afd57f
S1	test_delete_fails_closed_when_the_settling_replica_cannot_fetch	red	red (test_delete_fails_closed_when_the_settling_replica_cannot_fetch=FAIL)	green	green	VERIFIED	6afd57f
S2	test_delete_of_an_interaction_the_vendor_purged_sends_no_delete_and_keeps_the_row	red	red (test_delete_of_an_interaction_the_vendor_purged_sends_no_delete_and_keeps_the_row=FAIL)	green	green	VERIFIED	6afd57f
S3	test_reading_an_interaction_never_bills_it	red	red (test_reading_an_interaction_never_bills_it=FAIL)	green	green	VERIFIED	6afd57f
S4	test_delete_of_an_odd_or_unknown_id_is_refused_and_the_proxy_keeps_serving	green	green x4	green x4	green x4	VERIFIED	6afd57f
S5	test_a_missing_settlement_table_leaves_in_process_billing_intact	green	green	green	green	VERIFIED	6afd57f
S6	test_a_failed_create_registers_nothing	green	green	green	green	VERIFIED	6afd57f
E1	test_polling_disabled_replica_registers_nothing_and_never_bills	green	green	green	green	VERIFIED	6afd57f
E2	test_a_create_that_completes_at_once_is_billed_by_the_create_alone[synchronous]	green	green	green	green	VERIFIED	6afd57f
E3	test_a_create_that_completes_at_once_is_billed_by_the_create_alone[background]	green	green	green	green	VERIFIED	6afd57f
E4	test_identical_creates_settle_as_separate_interactions	red	red (test_identical_creates_settle_as_separate_interactions=FAIL)	green	green	VERIFIED	6afd57f
E5	test_settlement_on_another_replica_releases_the_creators_budget_reservation	red	red (test_settlement_on_another_replica_releases_the_creators_budget_reservation=FAIL)	green	green	VERIFIED	2cdc471
U1	test_a_poll_that_never_sees_a_terminal_status_records_unsettled_and_releases	red	red (test_a_poll_that_never_sees_a_terminal_status_records_unsettled_and_releases=FAIL)	green	green	VERIFIED	6afd57f
C1	test_an_upstream_outage_fails_deletes_closed_and_every_interaction_bills_once_after_recovery	red	red (test_an_upstream_outage_fails_deletes_closed_and_every_interaction_bills_once_after_recovery=FAIL)	green	green	VERIFIED	6afd57f
C2	test_killed_workers_leave_their_polls_to_the_respawned_workers	red	red (test_killed_workers_leave_their_polls_to_the_respawned_workers=FAIL)	green	green	VERIFIED	6afd57f
C3	test_concurrent_deletes_on_a_slow_upstream_settle_exactly_once	red	red (test_concurrent_deletes_on_a_slow_upstream_settle_exactly_once=FAIL)	green	green	VERIFIED	6afd57f

Every red in the Base column is the merge base missing the feature the cell checks: relation "LiteLLM_BackgroundInteractionSettlement" does not exist where a cell reads the table, State did not converge: [] where it waits for a bill that only a settler on another replica or a resumed poll can write, and a 200 where head fails closed. Two cells needed a second look

H5 (a replica booting later resumes and bills the rows nobody claimed) failed its first head run at 6afd57f and passed the second: the failing run's claimant was a worker pid outside the resumer's process group. The creator X runs one uvicorn worker, and under the loaded machine the supervisor's liveness ping timed out, logged Child process died, and respawned it; the respawned worker ran the same boot resume the cell is about and claimed the rows first, which is the designed behavior for any worker that boots. 57f2ede makes the cell accept a claimant that is either one of the resumer's workers or a process that booted after the creates, and the cell then passed both head runs and kept its base red

E5 (a settlement on another replica releases the creator's budget reservation) was wrong twice in the test, never in the product. At 6afd57f it expected 400 where the proxy answers 422 budget_exceeded since bf804f5 on main. At 57f2ede it sent the DELETE with the pinned key, and that DELETE is refused with the same 422 on both legs: the pre-call reservation is resized down to whatever budget is left, so the key's counted spend equals its max_budget and the auth budget check refuses every later request with that key until something releases the reservation (the pinned-key caveat). 2cdc471 asserts that refusal and deletes with a second key; the settlement then bills the creator's key from the stored attribution and releases its reservation, so the third create with the pinned key answers 200 on head (both runs) and base never gets there (no cross-replica bill, so the spend row the cell waits for never appears)

S2 and S3 are red on base at their tails rather than at the first assertion: the caller sees the same 404 on both legs, but base forwards the DELETE without a pre-delete GET and has no settlement row, and in S3 the DELETE on Y never bills on base

/live-pr-risk at 57f2ede, carried to 2cdc471 (a tests-only commit)

Verdict: two observable behavior changes, both chosen in this PR and recorded below, and no unintended ones. Across 22 scenarios run in the same order on a two-worker base proxy and a two-worker head proxy, every difference traces to those two changes, and a third leg on the merged tree of the head and origin/main answered every scenario the way the head did (compare-head-merge: 22 rows, none differ). The real vendor saw the same eight calls with the same header set and body sizes from both legs (GET 400 x2, POST 200, GET 200 x3, DELETE 200, GET 404), so the head sends nothing extra outbound. A background create costs about 4.5 ms more at the median against a local scripted upstream, 6.0 ms on base against 10.5 ms on head over 12 interleaved creates each (p90 9 ms against 13 ms), which is the settlement row's insert before the create answers

Rig: base at the merge base b28ce93 on port 51315 and head at 57f2ede on port 29294 (product-identical to the tip 2cdc471, since the two commits after it touch only tests/integration/), each a two-worker uvicorn proxy booted from its own worktree with PYTHONPATH set to that worktree, its own Postgres database and its own Redis namespace; a third two-worker leg on port 46870 from the merged tree of 57f2ede and origin/main at 2ccb7ed (commit-tree 0341813e41); a scripted upstream on port 45569 for the hostile inputs (unknown ids, a 404, a 500, a 500-then-200 outage, a 500 only for the pre-delete fetch) and a forwarding recorder in front of generativelanguage.googleapis.com carrying the real Gemini key for the vendor scenarios, which logs method, status, header set and body size of every outbound call; baselines for chat, streaming chat, /v1/messages, readiness and key info ran on every leg and matched. The tests/llm_translation/interactions suite ran against the head with the real key: 16 passed, 3 skipped (the suite's own skips), and 2 creates failed on the vendor's 503 service_unavailable on POST /v1beta/interactions and passed when rerun 3 minutes later

Breaking

1. A DELETE of a background interaction on a replica that did not create it fails when the pre-delete fetch fails, where main deleted it unbilled. Observed with the scripted upstream answering 500 to that fetch: base answered 200 and sent only the DELETE upstream, with no spend row on any replica; head answered 500 internal_server_error, sent only the GET, and left the settlement row unclaimed (claimed_by null, outcome null) for the next settler. On the creator replica both legs answer 200 and send GET then DELETE (head's row ends released). Edge: the settler on the deleting replica rebuilds the billing context from the stored row and fetches the interaction before the vendor delete, and a failed fetch is raised to the caller instead of falling through to the delete. Chosen in 73a2dc9 and recorded as Decision 15, approved by mateo-berri in the LGTM review of 2026-09-24 at 2ba9b7b, which already carried it
2. GET /v1beta/interactions/{id} and GET /interactions/{id} answer the vendor's non-2xx status where main answered 200 with every field null. Observed on both routes: an unknown id is 404 litellm.NotFoundError: GeminiException on head and 200 with null fields on base; a scripted upstream 500 is 500 internal_server_error on head and 200 null on base; the real vendor's 400 for a malformed id and its 404 Requested entity was not found. for a deleted id come back as 400 and 404 on head and as 200 null on base. Edge: the Gemini interactions GET transform raises GeminiError with the vendor's status and body on a non-2xx response, which the route maps through the usual exception mapping. Introduced on purpose in 44c3026 and recorded as Decision 19; mateo-berri's standing approval predates that commit

Backward incompatible

• SDK: litellm.aget_interaction and get_interaction for gemini/ and vertex_ai/ raise the mapped exception where they returned an InteractionsAPIResponse with status=None. Observed against the real vendors: Gemini raised BadRequestError (vendor 400 for a malformed id) on head and returned the null object on base; Vertex raised BadRequestError on head and returned the null object on base, where the vendor's 400 was Unsupported location for the rig's region rather than an unknown-id answer, so the observation is the raise against the return and not the 404 shape. Same commit and decision as Breaking 2
• Internal API of litellm/interactions/background_cost_polling.py: maybe_schedule_background_interaction_cost_polling is now a coroutine, maybe_settle_background_interaction takes delete_kwargs, the pre-delete fetch helper is renamed, and the background_interaction_settled kwarg no longer exists. grep -rn over litellm/, enterprise/ and tests/ finds no reader of the kwarg and no caller of these helpers outside the interactions handlers and their tests; an out-of-tree caller would fail at call time. The unit tests in CI and the 18 passing translation tests cover the in-tree callers
• Schema: the new table LiteLLM_BackgroundInteractionSettlement via migration 20260915000000_add_background_interaction_settlement, dated before main's latest 20261002220000. Verified on a fresh database that took the merge base's 197 migrations first: prisma migrate deploy from the head applied the new one on top and the table exists, since Prisma applies every unapplied migration regardless of its date. A deployment with DISABLE_SCHEMA_UPDATE applies it itself (caveat row)
• Credentials: the settler fetches with the delete request's credentials, never the create's, so a settlement from another replica needs the same environment credential the GET and DELETE routes already need on main; the audit cells assert the environment credential on every settler call. The config-only-key case is a caveat row below, unchanged from main

Regression risk

• Boot with a large unclaimed backlog: startup resumes every unclaimed row before the proxy serves, which the cells and the live cell exercised with a handful of rows and a killed worker (C2), never with thousands; a seeded table of N rows and a timed boot would show whether a backlog delays readiness
• Fields the rebuilt billing context does not carry: custom pricing is carried (cell H7 bills the custom rate 0.00224 from the other replica) and api_base is empty on the settled row (caveat row), but per-key and per-team callbacks, the privacy toggles, trace controls and data_residency were not exercised; a cell with a key-scoped callback and a check of the settled row's logging payload would decide each
• Database failures other than a missing table: S5 covers the table going missing mid-run (the delete still succeeds, the bill is left to the creating poll); a connection failure or a timeout during the claim is not exercised and the claim's failure branch decides whether a row is billed twice or never
• Reservation release without Redis: the rig ran both legs with Redis, so the key's budget reservation release on the settling replica without Redis is untested
• Rolling restart: a new worker's boot resume and an old worker's still-running poller can both settle the same row; the claim is one conditional update and C3 shows exactly one row under concurrent deletes, but the poller-against-resume race itself was not forced
• In-memory store for SDK users: InMemoryBackgroundSettlementStore keeps one None entry per claimed interaction for the process lifetime (record_outcome is a no-op and claim leaves the key), measured at 152 bytes per entry, 15.2 MB per 100k interactions; main dropped its _ACTIVE_POLLS entry when the task ended. The proxy installs the Prisma store and is unaffected; caveat row below and a follow-up ticket
• A CancelledError during the create's insert, a read replica lagging behind the create, the trace id contextvar set from the delete's context, polling disabled by config, SDK create and delete parity, and reading a stored context written by a later version are listed under Not verified with what each needs

Dependency graph

Dependent	Mark	Evidence
GET transform raises on non-2xx	verified-live	Breaking 2, both routes, scripted and real
Poll and pre-delete fetch on 4xx/5xx	verified-live	poll outage scenario, delete-500 scenarios
Migration dated before main's latest	verified-live	fresh database, deploy from base then head
Startup resume before serving	tested	cells C2 and boot resume, live L1
Backlog of thousands at boot	untested	Regression risk
Settlement table absent	tested	cell S5, DISABLE_SCHEMA_UPDATE caveat
Insert on the create path	verified-live	latency probe, +4.5 ms median
Circular metadata after the upstream create	unreachable	proxy metadata is JSON; SDK only
CancelledError during the insert	untested	Not verified
One store read per poll tick	untested	count not measured
Store read on every delete	verified-live	unknown-id delete 404 on both legs, cell S4
Cross-replica settle	verified-live	audit matrix, live L1, lpr non-creator delete
Fail-closed delete	verified-live	Breaking 1
Settler credentials	tested	cells at lines 415, 522, 541
Rebuilt context: custom pricing	tested	cell H7
Rebuilt context: api_base	tested	empty on the settled row, caveat
Rebuilt context: callbacks, privacy, trace, residency	untested	Regression risk
Reservation release without Redis	untested	Regression risk
Poller against boot resume after a rolling restart	untested	C3 covers concurrent deletes only
Resume scheduled before Redis and callbacks are wired	untested	Not verified
Store failure other than a missing table	untested	Regression risk
Read replica lag	untested	Not verified
Rows kept after settlement	tested	caveat row, outcome kept, context dropped
In-memory store growth	tested	measured, Regression risk
Plaintext JSONB context	untested	caveat row; no credentials or headers stored by design
Trace id contextvar from the delete's context	untested	Not verified
Internal API signatures	tested	translation suite, unit tests in CI
background_interaction_settled kwarg removed	verified	no reader in litellm/, enterprise/, tests/
Polling disabled by config	untested	Not verified
Claim unavailable at the terminal fetch, then back	tested	audit deadline cells, Decision 13
Exactly-once claim	verified-live	cell C3, live L1
SDK get parity	verified-live	Gemini and Vertex, Backward incompatible
SDK create and delete parity	untested	Not verified
Stored context written by a later version	untested	Not verified

Not verified

• Boot latency with a large backlog of unclaimed rows: needs a seeded table and a timed boot
• CancelledError while the create's insert is in flight: needs a client that disconnects during the create against a slowed database
• Number of store reads per poll tick: needs a counting store or query log on the poller
• Callbacks, privacy toggles, trace controls and data_residency on the rebuilt context: needs a key-scoped callback cell
• Reservation release without Redis: needs a no-Redis leg of the rig
• The poller-against-resume race after a rolling restart: needs a worker restarted while its poller is mid-flight on a reachable interaction
• Resume scheduled before Redis and the callback list are wired at boot: needs a boot log read of the startup order with a pending row
• Store connection failures and timeouts during the claim: needs a chaos cell that cuts the database mid-settlement
• Read replica lag between the create's insert and a delete elsewhere: needs a replica with induced lag
• Plaintext JSONB request context at rest: no run, the caveat row names what is stored
• Trace id contextvar set from the delete's context: needs a trace sink wired to both replicas
• Polling disabled by config: needs a leg with background polling off
• SDK create and delete parity: only SDK get ran against the real vendors
• A stored context written by a later version of the model: needs a forward-compat fixture
• The Vertex unknown-id 404 shape: the vendor rejected the rig's region before looking at the id, so only the raise against the return was observed
• The wording of the settler's debug log line on a failed fetch: observed status codes only, not the log text

Type

🐛 Bug Fix

Caveats (if any)

Low

• A replica dying between its claim and the bill loses that one billing
  • The claim and the spend write are not one transaction, and re-polling a claimed row would double bill whenever the crash came after the spend write, so settlement is at-most-once by design; the only real fix is an idempotent spend write, which is beyond this PR
  • A billing error after a successful claim ends the same way with the process alive: the reservation is released, the poll task raises, and the row stays claimed with no outcome, which is how an operator tells it apart from a settled one
• A delete during a settlement-table outage proceeds upstream unbilled
  • That interaction's usage is lost and its budget reservation ends through the TTL or the give-up release once the table is back; settling without the claim can double bill under a partial outage (Decision 13)
• A pending row that cannot be written at create time logs an ERROR and settles through whatever the store answers when the poll claims
  • The create still returns. When the row landed anyway, the creator's poll claims through it like any other settler. When the store answers that no row exists (an insert that never landed, or a replica without the settlement table), the creator's poll bills through a local gate no other process can reach, so a delete of it on another replica or worker goes unbilled exactly as on main. When the store stays unreachable through the poll's whole timeout, the poll ends without a bill and a landed row waits for the next boot's resume (Decision 16)
• A proxy booted against a database without the new table logs the mismatch and bills in-process only
  • With the migration held back by DISABLE_SCHEMA_UPDATE=true, boot logs the schema mismatch with its CREATE TABLE, the ERROR repeats on every background create, reads of the missing table count as no rows so the create's own poll still bills, and nothing else about the proxy changes
• A database unreachable for a poll's whole timeout leaves the row unclaimed for the next boot
  • The give-up path cannot claim either, so the poll ends without billing or releasing, and the reservation ends through its TTL
• A rolling restart polls one interaction from every worker of the restarted replica plus the creator, and a whole-fleet restart from every worker in the fleet
  • Each worker resumes every unclaimed row at boot while the creator's poll is still alive; the claim keeps the bill single and the extra polls are GETs against the provider until the first terminal read settles the row
• Settled rows stay in the new table as an audit trail with their context emptied, and there is no retention job
  • A settled row keeps the interaction id, provider, claimant, and outcome, so no key, user, end-user, IP, or user-agent attribution outlives the bill; a row claimed by a replica that died before settling keeps its context, since that is what an operator needs to settle it by hand; a settled row is small
• A bill settled elsewhere carries key, team, user, and end-user attribution but no request headers
• The stored context carries the key's metadata verbatim, callback settings an admin put on the key included, in the new table's JSONB until the bill settles
  • It is the same plaintext the key table already holds in the same database, copied once per background create and emptied on settlement
• Every in-flight background interaction costs each polling worker one primary-key read per poll interval, from 5s doubling to 60s
  • That read is what lets a poll stop as soon as another replica settled the row
• A pending row this version cannot parse is skipped with an ERROR on every boot until something settles or removes it
• Only interactions created with background: true take this path; foreground ones bill as before
• A retried background create writes a $0 spend row that hides the poll's billed row; pre-existing
  • When the first upstream attempt of a create fails and the proxy retries it, the failure handler runs first and the create-time cost is 0 instead of deferred, so the $0 row lands under the interaction id and the poll's billed row is later dropped as a duplicate; the key's spend and the settlement row are still right. Seen once in an earlier After run at 73a2dc9, on a path this PR does not touch
• A spend row settled off the creating worker carries an empty api_base
  • The stored context does not carry the deployment's api_base, so a bill written by a delete on another replica or worker, or by a poll resumed after a restart, shows api_base: "" in /spend/logs, while creator-settled rows show the deployment's base as before. Cost, tokens, model, and key attribution are unaffected. Ships as is by mateo's approval on this PR (the LGTM review), and the one-key fix is tracked in LIT-8573
• main was merged in at 4c2604a after GitHub reported the PR conflicting with main's move of tests/test_litellm/interactions/test_background_cost_polling.py into tests/unit (test: move tests/test_litellm root and small trees into tests/unit #43186); the PR's own diff is unchanged
  • The PR's diff against main before and after the merge is byte-identical apart from hunk offsets. The merge first moved the new settlement test next to its sibling under tests/unit/proxy/spend_tracking/, which the proxy-db shards claim file by file in .circleci/scripts/unit_selection.sh, so no CI shard ran it and codecov/patch dropped to 81.61%; 5ef919d moves it back under tests/test_litellm/proxy/spend_tracking/, which the proxy-infra shard collects whole
• main was merged in again at 6ce60b9 after GitHub reported the PR conflicting with main's move of tests/test_litellm/proxy to tests/unit/proxy (ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests #42903) and with the Lens models main added next to the settlement model in schema.prisma (refactor(lens)!: rename internal engine code and API #44034); main's new lint gates also needed five small adaptations in the PR's code, made in the same commit
  • The merge itself leaves the PR's diff unchanged (1694 added and removed lines against main before and after, with the settlement test's path renamed). The settlement test lives under tests/unit/proxy/spend_tracking/ now, which the proxy-infra GitHub Actions shard collects whole since ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests #42903, and the three schema.prisma copies carry the settlement model right before the Lens models
  • The adaptations: three # mutable-ok markers that no longer suppressed anything are deleted (LIT013), the two stacked-for comprehensions in _json_values and _pending_rows are split into a named intermediate and chain.from_iterable (LIT014), _create_context validates the logging object's untyped call_type, metadata, and litellm_params through pydantic TypeAdapters before reading them, and adelete's **kwargs is typed object (five reportUnknownArgumentType errors that main's basedpyright budget no longer has room for). None changes what any input does: the validated values are the same str and mappings the old code iterated, and the context model already rejected a non-str call_type (Decision 18).
• main was merged in a third time at eee6988 after GitHub reported the PR conflicting with main again (82 commits past 0f6a06a, with tests/integration/_support/upstream.py, tests/integration/_support/process.py, the three schema.prisma copies, and proxy_server.py changed on both sides); the PR's own diff is unchanged
  • The one conflict was in the scripted upstream test helper, where main refactored the AWS event-stream frame builder and reflowed the observation call the PR extends with the Gemini x-goog-api-key header; the merge keeps main's helper and the PR's observation. Unlike the two earlier merges, no proof is carried over this one: the audit matrix and the /live-pr-risk sections below ran at eee6988 against the new merge base b28ce93
• A key whose budget is filled by its own background create's reservation is refused on every route until a settlement or the reservation TTL frees it, its own DELETE and GET of that interaction included; pre-existing on main
  • The pre-call reservation is resized down to whatever budget is left, so the key's counted spend equals its max_budget and the auth budget check (spend >= max_budget) answers 422 budget_exceeded to every later request with that key. The audit's budget cell shows it on both legs (the pinned key's DELETE on another replica is 422 on base and head alike), so the cell deletes with a second key; a settlement by any key then bills the creator's key and releases its reservation, which is what this PR adds
• GET and DELETE of a background interaction, and so a settlement from another replica, need the Gemini credential in the environment (GEMINI_API_KEY); a deployment whose only key lives in the model config answers 500 litellm.APIConnectionError: Google API key is required on both routes, on main and on this branch alike
  • The request names no model, so the routes resolve the provider credential from the environment and the settler fetches with the delete's credentials. The earlier QA at f39a56b showed the read on the creating replica, the read on the other replica and the delete on the other replica all 500 on Before and After (21:03Z, config_cd.yaml with the key only in the deployment), and the audit cells at this tip assert the environment credential on every settler call. Pre-existing on main; a fix is a credential lookup by the stored model, out of this PR's scope
• The osv-scan workflow is red at the tip on GHSA-vfj7-8cjw-p6xm (npm braces 3.0.3, a dev dependency in ui/litellm-dashboard/package-lock.json, no fixed version published). This PR does not touch that lockfile and every other branch scanned in the same hour fails on the same finding (for example https://github.com/BerriAI/litellm/actions/runs/37087700586), while main's last scan before the advisory passed (https://github.com/BerriAI/litellm/actions/runs/36975463472). It is not a required check and is left for the dependency bump that fixes it on main
• The SDK's default in-memory settlement store keeps one entry per claimed background interaction for the process lifetime, 152 bytes each (15.2 MB per 100k interactions, measured with tracemalloc)
  • InMemoryBackgroundSettlementStore.claim leaves a None entry and record_outcome is a no-op, where main dropped its in-process poll entry when the task ended. The proxy installs the database store and is unaffected; a follow-up ticket covers dropping settled entries from the in-memory store
• A background create does one settlement row insert before it answers: about 4.5 ms more at the median on the audit rig, 6.0 ms on base against 10.5 ms on head over 12 interleaved creates each against a local scripted upstream (p90 9 ms against 13 ms)
  • Measured with the same two-worker proxies and the same scenario on both legs; a real vendor create takes hundreds of milliseconds, so the insert is a small share of what the caller sees

Decisions

1. The pending row is written before the create returns, so a delete on any replica can find it; a failed write is logged and the create still returns, and the create's own poll learns at claim time whether the row landed (Decision 16)
2. The stored context is only what billing needs: the create's model, call ids, start time, provider, the user_api_* attribution metadata minus the auth object, and the deployment's custom pricing keys. Each value must be JSON; a non-JSON value is dropped rather than stringified
3. Provider credentials are never stored. The settling replica fetches with the delete request's own credentials, and a resumed poll uses the replica's environment, exactly like the in-process poll did
4. The claim is one conditional update on claimed_at IS NULL; there is no heartbeat or lease renewal, which is why a crash between claim and bill is the first Low caveat above rather than a re-poll, which would double bill whenever the crash came after the spend write
5. Startup schedules a poll for every unclaimed row before serving traffic; each resumed poll keeps the remaining share of the original timeout and always gets at least one fetch, so a completed interaction is still billed however late the resume comes
6. Settled rows are kept with their outcome, settlement time, and claimant instead of deleted, so an operator can audit what was billed or released; the same update empties the stored request context, so attribution sits in the row only for as long as a settler might need it
7. A float start_time on the logging object (only test fixtures pass one; the proxy always passes a datetime) is read as UTC epoch seconds
8. The poll schedule and the fetch call are injected, so tests drive real code with millisecond intervals instead of monkeypatching module constants
9. The SDK default store stays in memory; only the proxy installs the database store, so non-proxy users see no change
10. QA runs two proxy processes with --num_workers 2 each: that is the shape most customers deploy, and it is what showed the symptom also bites inside one instance, where the delete only bills on the worker that served the create
11. The request headers and the raw proxy request are never copied into the settlement row: a bill settled on another replica or after a restart carries key, team, user, and end-user attribution and nothing from the original HTTP request. requester_custom_headers was dropped from the carried keys as a dead key, since the enterprise logging path rebuilds it from the raw request at logging time and no settling replica has that request
12. A failure to install the database store at boot (a generated Prisma client without the new table) logs a warning and keeps the in-process store. A client whose table is missing or whose database is down installs the store anyway, and each operation then answers on its own with its own log line (the startup listing leaves rows for the next boot, a create's registration logs an ERROR, a read of a missing table counts as no rows so the create's own poll bills in process, and a claim against a database that is down owns nothing), so the proxy starts and bills the way it did before this PR
13. A claim the store cannot answer owns nothing: the poll retries on its next interval, and a delete proceeds upstream unbilled. A poll that already fetched the terminal usage but could not claim before its deadline settles from that usage when its final claim succeeds, instead of recording the interaction as unsettled and dropping the bill. Alternatives considered: settle without a claim (double bills under a partial outage when the creator's poll can still claim, and leaves an unclaimed row that the next boot resumes into a second reservation release and a misleading unsettled record); settle in-process when this process holds the live poll (the same leftover row); stash the pre-delete usage for the local poll to bill once a claim succeeds (mutable cross-task state for an outage-only window, and no help when the delete lands on a non-creator replica)
14. An interaction created with an agent and no model reaches the poll with no model name, exactly as on main, so the stored context carries that None instead of rejecting the create: the first live rig pass at 93d56fb answered such a create with a 500 after the provider had already accepted it, while main answered 200 and billed it at 0.0 with an empty model. Fixed in 59c2ce4 with a regression test
15. A delete on a replica other than the creator that cannot fetch the interaction with the delete's credentials fails with that fetch's error and never reaches the provider, so the creating replica's poll keeps the bill and a retried delete that can fetch settles first. Alternatives considered: delete anyway and mark the row unsettled (drops the bill, the loss Greptile flagged at ac68891); retry the fetch inside the delete (a second retry layer over the router's own, slower deletes, and the same outcome when the provider is down). On the creating replica a failed pre-delete fetch keeps main's release-then-delete, since its in-process poll ends with the delete
16. A registration that raises no longer bills through a private in-process gate. The poll's claim asks the durable store first and falls back to a local gate only when the store answers that no row exists, so a write that raised after its row had landed (a connection dropped after the commit) claims through that row like every other settler, and a resumed poll after a restart or a delete on another replica finds it claimed instead of billing the interaction a second time. A missing settlement table reads as no rows, so a replica running without the migration still bills in process with one ERROR per create. A store unreachable through the poll's whole timeout ends without a bill, exactly like a registered row, and the landed row, if any, is resumed at the next boot. Alternatives considered: a single read-back right after the failed write (Greptile's finding at 0a31f7b: a read-back that failed during the same outage put the poll on the in-process gate while the landed row stayed unclaimed for every later boot's resume to bill again); insert-on-claim (a protocol change every store implements, for a window the deferred claim already closes); a best-effort durable claim after the in-process bill (still a race with a concurrent settler)
17. A delete on a worker that resumed the poll after a restart takes the same fail-closed path as any non-creator: only the poll the create itself started keeps main's release-then-delete on a failed pre-delete fetch, because after a fleet restart every worker holds a resumed poll and Decision 15 would otherwise apply nowhere. Such a delete fails with the fetch's error and a retry that can fetch settles first
18. _create_context validates the logging object's call_type, metadata, and litellm_params through pydantic TypeAdapters (str and Mapping[str, object]) instead of reading the untyped attributes directly. A non-str call_type already failed the context model's own validation and a non-mapping already failed the .items() call, so the validated copies change no outcome; what they clear is the five reportUnknownArgumentType errors main's basedpyright budget no longer has room for. Alternatives considered: typing Logging.call_type and Logging.litellm_params at their source (a change to the core logging class far outside this PR); building the context with model_validate on a dict (one validation, but the field names leave the type checker's sight); str() and dict() coercions (a coercion where a validation that raises on a wrong shape is the honest one)
19. The Gemini GET transform raises GeminiError with the vendor's status and body on a non-2xx fetch (44c3026). AsyncHTTPHandler.get never raises for status and the transform only raised when the body was not JSON, so a 404 for an unknown or deleted id and a 500 with Gemini's JSON error body both parsed as an interaction with every field null: the proxy GET route answered 200 with null fields, the poll loop read a null status and kept polling, and a delete on another replica read that null status as "fetched", claimed the settlement as released, and forwarded the delete, so the fail-closed branch of Decision 15 was unreachable and the bill was lost (the audit's S1 and S2 cells at 6ce60b9, settlement row settle-in-progress-8309a672-7bb1e9e5 claimed with outcome released). The delete transform already raised the same way. Alternatives considered: checking the status in the shared get_interaction and async_get_interaction handlers (every provider's GET would change at once and _handle_error only maps httpx.HTTPStatusError); raise_for_status inside AsyncHTTPHandler.get (every caller of the shared HTTP client would change); keeping the null-field response and teaching the three readers to treat a null status as an error (three readers to keep in step, and the GET route would still answer 200 for a vendor error)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

High Risk
Changes multi-replica spend tracking, budget reservations, and delete semantics, plus a new migration and exactly-once claim logic where store or claim failures can leave interactions unbilled or block deletes.

Overview
Background interaction billing no longer relies on an in-process flag on the replica that created the interaction. Creates register a pending settlement (billing attribution in JSON, no provider credentials) in a pluggable store; the proxy uses a new LiteLLM_BackgroundInteractionSettlement table with a conditional claimed_at update so poll, delete, and post-restart resume settle exactly once across workers and pods.

Delete on any replica can load that row, rebuild the logging context, pre-fetch with the delete’s credentials, then bill or release budget reservations. If that fetch fails on a non-creator replica, the delete fails closed instead of deleting unbilled. Proxy startup installs the Prisma store and resumes unclaimed rows. The SDK default stays in-memory.

Gemini interaction GET now raises on non-2xx responses (instead of 200 with null fields). Scheduling cost polling is async and registers the durable row before returning.

^{Reviewed by Cursor Bugbot for commit 2cdc471. Bugbot is set up for automated code reviews on this repo. Configure here.}

• 57f2ede passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_durable_background_interaction_settlement (2cdc471) with main (2ccb7ed)}

[codecov]

Codecov Report

❌ Patch coverage is 92.37805% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/interactions/background_cost_polling.py	90.66%	21 Missing ⚠️
litellm/interactions/main.py	0.00%	2 Missing ⚠️
...pend_tracking/background_interaction_settlement.py	97.89%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Critical risk] Adds durable cross-process billing settlement for background interactions.

The PR appears safe to merge based on the reviewed changes and the absence of outstanding findings.

Summary

The PR adds durable, cross-worker settlement for background interaction billing. Since the previous review, it also makes failed Gemini interaction fetches raise errors and adds multi-process integration coverage for deletes, recovery, and budget release.

• No new actionable issue was established in the changes since the previous review.
• The previous Greptile findings are resolved, withdrawn, or otherwise not outstanding.

_{Reviews (18) · Last reviewed commit: "test(integration): delete the pinned key..."}

[greptile-apps]

Comments Outside Diff

These findings could not be posted inline.

•  Request Path Queries Database litellm/interactions/main.py:182 ▶

  Awaiting settlement registration adds a direct database write before every pollable create returns, while delete settlement similarly performs a direct lookup. This violates the repository directive forbidding direct database queries in critical request paths and must be resolved before merging.

  Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2cdc471. Configure here.}