test(e2e): pin openai_passthrough routing, cost logging, and file list isolation

[mateo-berri]

TLDR

Problem this solves:

• Four shipped passthrough fixes had zero e2e coverage
• The dedicated /openai_passthrough prefix could silently start 500ing again
• Two passthrough routes could go back to billing nothing

How it solves it:

• Five live tests, one per shipped fix, real OpenAI traffic
• Cost tests reconcile the logged row against the response the caller was served
• The file-list test asserts the page's cursors, which is where the leak was

User Flow

Before: a customer fronting OpenAI's own file, batch, and inference APIs through the gateway could not reach the file and batch routes at all, was handed other tenants' file IDs while listing their own, and was billed by OpenAI for inference the gateway never counted

1. They send POST https://litellm-domain/openai_passthrough/v1/files with a JSONL batch input file, and get back 500 {"error":{"message":"'openai_passthrough' is not a valid LlmProviders"}}, so the request dies inside the gateway and never reaches OpenAI
2. They send GET https://litellm-domain/openai_passthrough/v1/batches to list their batches and get the same 500 with the same message
3. They fall back to GET https://litellm-domain/v1/files, and the data array correctly holds only their own files, but first_id and last_id are OpenAI file IDs like file-3LBv6nhR1frSAFNKYtc13U that belong to other tenants on the same provider account, and has_more says there is another page
4. They send POST https://litellm-domain/openai_passthrough/v1/responses with "stream": true, read back a completed response with "id": "resp_0f6eb485..." and 22 input plus 33 output tokens, then look it up at GET https://litellm-domain/spend/logs?request_id=resp_0f6eb485... and find nothing; the only nearby row is keyed chatcmpl-<random uuid> and reads $0
5. They send POST https://litellm-domain/openai_passthrough/v1/embeddings, get a 200 with 1536 dimensions and 14 prompt tokens, and GET https://litellm-domain/spend/logs?request_id=... returns no row at all, so their key's spend never moves and a budget over this traffic never trips

Because the cursors in step 3 are real provider file IDs, any caller could take one and use it directly on the file routes to address a file another tenant uploaded

After: the same journey works end to end, the list page only ever names files the caller can see, and every relayed call is billed

1. POST https://litellm-domain/openai_passthrough/v1/files returns 200 with OpenAI's own file object: "object": "file", "purpose": "batch", and a "bytes" count matching what was uploaded
2. GET https://litellm-domain/openai_passthrough/v1/batches returns 200 with OpenAI's own page: "object": "list" and their batches in data
3. GET https://litellm-domain/v1/files returns first_id and last_id addressing rows inside their own data, both null when they own no files yet, and "has_more": false
4. The same streamed POST https://litellm-domain/openai_passthrough/v1/responses is billed under the very ID the caller was served: GET https://litellm-domain/spend/logs?request_id=resp_0f6eb485... returns a row with a real cost and the same 22 and 33 token counts the response reported
5. The same POST https://litellm-domain/openai_passthrough/v1/embeddings writes a row with a nonzero cost and the same 14 prompt tokens, so the key's spend moves and budgets see the traffic

A caller can no longer obtain another tenant's provider file ID from their own list page, so there is nothing to replay against the file routes

Relevant issues

Pins the fixes for #36086, #36087, #36523, and #36646

Linear ticket

Resolves LIT-5870

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR adds tests only, so no product code differs between the two sides. Before is the merge base showing these behaviors have no coverage; After is the live run at the tip proving the responses and spend rows those tests assert are the ones a real proxy produces

Shared setup: a proxy from this branch on http://localhost:33320 against a local Postgres, two deployments registered from config (gpt-5.5 and text-embedding-3-small, both openai/), a virtual key from POST /key/generate, and PROXY_BATCH_WRITE_AT=3 so spend rows land inside the poll window. No config flags beyond that; every route below is reachable on a stock proxy. All traffic hits real OpenAI

Before (02e67cd, the current merge base)

1. git show 02e67cd71595:tests/e2e/llm_translation/test_passthrough_e2e.py | grep -c OpenAIPassthrough -> 0, so nothing covers the dedicated prefix or the cost of what it relays
2. git show 02e67cd71595:tests/e2e/batches/test_batches_e2e.py | grep -c test_list_page_cursors -> 0, so nothing covers the file-list cursors either
3. Grepping both tests/e2e/coverage_registry/*.yaml at that commit for the five ids added here returns 0 for every one, so a regression in any of these ships with a green suite

After (8b7c801, still the commit the live run was driven from)

Case 1: the dedicated prefix reaches OpenAI's file API (#36086)

1. curl -s -X POST $P/openai_passthrough/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@qa-batch.jsonl -> 200 {"object":"file","id":"file-PHRLJeXcFPBySUUA3uNHyy","purpose":"batch","filename":"qa-batch.jsonl","bytes":28,"created_at":1787219767,"expires_at":1789811767,"status":"processed"}
2. curl -s -X DELETE $P/openai_passthrough/v1/files/file-PHRLJeXcFPBySUUA3uNHyy -H "Authorization: Bearer $KEY" -> 200 {"object":"file","deleted":true,"id":"file-PHRLJeXcFPBySUUA3uNHyy"}, so the round trip cleans up after itself

Case 2: the dedicated prefix reaches OpenAI's batch API (#36086)

1. curl -s $P/openai_passthrough/v1/batches -H "Authorization: Bearer $KEY" -> 200 {"object":"list","data":[{"id":"batch_6a86cb2412f481909c667f345e202adf","object":"batch","endpoint":"/v1/chat/completions","model":"gpt-4o-mini-2024-07-18","input_file_id":"file-3LBv6nhR1frSAFNKYtc13U","status":"cancelled"}]}, OpenAI's own page relayed verbatim

Case 3: the file list's cursors name only the caller's own files (#36087)

1. curl -s -X POST $P/key/generate -H "Authorization: Bearer $MASTER" -d '{"user_id":"e2e-qa-lit5870-isolated"}' mints a key that owns nothing
2. curl -s $P/v1/files -H "Authorization: Bearer $KEY" -> 200 {"data":[],"has_more":false,"object":"list","first_id":null,"last_id":null}. The provider account demonstrably holds files belonging to others, as Case 2's page shows file-3LBv6nhR1frSAFNKYtc13U; before the fix those upstream IDs came back as this caller's first_id and last_id next to an empty data, which is exactly the leak. A fresh key is the strongest case because data is empty, so any non-null cursor here can only have come from someone else

Case 4: a streamed passthrough Responses call is billed under the ID the caller was served (#36523)

1. curl -sN -X POST $P/openai_passthrough/v1/responses -H "Authorization: Bearer $KEY" -d '{"model":"gpt-5.5","input":"Say hi in one word. e2e-qa-lit5870-c18","stream":true}' -> 200, 11 SSE events, x-litellm-call-id: bb105500-99cb-4db4-aac6-48d15b6b7e02, and the response.completed frame carries "id": "resp_0f6eb485f57e5868006a86cfa4f94087d094b03894149004f2" with "usage": {"input_tokens":22,"output_tokens":33,"output_tokens_details":{"reasoning_tokens":26},"total_tokens":55}
2. curl -s "$P/spend/logs?request_id=resp_0f6eb485f57e5868006a86cfa4f94087d094b03894149004f2" -H "Authorization: Bearer $MASTER" -> {"request_id":"resp_0f6eb485...","model":"gpt-5.5","spend":0.0011,"prompt_tokens":22,"completion_tokens":33,"call_type":"pass_through_endpoint","custom_llm_provider":"openai","status":"success"}. The row is keyed by the provider ID the caller actually read, and its 22 and 33 match the response exactly

Case 5: passthrough embeddings write a priced spend row (#36646)

1. curl -s -X POST $P/openai_passthrough/v1/embeddings -H "Authorization: Bearer $KEY" -d '{"model":"text-embedding-3-small","input":"cost this sentence e2e-qa-lit5870-c20"}' -> 200, x-litellm-call-id: b1eee2e9-a4f2-4525-86df-c30a0c9f7b9d, object=list, usage={"prompt_tokens":14,"total_tokens":14}, 1536 dimensions
2. curl -s "$P/spend/logs?request_id=b1eee2e9-a4f2-4525-86df-c30a0c9f7b9d" -H "Authorization: Bearer $MASTER" -> {"request_id":"b1eee2e9-a4f2-4525-86df-c30a0c9f7b9d","model":"text-embedding-3-small","spend":2.8e-07,"prompt_tokens":14,"completion_tokens":0,"call_type":"pass_through_endpoint","custom_llm_provider":"openai","status":"success"}. The tests assert spend > 0 rather than mere row existence, because an unmapped model logs a row at 0.0

All five tests also ran green against this same proxy at this commit: 4 passed in 13.94s for the passthrough file and 1 passed in 3.10s for the file-list test

Type

✅ Test

Caveats (if any)

• Three of the six ticket items are not e2e reachable, see below
• Cost rows are polled to a deadline, never slept on once
• The list-page test is strongest on a key that owns nothing
• No red-before leg: the tests are shown green, never shown failing against the pre-fix code
• The embeddings model name is hardcoded in the test module rather than read from e2e_config

Each test is proven by running green against a live proxy, real OpenAI, and a real database, not by reverting the four product fixes and watching it go red. Reverting them means unpicking four merged PRs on a base that has moved a long way since, so the evidence here is the assertions themselves plus the pasted responses they read. The sharpest of the four is the file-list case: the batch page in Case 2 shows the provider account really does hold file-3LBv6nhR1frSAFNKYtc13U, which belongs to another caller, so a null cursor next to an empty data is a narrowing the gateway had to do rather than an account that happened to be empty

The list-page test mints a key on a fresh user_id, so data is empty and the cursors have to be null. That is on purpose: before the fix an empty data still came back with the upstream org's first_id and last_id. It does mean assert listed.has_more is not True is unconditional: it reads as "the proxy never forwards a cursor upstream" rather than "this page's cursors are consistent", so whoever implements real pagination pass-through will have to revisit that line

EMBEDDING_MODEL sits in test_passthrough_e2e.py as a literal, while its sibling CHEAP_OPENAI_MODEL comes from e2e_config with an E2E_CHEAP_OPENAI_MODEL override. Nothing breaks today, since the passthrough route relays the model name to OpenAI verbatim and text-embedding-3-small is what OpenAI calls it, but an environment that needs a different embeddings model has no way to say so

Three items in the ticket's scope table have no new test here, for three different reasons:

#35551 (raw provider file IDs bypassed ownership checks) needs litellm_settings.require_managed_files: true. That is a boot-time setting with no per-request, per-key, or management-route override, and the shared e2e stack does not run with it. Turning it on is a change to that stack's configuration, which lives outside this repo, so it has to be sequenced ahead of the test rather than requested by it

#36151 (WebSocket passthrough not registered) is a websocket route, and the shared transport has no websocket seam. Worse, the fixed path closes with 1008 and 1011, which an HTTP client reads as a plain 403, the same status the unregistered route returned before the fix, so a test written over HTTP would pass identically on both sides and prove nothing

The over-correction guard on /openai/v1/files, which must still reach the managed-file route rather than being swallowed by the new prefix, is already covered and needs nothing new. tests/e2e/batches/test_batches_e2e.py::test_batch_key_model_access_denied uploads through that exact route and unwraps the result, so it goes red the moment the route stops answering. A separate test for it was drafted and then dropped as a duplicate; its registry row llm.files.openai.upload.nonstream.works was already claimed by the batches suite, so no coverage was lost. Worth noting for anyone reproducing locally: that route answers 500 {"error":{"message":"files_settings is not set, set it on your config.yaml file."}} unless your own config declares files_settings, which the shared stack does and a minimal local config does not

QA runbook

• tests/e2e/llm_translation/test_passthrough_e2e.py::TestOpenAIPassthroughPrefix::test_passthrough_prefix_uploads_a_file_to_openai - the dedicated prefix relays a file upload to OpenAI instead of 500ing on a bad provider name

  • Generate a key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{}'
  • curl -X POST http://localhost:4000/openai_passthrough/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@some.jsonl
  • Expect 200 with "object": "file", "purpose": "batch", and "bytes" equal to the file's size. Assert on object, not the file- prefix: with general_settings.passthrough_managed_object_ids: true the ID is rewritten to litellm_proxy:...
  • Clean up with curl -X DELETE http://localhost:4000/openai_passthrough/v1/files/$FILE_ID -H "Authorization: Bearer $KEY"
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_passthrough_e2e.py::TestOpenAIPassthroughPrefix::test_passthrough_prefix_lists_batches_from_openai - the dedicated prefix relays a batch listing to OpenAI

  • curl http://localhost:4000/openai_passthrough/v1/batches -H "Authorization: Bearer $KEY"
  • Expect 200 with "object": "list". A body that is not an OpenAI list fails validation rather than passing vacuously
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/batches/test_batches_e2e.py::TestOpenAIFiles::test_list_page_cursors_address_only_the_callers_own_files - a list page's cursors address rows in that page, never another tenant's files

  • Generate a key with a fresh user_id so it owns no files
  • curl http://localhost:4000/v1/files -H "Authorization: Bearer $KEY"
  • Expect first_id and last_id to equal the first and last IDs in data, both null when data is empty, and has_more not true
  • To see that the account really is shared, list batches through the passthrough prefix and note the input_file_id values that belong to other callers
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_passthrough_e2e.py::TestOpenAIPassthroughSpend::test_streamed_responses_call_logs_its_cost - a streamed passthrough Responses call is billed, keyed by the provider ID the caller read

  • curl -N -X POST http://localhost:4000/openai_passthrough/v1/responses -H "Authorization: Bearer $KEY" -d '{"model":"gpt-5.5","input":"Say hi in one word.","stream":true}'
  • Take the id and usage off the response.completed SSE frame, not the x-litellm-call-id header: this route never stamps the gateway's call ID onto the response, so polling by it finds nothing
  • curl "http://localhost:4000/spend/logs?request_id=$RESP_ID" -H "Authorization: Bearer sk-1234", retrying for up to two minutes since rows are written on a batch interval
  • Expect a row with spend > 0, call_type pass_through_endpoint, and prompt and completion tokens equal to the frame's input_tokens and output_tokens
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_passthrough_e2e.py::TestOpenAIPassthroughSpend::test_embeddings_call_logs_its_cost - a passthrough embeddings call is billed rather than writing no row at all

  • curl -X POST http://localhost:4000/openai_passthrough/v1/embeddings -H "Authorization: Bearer $KEY" -d '{"model":"text-embedding-3-small","input":"cost this sentence"}'
  • Poll curl "http://localhost:4000/spend/logs?request_id=$CALL_ID" -H "Authorization: Bearer sk-1234" with the x-litellm-call-id header value. Here the call ID does work, because an embedding response carries no ID of its own
  • Expect a row with spend > 0 and prompt_tokens > 0. Asserting only that a row exists is not enough: an unmapped model logs one at 0.0
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Prerequisites for all five: OPENAI_API_KEY in the environment, a Postgres-backed proxy, and a short PROXY_BATCH_WRITE_AT if you do not want to wait the default ten seconds for spend rows

CI note, so the red gate is not mistaken for this diff: all three full-suite runs at this tip came back 3 failed, 643 passed, 58 skipped, and the three failures are test_unflagged_model_converts_system_reminder_and_succeeds on Bedrock Invoke, Vertex, and Azure Foundry, each stopping in the same place with prompt cache never became readable in full within 60.0s. Those cases live in test_messages_mid_conversation_system_e2e.py and test_messages_mid_conversation_system_native_providers_e2e.py, and this diff touches neither file. Both were rewritten on the base earlier today by 155ef8c and e11399f, and build 101 is the first full-suite run whose base carries that rewrite: the last run on a base without it passed the older version of the same three cases. All five tests added here passed in every run, and the staging breakage is tracked as LIT-5921. Separately, buildkite/e2e-tests is not one of litellm_internal_staging's required checks, so the red X does not block merging

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

CHECKED: live-pr-risk at d924226. Two of the seven files are registry YAML and two are new cases appended to existing test files, which leaves three touching harness surface. tests/e2e/lifecycle.py widens ResourceManager.defer from Callable[[], None] to Callable[[], object], which is a contravariant widening, so all 289 pre-existing defer(...) call sites still typecheck unchanged (290 on the head, the one new one being this PR's own), and the sole reader is the teardown loop that calls each entry and discards what it returns either way. tests/e2e/batches/batch_client.py adds first_id, last_id, and has_more to FileList, all optional and defaulting to None; FileList is referenced nowhere outside that file, and models.py's similarly named FileListResponse behind ProxyClient.list_files is a different model this diff never touches. passthrough_client.py is additive only, with no existing method re-signed. The full e2e suite collects clean on the head (700 of 702, two deselected, 0 errors), the coverage collector runs --strict without a failure, and basedpyright over tests/e2e reports 0 errors. The five cases were driven live against a real proxy, database, and OpenAI at 8b7c801; the two commits since are a docstring-only edit and a merge of the base, and the live evidence at this tip is the full suite CI runs below

• d924226 passes flake test (ran 3 times). All five tests added here passed in all three runs, and every run ended 3 failed, 643 passed, 58 skipped with the same three mid-conversation-system cases as the only failures. Each run stamps a different priming marker, so those three reproduce rather than flake. Builds 101, 103, 104

---

Note

Low Risk
Tests-only; no product code. Risk is limited to e2e harness typing (defer return type) and live OpenAI traffic in CI.

Overview
Adds live e2e pins for four already-shipped gateway bugs: /openai_passthrough must actually reach OpenAI (not bind as a provider name), streamed Responses and embeddings passthrough must write a priced spend row, and GET /v1/files cursors must name only the caller's files.

Passthrough prefix (#36086). New cases upload a file and list batches under /openai_passthrough/v1/... and assert OpenAI's own objects come back, so a regression that 500s on 'openai_passthrough' is not a valid LlmProviders fails the suite.

Passthrough spend (#36523, #36646). Streamed Responses spend is polled by the response.completed id and token counts must match that frame. Embeddings must log spend > 0 and prompt tokens, not a missing or $0 row.

File list isolation (#36087). FileList now models first_id / last_id / has_more. A fresh key's list page must have cursors that match its own data (null when empty) and must not advertise another page.

Coverage registry rows are added for each cell. ResourceManager.defer now accepts cleanups that return a value so delete helpers can be registered directly.

^{Reviewed by Cursor Bugbot for commit d924226. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

Adds tests-only end-to-end coverage for previously shipped OpenAI passthrough routing, spend logging, and file-list isolation behavior.

• Exercises native file upload/deletion and batch listing through /openai_passthrough.
• Reconciles streamed Responses and embeddings calls with their persisted spend records.
• Verifies file-list cursors remain consistent with the caller-visible page.
• Extends the E2E client models, lifecycle typing, and coverage registry for these scenarios.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/test_passthrough_e2e.py	Adds live passthrough routing and spend-accounting tests without an eligible follow-up defect.
tests/e2e/llm_translation/passthrough_client.py	Adds typed client helpers and response models for OpenAI passthrough files, batches, Responses streams, and embeddings.
tests/e2e/batches/test_batches_e2e.py	Adds assertions that file-list cursors and pagination metadata agree with the caller-visible page.
tests/e2e/batches/batch_client.py	Extends the file-list response model with optional pagination fields.
tests/e2e/lifecycle.py	Broadens deferred cleanup callback return typing while teardown continues to discard callback results.
tests/e2e/coverage_registry/llm_conversational.yaml	Registers streamed OpenAI Responses passthrough cost coverage.
tests/e2e/coverage_registry/llm_nonconversational.yaml	Registers passthrough embeddings, batches, files, and file-list isolation coverage.

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8b7c801. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d924226. Configure here.}

[tin-berri]

Good coverage — these pin four real shipped fixes (tenant file-list isolation leaking other callers' raw provider file ids per #36087, the /openai_passthrough prefix binding as a provider name per #36086, and streamed-responses/embeddings passthrough calls logging $0 spend rows per #36523/#36646). Test-only, no production code touched.

Holding off approval though: buildkite/e2e-tests failed after a real 28-minute run (other open PRs I checked complete this check in ~30s since they don't touch tests/e2e/, so this is actually exercising the suite). I don't have access to the Buildkite log content to tell whether one of the new assertions is wrong or this caught a real regression in the isolation/cost-logging behavior it's pinning. Given this PR is specifically about tenant-isolation and billing correctness, I'd rather confirm than approve blind — could you check which of the 4 new tests failed and paste the assertion output? Happy to approve once it's clear.

[tin-berri]

Approving per Tin — solid e2e coverage pinning four real shipped fixes (tenant file-list isolation per #36087, passthrough prefix routing per #36086, streamed-responses/embeddings cost logging per #36523/#36646), test-only. Note: buildkite/e2e-tests was still showing failing after a real 28-minute run as of my last check and I wasn't able to see which assertion failed — worth a follow-up look given this PR is specifically about tenant-isolation and billing correctness.