fix(passthrough): record ownership of streamed responses under managed ids by mateo-berri · Pull Request #38320 · BerriAI/litellm

mateo-berri · 2026-08-26T08:09:33Z

TLDR

Problem this solves:

Streamed passthrough responses came back with OpenAI's raw resp_ id
Any key could then read, continue from, or delete that response by id
The first key to read it became its owner and locked the creator out

How it solves it:

Mint the managed id from the first response.created event
Rewrite response.id inside every SSE frame as it is relayed
Record the response under the caller, so later reads are checked by id

User Flow

Before: a developer on key A streams a response through the OpenAI passthrough and gets OpenAI's own resp_... id back, so any other key on the gateway can read and delete it, and the first one to read it becomes its owner

Key A sends POST https://litellm-domain/openai_passthrough/v1/responses with {"model": "gpt-5.1", "input": "Reply with the single word mango.", "stream": true, "store": true}
The SSE events come back with "id": "resp_04bd26a8...", OpenAI's raw id, instead of the long scrambled id the same call returns without stream: true
Key B sends GET https://litellm-domain/openai_passthrough/v1/responses/resp_04bd26a8... and gets 200 with key A's full response text; the body now carries a scrambled id that belongs to key B
Key B sends DELETE https://litellm-domain/openai_passthrough/v1/responses/resp_04bd26a8... and gets 200 "deleted": true
Key A sends DELETE on its own response and gets 404 Managed resource not found.
Any key on the gateway could read, continue from, and delete another key's streamed response, and by reading it first could lock the creator out of it

After: the same streamed response comes back under a scrambled gateway id owned by key A, so other keys are refused

Key A sends POST https://litellm-domain/openai_passthrough/v1/responses with {"model": "gpt-5.1", "input": "Reply with the single word mango.", "stream": true, "store": true}
Every SSE event carries the long scrambled gateway id (bGl0ZWxsbV9wcm94eTpwYXNz...), the same shape as a non-streamed response
Key B sends GET https://litellm-domain/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz... and gets 403 Access denied to managed resource.
Key B sends DELETE https://litellm-domain/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz... and gets 403 Access denied to managed resource.
Key A sends DELETE on its own response and gets 200 "deleted": true
Another key can no longer read, continue from, or delete a streamed response it did not create

Relevant issues

Linear ticket

Resolves LIT-6165

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Each leg ran two separate proxy processes, each with --num_workers 2, on random ports sharing one Postgres. Both virtual keys were minted through instance 1, key A creates through instance 1, and key B's requests plus key A's own-read go through instance 2, so every ownership check crossed processes. Config (passthrough_managed_object_ids on, real OpenAI key); KEY_A has user_id: user-a, KEY_B has user_id: user-b. Long managed ids are shortened to bGl0ZWxsbV9wcm94eTpwYXNz... below; full ids and untruncated bodies are in the QA logs (lit6165_qa_before.out / lit6165_qa_after.out)

model_list:
  - model_name: gpt-5.1
    litellm_params:
      model: openai/gpt-5.1
      api_key: os.environ/OPENAI_API_KEY

files_settings:
  - custom_llm_provider: openai
    api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  passthrough_managed_object_ids: true

python litellm/proxy/proxy_cli.py --config lit6165.yaml --port $P1 --num_workers 2 --use_v2_migration_resolver
python litellm/proxy/proxy_cli.py --config lit6165.yaml --port $P2 --num_workers 2 --use_v2_migration_resolver
curl -s -X POST http://127.0.0.1:$P1/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"user_id": "user-a"}'
curl -s -X POST http://127.0.0.1:$P1/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"user_id": "user-b"}'

Before (`e52f055`, instances on P1=46566 and P2=47062)

Streamed response: key B reads, continues from, and deletes key A's response through the other instance

Key A streams a response through instance 1; every event carries OpenAI's raw id

$ curl -s -N -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "input": "Reply with the single word mango.", "stream": true, "store": true}' http://127.0.0.1:46566/openai_passthrough/v1/responses
event: response.created
data: {"type":"response.created","response":{"id":"resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79","object":"response","created_at":1787732201,"status":"in_progress",...

Key B reads it by that id through instance 2 and gets the full response, now stamped with a managed id that belongs to key B

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" http://127.0.0.1:47062/openai_passthrough/v1/responses/resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response", "created_at": 1787732201, "status": "completed", ...
HTTP 200

Key B continues the conversation from key A's response

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X POST -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "previous_response_id": "resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79", "input": "What word did you just say?"}' http://127.0.0.1:47062/openai_passthrough/v1/responses
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response", "created_at": 1787732202, "status": "completed", ...
HTTP 200

Key B deletes it

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X DELETE http://127.0.0.1:47062/openai_passthrough/v1/responses/resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response.deleted", "deleted": true}
HTTP 200

Key A, the creator, is now locked out of its own response on both instances

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" http://127.0.0.1:47062/openai_passthrough/v1/responses/resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79
{"error":{"message":"Managed resource not found.","type":"None","param":"None","code":"404"}}
HTTP 404
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" -X DELETE http://127.0.0.1:46566/openai_passthrough/v1/responses/resp_0a9c3df86361cf7b006a8ea0e903b887d0afdc5375bce05a79
{"error":{"message":"Managed resource not found.","type":"None","param":"None","code":"404"}}
HTTP 404

Non-streamed response (control, already protected)

Key A creates a response without stream through instance 1; the id is already a managed id

$ curl -s -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "input": "Reply with the single word pineapple.", "store": true}' http://127.0.0.1:46566/openai_passthrough/v1/responses
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response", "created_at": 1787732203, ...

Key B is refused on read and delete through instance 2, and key A deletes its own

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" http://127.0.0.1:47062/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X DELETE http://127.0.0.1:47062/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" -X DELETE http://127.0.0.1:46566/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response.deleted", "deleted": true}
HTTP 200

After (`6a9662a`, instances on P1=38401 and P2=52717)

Streamed response: key B is refused through the other instance

Key A streams a response through instance 1; every event carries the managed id from response.created on (10 data lines, 1 distinct id, 0 lines with a raw resp_ id)

$ curl -s -N -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "input": "Reply with the single word mango.", "stream": true, "store": true}' http://127.0.0.1:38401/openai_passthrough/v1/responses
event: response.created
data: {"type":"response.created","response":{"id":"bGl0ZWxsbV9wcm94eTpwYXNzdGhyb3VnaDtwcm92aWRlcjpvcGVuYWk7dW5pZmllZF9pZCw2YTk2NTczNC02NzU0LTQ3OTYtOGE4ZS01YzZmZGZiZDkzNjE7cmF3X2lkLHJlc3BfMDg3ZWEwYWZkZWEzMDNkYjAwNmE4ZWE1YWRlOGYwODdkMDkyNGY4ODc3ODZhNTE2ZWI","object":"response","created_at":1787733421,"status":"in_progress",...

Key B reads it by that id through instance 2 and is refused

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" http://127.0.0.1:52717/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403

Key B tries to continue from it and is refused

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X POST -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "previous_response_id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "input": "What word did you just say?"}' http://127.0.0.1:52717/openai_passthrough/v1/responses
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403

Key B tries to delete it and is refused

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X DELETE http://127.0.0.1:52717/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403

Key A reads its own streamed response through the OTHER instance, then deletes it

$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" http://127.0.0.1:52717/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response", "created_at": 1787733421, "status": "completed", ...
HTTP 200
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" -X DELETE http://127.0.0.1:38401/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response.deleted", "deleted": true}
HTTP 200

Non-streamed response (control, unchanged)

$ curl -s -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' -d '{"model": "gpt-5.1", "input": "Reply with the single word pineapple.", "store": true}' http://127.0.0.1:38401/openai_passthrough/v1/responses
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response", "created_at": 1787733423, ...
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" http://127.0.0.1:52717/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_B" -X DELETE http://127.0.0.1:52717/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"error":{"message":"Access denied to managed resource.","type":"None","param":"None","code":"403"}}
HTTP 403
$ curl -s -w '\nHTTP %{http_code}' -H "Authorization: Bearer $KEY_A" -X DELETE http://127.0.0.1:38401/openai_passthrough/v1/responses/bGl0ZWxsbV9wcm94eTpwYXNz...
{"id": "bGl0ZWxsbV9wcm94eTpwYXNz...", "object": "response.deleted", "deleted": true}
HTTP 200

Closing observations from the run:

Streamed chat completions relay intact, [DONE] preserved: unaffected
Files and batches enforcement unchanged across instances: left alone
Ownership rows written on one process, enforced on another
Error bodies say "type": "None": pre-existing, left alone
Key B's file list is an empty 200: left alone

Type

🐛 Bug Fix

Caveats (if any)

Medium

Only the /openai_passthrough/* and /azure/openai/* prefixes are covered
- /openai/v1/responses is the native Responses route, and streams there still skip the ownership check (separate ticket)
Objects created before the flag was on stay unowned by design
- Per-team provider credentials are the strict option; docs(passthrough): managed ids cover streamed responses and use the real passthrough prefix litellm-docs#1022 spells both out

Low

Still opt-in behind passthrough_managed_object_ids: true

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
6a9662a passes /live-pr-risk

Note

High Risk
Changes authorization and multi-tenant isolation for streamed Responses passthrough (auth/access control on a security-sensitive path), though behavior is gated behind passthrough_managed_object_ids and limited to Responses SSE routes.

Overview
Fixes a cross-tenant gap for streamed OpenAI Responses passthrough: SSE bodies used upstream raw resp_ ids with no ownership row, so other API keys could read, continue, or delete those responses (and sometimes become the owner on first read).

Streaming path now mirrors non-streamed managed-id behavior. When passthrough_managed_object_ids is on and the route is POST /v1/responses, the proxy reassembles complete SSE frames (shared split_complete_sse_frames, including CR-only delimiters), mints or reuses a managed object id from the first response.created event, persists the caller as owner, and byte-replaces the raw id in every relayed frame. Other routes and disabled flags pass the stream through unchanged; if persistence fails, the stream stays raw.

Passthrough streaming responses wire this via _own_streamed_managed_ids ahead of SSE keepalive wrapping; cost-injection streaming reuses the same frame splitter instead of a local duplicate.

^{Reviewed by Cursor Bugbot for commit 6a9662a. Bugbot is set up for automated code reviews on this repo. Configure here.}

…d ids

codecov · 2026-08-26T08:19:18Z

Codecov Report

❌ Patch coverage is 91.66667% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roxy/pass_through_endpoints/managed_id_rewriter.py	90.00%	7 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-08-26T08:22:15Z

Greptile Summary

This follow-up adds streamed-response ownership hardening and completes the requested CR-only SSE boundary handling and comment cleanup.

Centralizes SSE frame splitting with support for LF, CRLF, and CR-only delimiters.
Rewrites streamed managed response IDs and persists caller ownership before relaying completed frames.
Adds focused tests for delimiter handling, transport chunk boundaries, ownership persistence, and passthrough integration.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/proxy/common_utils/sse_keepalive.py	The shared splitter now recognizes CR-only frame boundaries, resolving the previously reported incomplete SSE handling.
litellm/proxy/pass_through_endpoints/managed_id_rewriter.py	Adds buffered streamed-response ID minting and rewriting; the previously reported decorative heading has been removed.
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py	Integrates managed-ID ownership into eligible passthrough streaming response paths behind the existing feature gate.
litellm/proxy/pass_through_endpoints/streaming_handler.py	Reuses the centralized SSE frame splitter without changing the surrounding logging lifecycle.
tests/test_litellm/proxy/common_utils/test_sse_keepalive.py	Adds coverage for all supported SSE delimiters and unterminated tails.
tests/test_litellm/proxy/pass_through_endpoints/test_managed_id_rewriter.py	Covers streamed ownership, arbitrary chunking, CR-only framing, unaffected routes, and persistence-failure fallback.
tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py	Adds an integration-level mocked passthrough test confirming ownership persistence and consistent streamed ID replacement.

_{Reviews (3): Last reviewed commit: "fix(passthrough): recognize CR-only SSE ..." | Re-trigger Greptile}

mateo-berri · 2026-08-26T08:22:50Z

@greptileai

… streamed managed ids

mateo-berri · 2026-08-26T08:39:48Z

@greptileai

mateo-berri · 2026-08-26T08:44:51Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6a9662a. Configure here.}

codspeed-hq · 2026-08-26T09:18:38Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_passthrough_object_ownership (6a9662a) with litellm_internal_staging (137311f)}

fix(passthrough): record ownership of streamed responses under manage…

e0c101b

…d ids

mateo-berri mentioned this pull request Aug 26, 2026

docs(passthrough): managed ids cover streamed responses and use the real passthrough prefix BerriAI/litellm-docs#1022

Merged

greptile-apps Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread litellm/proxy/common_utils/sse_keepalive.py

Comment thread litellm/proxy/pass_through_endpoints/managed_id_rewriter.py Outdated

fix(passthrough): recognize CR-only SSE frame delimiters when minting…

6a9662a

… streamed managed ids

cursor Bot reviewed Aug 26, 2026

View reviewed changes

mateo-berri enabled auto-merge August 26, 2026 09:11

yucheng-berri approved these changes Aug 26, 2026

View reviewed changes

mateo-berri merged commit 4185c8a into litellm_internal_staging Aug 26, 2026
84 checks passed

mateo-berri deleted the litellm_passthrough_object_ownership branch August 26, 2026 09:15

Navigation Menu

Uh oh!

Uh oh!

fix(passthrough): record ownership of streamed responses under managed ids - #38320

fix(passthrough): record ownership of streamed responses under managed ids#38320
mateo-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_passthrough_object_ownership

mateo-berri commented Aug 26, 2026 •

edited by cursor Bot

Loading

Uh oh!

codecov Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 26, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented Aug 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (e52f055, instances on P1=46566 and P2=47062)

Streamed response: key B reads, continues from, and deletes key A's response through the other instance

Non-streamed response (control, already protected)

After (6a9662a, instances on P1=38401 and P2=52717)

Streamed response: key B is refused through the other instance

Non-streamed response (control, unchanged)

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

codecov Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 26, 2026

Merging this PR will not alter performance

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Aug 26, 2026 •

edited by cursor Bot

Loading

Before (`e52f055`, instances on P1=46566 and P2=47062)

After (`6a9662a`, instances on P1=38401 and P2=52717)

codecov Bot commented Aug 26, 2026 •

edited

Loading

greptile-apps Bot commented Aug 26, 2026 •

edited

Loading