fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering

[yucheng-berri]

TLDR

Problem this solves:

• A post_call guardrail on /bedrock/.../converse-stream held the whole stream until Bedrock finished
• AWS SDK apps hit their read timeout, and a block came back as an empty ClientError
• Noma v2 got one buffered scan per reply, with no stream mark and no finished answer
• A held stream kept reading Bedrock after the client left, and billed the whole answer

How it solves it:

• Converse frames are relayed as they arrive while the guardrail scans them, like /v1/messages
• Same knobs as chat, scan every 5th frame plus the end, or streaming_buffer_until_moderated: true to hold and mask
• A block after the first frame ends the stream with an AWS exception frame boto3 raises with the guardrail message
• Guardrails that need the whole response (own success hook, mask_response_content, a held stream with streaming_buffer_release_on_scan, run_in_parallel, native hooks, policy pipelines) keep the buffered path
• Noma v2 scans carry stream: true, and the last one carries the finished answer with its tool calls, one record per reply like chat
• A held stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat

Intentional product change: in the default sampled mode, frames reach the caller before the guardrail's verdict and masks are not applied, exactly as on /v1/chat/completions, /v1/messages and /v1/responses. Set streaming_buffer_until_moderated: true to keep masking. Noma v2 honors that setting since #46606 merged to main, and custom code still ignores it on every route

User Flow

Before: an app using the AWS SDK ConverseStream through the proxy stalls, then times out, once a post_call guardrail is on

1. The proxy admin attaches a post_call guardrail (Noma, Bedrock Guardrails, custom code) to the app's key
2. The app calls converse_stream with endpoint_url=https://litellm-domain/bedrock and a 5 s read timeout
3. No response headers arrive until Bedrock finishes (about 11 s), so the SDK raises ReadTimeoutError
4. When the guardrail blocks, the SDK raises ClientError: An error occurred (400) ... with an empty message
5. The proxy admin opens https://litellm-domain/ui/?page=logs for the blocked request and finds one Success row billed for the whole generation (36 prompt + 170 completion tokens), no Failure row

After: the same app streams normally and sees blocks as readable SDK errors

1. The proxy admin attaches the same post_call guardrail to the app's key
2. The app calls converse_stream with the same 5 s read timeout
3. Headers and the first text arrive in about a second, then text keeps streaming until Bedrock finishes
4. When the guardrail blocks mid-stream, the SDK raises EventStreamError (validationException) with the guardrail's message, after the text already received
5. The proxy admin opens the same Logs page and finds a session of 2, a Success row billing the prompt (36 tokens) plus the 60 tokens relayed before the block, and a Failure row for the block

Linear ticket

Refs LIT-9515

When this PR opened, a block before the first frame (held mode) returned LiteLLM's JSON error, and Noma v2 ignored the streaming settings. Both are now fixed on main, by #46522 (LIT-9617) and #46606 (LIT-9633), and this branch has both since main was merged in at 8ab2027. invoke-with-response-stream is LIT-9618

#46242 (an earlier attempt at this ticket) was closed by its author before this PR opened

Docs: BerriAI/litellm-docs#2340

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup for the boto3 cases. One model claude-haiku = bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 (real Bedrock, us-east-1), and a real AWS Bedrock guardrail (guardrail: bedrock, word zzzforbidden blocked, EMAIL anonymized) attached as a post_call guardrail to the caller's key, once with streaming_buffer_until_moderated: false (sampled), once with that plus streaming_end_of_stream_only: true, and once with the default (held). The client is boto3 bedrock-runtime converse_stream with endpoint_url=http://<proxy>/bedrock, Postgres-backed proxy, 4 workers

Setup for the Noma cases. Real Noma v2 (guardrail: noma, use_v2: true, api.noma.security) behind a recorder that logs each scan request and forwards it, so the request the proxy sent is visible. The held-stream disconnect case stacks a real Straiker guardrail, which holds the stream by default

Before (464bb5a)

no guardrail, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with a key with no guardrail, asking for a 700-word story, read_timeout=5
2. Response headers after 2.13 s, first text after 2.14 s, done after 12.91 s, 457 events
3. Stream ended with stopReason: end_turn, text: # The Keeper's Vigil Samuel had kept the lighthouse for for ... atient vigil above the rocks and the restless sea.

post_call guardrail, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (sampled), asking for a 700-word story, read_timeout=5
2. No response headers; the call ended after 10.76 s with no events
3. boto3 raised ReadTimeoutError: Read timeout on endpoint URL: "http://127.0.0.1:4971/bedrock/model/claude-haiku/converse-stream"

post_call guardrail scanning at end of stream only, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (streaming_end_of_stream_only: true), asking for a 700-word story, read_timeout=5
2. No response headers; the call ended after 10.86 s with no events
3. boto3 raised ReadTimeoutError: Read timeout on endpoint URL: "http://127.0.0.1:4971/bedrock/model/claude-haiku/converse-stream"

post_call guardrail blocks mid-stream

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (sampled), asking for about 120 words whose third sentence contains zzzforbidden, read_timeout=60
2. No response headers; the call ended after 3.2 s with no events
3. boto3 raised ClientError: An error occurred (400) when calling the ConverseStream operation:

post_call guardrail masks (held stream)

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (held, the default), asking for a note asking to email jane.doe@example.com, read_timeout=60
2. Response headers after 1.22 s, first text after 1.22 s, done after 1.22 s, 16 events
3. Stream ended with stopReason: end_turn, text: # Note Could you please email Jane Doe at {EMAIL} regarding the meeting? Thank you for following up on this.

Client disconnect on a 3.5k-token prompt

1. curl -sN http://<proxy>/bedrock/model/claude-haiku/converse-stream -H "x-litellm-api-key: $KEY" -d @big.json | head -c 400, where big.json is a 3.5k-token prompt asking for a 250-word story and $KEY has the sampled guardrail, so the client hangs up after the first frames
2. https://litellm-domain/ui/?page=logs searched by its request id 9227e0ce-4125-46be-a861-bff321aa9f5c
3. One Success row billed for the whole generation the proxy kept reading after the client left, 3,477 prompt + 337 completion tokens, $0.005678, 4.88 s

Client leaves a held stream before the first frame

1. A raw POST /bedrock/model/claude-haiku/converse-stream asking for a 300-word answer, with a key whose guardrail holds the stream (Straiker), and the client gives up after 1.5 s, before any frame
2. The proxy kept reading Bedrock for 5.42 s after the client left and sent Straiker one more scan
3. https://litellm-domain/ui/?page=logs searched by its request id 68456d48-ab27-4b76-984e-9e6ce558bac8 shows one Success row for the whole answer, 25 prompt + 495 completion tokens, $0.002750, 6.71 s

Admin UI Logs for a blocked stream (custom_code guardrail)

1. A raw POST /bedrock/model/claude-haiku/converse-stream stream is blocked by a custom_code post_call guardrail, HTTP 400 JSON code says no
2. https://litellm-domain/ui/?page=logs searched by its request id a9c470fb-b8f6-4942-84e1-1e67a3d2a0e8 shows the Before screenshots under User Flow (one Success row, 36 + 170 tokens, $0.000975, no Failure row)

Noma v2 scans of a streamed answer

1. boto3 converse_stream with a Noma v2 key, five prompts (card number, email, prompt injection, PII, a long answer), three runs each
2. 15 of 15 replies got one Noma scan after Bedrock finished, with no stream mark and no finished response in request_data

Noma v2 scan of a reply that only calls a tool

1. boto3 converse_stream with toolChoice forcing get_time, a tool with no parameters, on a Noma v2 key, sampled and held, two runs each
2. 4 of 4 replies got no Noma scan, since the reply has no text

After (11bf918)

no guardrail, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with a key with no guardrail, asking for a 700-word story, read_timeout=5
2. Response headers after 1.29 s, first text after 1.29 s, done after 12.1 s, 409 events
3. Stream ended with stopReason: end_turn, text: # The Keeper's Watch Marcus had tended the lighthouse for t ... s his light. In the morning, he would climb again.

post_call guardrail, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (sampled), asking for a 700-word story, read_timeout=5
2. Response headers after 1.07 s, first text after 1.07 s, done after 32.61 s, 415 events
3. Stream ended with stopReason: end_turn, text: # The Keeper's Watch Thomas had been keeping the lighthouse ... one. That was enough. That had always been enough.

post_call guardrail scanning at end of stream only, read_timeout=5

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (streaming_end_of_stream_only: true), asking for a 700-word story, read_timeout=5
2. Response headers after 0.84 s, first text after 0.84 s, done after 11.45 s, 422 events
3. Stream ended with stopReason: end_turn, text: # The Lighthouse Keeper Thomas had been keeping the lightho ... his responsibility, his conversation with the sea.

post_call guardrail blocks mid-stream

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (sampled), asking for about 120 words whose third sentence contains zzzforbidden, read_timeout=60
2. Response headers after 0.58 s, first text after 0.58 s, done after 1.88 s, 19 events
3. boto3 raised EventStreamError validationException: Violated guardrail policy
4. Text received before the error: # The Joy of Gardening Gardening is a rewarding hobby that ... ony, there's always space to grow something. The z

post_call guardrail masks (held stream)

1. client.converse_stream(modelId="claude-haiku", ...) with the key with the Bedrock guardrail (held, the default), asking for a note asking to email jane.doe@example.com, read_timeout=60
2. Response headers after 1.36 s, first text after 1.36 s, done after 1.36 s, 19 events
3. Stream ended with stopReason: end_turn, text: # Note Please email {EMAIL} to confirm your attendance at t ... if you have any questions or scheduling conflicts.

Client disconnect on a 3.5k-token prompt

1. The same curl with the same prompt and key
2. https://litellm-domain/ui/?page=logs searched by its request id e9dc30df-4e10-4293-a754-e4ffd872c65d
3. One Success row for what was relayed, 3,193 prompt tokens (estimated from the request, the same estimate a cut /v1/chat/completions stream gets) + 12 completion tokens, $0.003578, 1.14 s

Client leaves a held stream before the first frame

1. The same request with the same key, and the client gives up after 1.5 s
2. The proxy stopped reading Bedrock 0.03 s after the client left and sent Straiker no further scan
3. https://litellm-domain/ui/?page=logs searched by its request id 62b20d65-1d39-4efe-9de7-e4a9d4d48dae shows one Success row for what Bedrock sent before the stop, 25 prompt + 27 completion tokens, $0.000176, 1.47 s. A chat completions stream on the same key gave 25 + 40 tokens and stopped 0.01 s after the client left

Admin UI Logs for a blocked stream (custom_code guardrail)

1. The same stream returns 200, the text frames, then a validationException frame with code says no
2. https://litellm-domain/ui/?page=logs searched by its request id bf848796-84ea-46e3-aee5-8cc02cd29666 shows the After screenshots under User Flow (Success row for 36 prompt + 60 relayed tokens, $0.000370, plus a Failure row)

Noma v2 scans of a streamed answer

1. The same five prompts plus three more (an email in the middle of a long answer, a record, a secret), one run each
2. 8 of 8 replies had stream: true on every Noma scan and the finished response on the last scan only, as chat sends it. A card-number reply got 2 scans, a long answer 34
3. A reply with text and a get_weather tool call ended with a Noma scan whose response holds {"text": "It's a beautiful day to check the weather!"} and {"toolUse": {"toolUseId": "tooluse_JQN1...", "name": "get_weather", "type": "tool_use", "input": {"city": "Paris"}}}, stopReason: tool_use

Noma v2 scan of a reply that only calls a tool

1. The same forced get_time call, sampled and held, two runs each
2. 4 of 4 replies got one Noma scan after the stream ended, texts: [], stream: true and response content [{"toolUse": {"toolUseId": "tooluse_mqty...", "name": "get_time", "type": "tool_use", "input": {}}}]. Chat completions on the same key also gives one scan, with the tool call in tool_calls
3. An Akto guardrail on the same reply got no scan on base or head

With main's #46606 (11bf918 merged with main ecbea93, the same code as head 8ab2027)

Noma v2 masks in held mode

1. boto3 converse_stream on real Bedrock Haiku with a real Noma v2 guardrail (use_v2: true), asking for a line with a test card number, alone and at the end of a 150-word note, two runs each, on main and on this branch merged with main
2. streaming_buffer_until_moderated: true: 4 of 4 replies masked on both (Card on file: *******, exp 12/29.), with one Noma scan per reply. On the merge that scan carries stream: true and the finished response
3. Default sampled and streaming_end_of_stream_only: true: main masked 8 of 8 because it buffers the whole reply. The merge sent the first frame at 0.7-0.8 s and delivered the card unmasked 8 of 8, as chat does (the breaking change below)
4. Chat completions on the same keys delivered the card unmasked in every mode, held included, on main and on the merge (LIT-9559)

Integration tests and audit

tests/integration/observability/test_bedrock_converse_stream_post_call_guardrail.py drives a real proxy with Postgres, Redis, a scripted Bedrock upstream and a scripted guardrail vendor through boto3 and raw HTTP. Its 28 cells cover streaming before the upstream finishes, mid-stream blocks, held-mode masks (including a held Bedrock guardrail with streaming_buffer_release_on_scan: true), upstream and vendor failures, end-of-stream-only mode, stacked guardrails (including a run_in_parallel guardrail listed before a masking one), client disconnects in sampled and held mode, a parallel request slot freed by a held disconnect, spend and failure rows, Noma v2 stream marks (including a reply that only calls a tool), and bursts. Base 464bb5a fails 16 of them, and head 11bf918 passes all 28 on two runs. The tool-only Noma cell was red on this PR's earlier head fde82b3, and the parallel-slot cell was red on e82e15f (the follow-up request got 429)

Main was merged in at 8ab2027. Main's #46522 now answers a held block before the first frame as ValidationException with the guardrail message instead of a bare 400, so that cell checks for it. At 8ab2027 this file plus #46606's tests/integration/streaming/test_noma_v2_streaming_settings.py pass on two runs (277 passed, identical selections), and the new merge base ecbea93 fails the same 16 relay cells

Two earlier regression sweeps are folded in. A held guardrail with streaming_buffer_release_on_scan: true and a run_in_parallel guardrail listed before a masking one both keep the buffered path, so they mask and scan the masked text as main does (real Bedrock A/B, base and head masked on every run)

After merging main, 8ab2027 re-ran the integration files, the live A/B batch against the new merge base ecbea93, worker-kill chaos and billing, with the same results as below, and the final audit at 8ab2027 is PASS. The audit at 11bf918 re-ran the integration file (28 of 28 twice on head, 16 failing on base), the live A/B batch, worker-kill chaos and billing on both legs, and the Noma v2 and held-disconnect cells on real Bedrock and real Noma. Every behavior change below was signed off by the ticket owner, including the 5xx exception frame and the metrics and hooks the relay skips

Type

🐛 Bug Fix

Behavior changes

Removed / renamed

• None

Changed on /bedrock/.../converse-stream with a post_call guardrail that scans through apply_guardrail

• Breaking: default (sampled) mode streams frames before the verdict and does not apply masks (chat parity). Guardrails that rewrite text without their own streaming hook (Noma v2, Generic Guardrail API, custom code, secret detection) masked this route on base and stop masking after the upgrade, with no warning. streaming_buffer_until_moderated: true keeps per-content-block masking for guardrails that read it, such as the Generic Guardrail API and, since fix(guardrails): honor streaming settings on the Noma v2 guardrail #46606 merged to main, Noma v2 (live, see Proof). Custom code still ignores that setting on every route, so it can't keep masking here. Signed off by the ticket owner, @yucheng-berri, in the LIT-9515 design review on 2026-10-13, and accepted again on 2026-10-14 as a known risk after LiteReview flagged it, to match chat
• Mid-stream block becomes an AWS exception frame (validationException for 4xx, internalServerException for 5xx) instead of HTTP 400/500 JSON, and an end-of-stream block arrives after the text
• Mid-stream upstream or guardrail-vendor failure in sampled mode sends the frames then internalServerException instead of HTTP 500 JSON
• Bedrock guardrail with disable_exception_on_block on a held stream returns 200 with the guardrail message and stopReason: guardrail_intervened instead of HTTP 500
• Guardrail vendor calls happen once per scan round (every 5th frame plus the end) instead of once per stream, with the text so far, as on chat. A Bedrock guardrail bills each call, and the spend rows carry it
• Each sampled scan pauses the stream until the guardrail answers, as on chat. With the Bedrock guardrail a 700-word answer took about 33-39 s instead of about 12 s (no read timeout fires), and streaming_end_of_stream_only: true keeps it at about 12 s
• Spend row guardrail_information holds one entry per scan round, as on chat
• A mid-stream block writes a success row for what was relayed plus a failure row, for every guardrail (base wrote one success row for the whole generation for custom_code)
• Response headers are a chunked event stream with the upstream x-amzn-requestid, instead of a body with content-length
• A blocked or client-disconnected stream bills the prompt plus the relayed part (the proxy stops reading Bedrock, and Bedrock stops generating), not the whole generation
• Noma v2 (use_v2: true) gets stream: true on every scan and the finished answer as a Converse response (text, reasoning and tool-call blocks, stopReason) on the scan after the stream ends, as chat sends it. When the last sampled scan already covered every text, Noma gets one more scan at the end. A reply that only calls a tool gets one scan at the end (base sent none), so Noma can now block it, or fail it closed during a Noma outage, as on chat. Other guardrails get the request as before

Changed on /bedrock/.../converse-stream with a held post_call guardrail

• A client that leaves before the first frame makes the proxy stop reading Bedrock right away, as chat does. Base kept reading to the end, scanned the rest and billed all of it. The proxy logs a 499, writes a spend row for the prompt plus what Bedrock sent before the stop, and when nothing arrived it releases the key's budget reservation and parallel request slot

Changed on every streaming route

• A request cancelled while the proxy waits for the first chunk now cancels the upstream read too (chat, messages, responses and passthrough share that wait). Before, the upstream read kept running
• A Noma v2 scan cancelled by a client disconnect records not_run in the spend row instead of a verdict, on chat too

Changed on every Bedrock streaming passthrough

• A stream that ends before Bedrock's final usage frame (client disconnect, guardrail block, upstream cut) logs prompt tokens estimated from the request's system prompt and messages instead of 0, the same estimate /v1/chat/completions streams use. Unguarded converse-stream disconnects get it too

Unchanged

• Guardrails that scan responses with their own hooks instead of apply_guardrail (for example Noma without use_v2: true), mask_response_content, a held stream with streaming_buffer_release_on_scan: true, post_call guardrails with run_in_parallel: true, native-lifecycle guardrails, and post_call policy pipelines keep the whole stream buffered, as before
• What other guardrails receive. Akto, Straiker and Alice got the same request keys, and no stream or response mark, from this PR before and after the Noma change (e82e15f vs c694684, 15 live cells)
• invoke-with-response-stream, non-streaming /converse, chat/messages/responses streaming apart from the two items above

Caveats (if any)

Medium

• Sampled scans stretch long answers by one guardrail round trip per 5 frames (chat does the same), and streaming_end_of_stream_only: true avoids it
• Vendor volume grows about 30x on long streams in sampled mode (chat already does this), and Akto ingests every round
• The Prometheus guardrail latency metric and non-guardrail success hooks don't run on the relay (same as /v1/messages). OTEL does get a guardrail span per scan round on the relay, as on /v1/messages, where the buffered path emitted none

Low

• On a blocked or disconnected stream the prompt tokens are a tokenizer estimate, like chat (3,193 estimated vs 3,477 reported by Bedrock for the same 3.5k-token prompt). Image, document and guardContent blocks and tool definitions are not counted (chat's estimate leaves out tools too), so document-heavy or tool-heavy requests log a low estimate, still above base's 0
• A block decided by the end-of-stream scan arrives after the upstream's messageStop (same order as /v1/messages), so a client that stops reading at the first messageStop misses it
• Chat on main still skips Noma's last scan when the last sampled scan already covered the text (1 of 15 live chat replies), so Converse now sends Noma one more scan than chat in that case
• In sampled mode a Noma scan already running when the client leaves still finishes, because the proxy notices the disconnect on its next write
• A Noma GUARDRAIL_INTERVENED verdict on a reply that only calls a tool is logged but changes nothing, because tool calls are not sent as texts (text plus tool replies behave the same)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Ran /live-pr-risk at 8ab2027 (real Bedrock, 4 workers, base ecbea93 vs head, plus real Noma v2 cells): no regressions. The backward-incompatible changes it found are exactly the ones listed in Behavior changes, and the base vs head differences are the same 30 cells as at 11bf918

REVIEWER MUST KNOW BEFORE APPROVING

• Breaking: sampled mode (the default) now shows the caller text before the guardrail has judged it, and drops masks. Existing configs that mask this route with Noma v2, the Generic Guardrail API, custom code or secret detection stop masking after the upgrade, with no warning. This is the chat/messages/responses contract, signed off by the ticket owner, @yucheng-berri, in the LIT-9515 design review on 2026-10-13, and accepted again on 2026-10-14 as a known risk after LiteReview flagged it. Operators who mask must set streaming_buffer_until_moderated: true. Noma v2 honors it since fix(guardrails): honor streaming settings on the Noma v2 guardrail #46606 merged to main (a held Noma v2 guardrail masked a card in 4 of 4 live replies on this branch merged with main). Custom code still ignores it on every route, so custom code can't keep masking here
• Latency: sampled mode scans inline every 5th frame, so a slow guardrail stretches the whole answer (Bedrock guardrail about 33-39 s vs about 12 s for 700 words), exactly as on /v1/chat/completions. Recommend streaming_end_of_stream_only: true to customers who only need a final verdict. Noma v2 honors it since fix(guardrails): honor streaming settings on the Noma v2 guardrail #46606 merged to main (one Noma scan per reply, live)
• Billing: a guardrail-blocked or client-disconnected stream used to be billed for the full generation (the proxy read all of it). It is now billed for the prompt plus what was relayed, plus a failure row for a block, and a held stream left before its first frame no longer runs to the end. Bedrock reports usage only in its last frame, so on a cut stream the prompt tokens are estimated from the request the way chat does it. Measured on Bedrock CloudWatch, closing the stream stops AWS output billing (a stream closed after 3 events billed 1 output token, the full read billed 531)
• Shared code: the first-chunk wait used by every streaming route now cancels the upstream read when the request itself is cancelled, and Noma v2 records not_run for a scan cut by a disconnect on every route. The unified guardrail gained needs_ended_stream_scan (default False), and only the Bedrock Converse translation and Noma v2 opt in, so no other route or guardrail gets the extra end scan
• Noma: only use_v2: true (NomaV2, which scans through apply_guardrail) streams. The v1 Noma guardrail runs its own hooks and still holds the whole stream, so the customer's POC needs use_v2: true

---

---

Note

High Risk
Changes guardrail streaming semantics on a security-sensitive path with intentional breaking behavior (unmasked sampled relay), new billing/disconnect handling, and complex frame rewrite logic.

Overview
Bedrock converse-stream passthrough no longer buffers the full AWS event stream for apply_guardrail post_call guardrails. Eligible guardrails scan frame-by-frame (same sampling/hold semantics as chat), with guardrails that need the whole body still on the buffered path.

The Bedrock translation decodes/re-encodes Converse event-stream frames, groups text per content block, applies rewrites when the stream ends (held mode), and emits in-stream validationException / internalServerException frames for mid-stream blocks/failures. The passthrough route dispatches via LlmPassthroughRouteHandler; the proxy adds _relay_guarded_passthrough_event_stream and async_post_call_event_stream_iterator_hook.

Noma v2 sets reads_stream_marks so scans get stream: true and a finished Converse response on the last round; unified guardrail skips redundant end scans unless needs_ended_stream_scan applies. Disconnect/cleanup: first-chunk buffering cancels upstream on caller cancel; held-stream early disconnect returns 499 and stops upstream; passthrough spend flush is awaitable. Logging: cut Converse streams estimate prompt tokens from the request body.

Breaking (default sampled mode): frames reach clients before verdict and masks are not applied unless streaming_buffer_until_moderated: true (chat parity).

^{Reviewed by Cursor Bugbot for commit 8ab2027. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yucheng-berri]

@litereview code-review

[litereview]

✅ code-review

5/5 · Ready for human review

Scope: full · Full PR review.

Completed · Some checks not run.

No correctness findings.

The latest commit, 9088833b35, keeps a converse-stream on the buffered path when a guardrail resolves streaming_buffer_release_on_scan (unless streaming_end_of_stream_only is set). It reads the flag through unified_guardrail.resolve_streaming_flag, the same resolver the stream hook uses. The commit's new integration case covers this path.

The finding about the recursive _block_texts stays fixed at this head.

Checks and execution details

• Passed · Real testing evidence (required): Scope: this rule applies. The PR changes LLM translation code under litellm/llms/bedrock/passthrough/**:

• Converse frame decoding and encoding

• AWS exception and block frames

• guardrail text write-back

• prompt-token estimation for cut streams

It also changes litellm/llms/pass_through/guardrail_translation/handler.py.

Evidence:

• Manual E2E: the PR description records manual E2E runs against implementation 2b4194eaf0. The setup was real Bedrock (us.anthropic.claude-haiku-4-5, us-east-1) with a real AWS Bedrock guardrail. The client was boto3 converse_stream through /bedrock on a Postgres-backed 4-worker proxy. The runs covered sampled, end-only and held modes, a mid-stream block (EventStreamError validationException), masking, and a client disconnect, with timings, event counts and Logs UI screenshots for each result.
• No translation changes since then: the only commit after 2b4194eaf0 (9088833b35, per the GitHub compare API) touches litellm/proxy/common_request_processing.py and tests. It does not touch the scoped translation files, so the manual evidence still describes the reviewed translation code.
• Integration test: tests/integration/observability/test_bedrock_converse_stream_post_call_guardrail.py drives boto3 against a real proxy with scripted upstreams. It covers success and failure paths and adds a release-on-scan case.
• CI: buildkite/e2e-tests reports success on head 9088833.
• Passed · Targeted unit tests (optional): Ran uv run --no-sync pytest -n 8 tests/unit/llms/bedrock/passthrough/ tests/unit/llms/pass_through/guardrail_translation/test_handler.py tests/unit/passthrough/test_streaming_interrupt_spend_tracking.py tests/unit/proxy/test_common_request_processing.py. Result: 578 passed.
• Passed · recursive_detector and ruff on changed modules (optional): recursive_detector.py reported no unignored recursive functions. ruff check on the changed litellm modules reported All checks passed!
• Not run · Integration suite run locally (optional): Not run here. The suite needs the CircleCI-owned proxy, Postgres and Redis harness (.circleci/scripts/run_integration.sh). The integration file and the Buildkite e2e status were inspected instead.

---

View review · Reviewed 9088833b3500

[devin-ai-integration]

Devin Review found 1 potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[codspeed]

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_completion_streaming

✅ 32 untouched benchmarks

---

_{Comparing litellm_converse_stream_guardrail_relay (8ab2027) with main (1b1ddef)¹}

Footnotes

1. No successful run was found on main (ecbea93) during the generation of this report, so 1b1ddef was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[codecov]

Codecov Report

❌ Patch coverage is 89.08382% with 56 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	81.30%	23 Missing ⚠️
...drail_hooks/unified_guardrail/unified_guardrail.py	38.09%	13 Missing ⚠️
...llms/pass_through/guardrail_translation/handler.py	81.48%	10 Missing ⚠️
...drock/passthrough/guardrail_translation/handler.py	97.51%	6 Missing ⚠️
litellm/proxy/utils.py	91.30%	2 Missing ⚠️
litellm/llms/bedrock/passthrough/transformation.py	97.22%	1 Missing ⚠️
litellm/passthrough/main.py	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview full

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litereview]

Medium · Default sampled relay drops guardrail masks on Bedrock converse-stream that the buffered path used to apply (PII leak)

Still open, and the code is unchanged since the last decision.

_passthrough_stream_guardrails sends every guardrail that defines apply_guardrail to the new relay path. A guardrail without a leaf-class async_post_call_streaming_iterator_hook gets holds_until_moderated = False, so it runs in sampled mode. In that mode, BedrockPassthroughGuardrailHandler.process_output_streaming_response writes the guardrail's rewritten texts back only once the stream has ended (handler.py:745-746). By then every frame has already been relayed. Before this PR, the same request was buffered through de_anonymize_event_stream → post_call_success_hook, and the rewritten text was re-encoded into the frames.

Concrete failure: attach a text-rewriting post_call guardrail to a key that calls /bedrock/model/<id>/converse-stream. Examples:

• enterprise secret detection (apply_guardrail returns redact_text(text))
• Noma v2 anonymization
• Generic Guardrail API
• custom code

After the upgrade, secrets and PII reach the client unredacted, with no config change and nothing logged.

The PR now carries fix(bedrock)!: and a named approver. That records acceptance of the risk but doesn't fix it. Operators have to opt in to streaming_buffer_until_moderated: true to get masking back, and per the PR text that doesn't work for Noma v2 or custom code.

Possible fixes: keep these guardrails on the buffered path, or default them to held mode, when their apply_guardrail can rewrite output.

Evidence

common_request_processing.py:3466: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

common_request_processing.py:3467-3473: the fallback to the buffered path only triggers on a missing apply_guardrail, use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan.

bedrock/passthrough/guardrail_translation/handler.py:745-746: if self._check_streaming_has_ended(frames): _write_back_stream_texts(...).

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py:~606: return {**inputs, "texts": [self.redact_text(text) for text in texts]}, with no streaming iterator override, so it runs in sampled relay.

Leaf-class sweep: about 30 guardrails (Noma v2, generic_guardrail_api, custom_code, secret_detection, ...) define apply_guardrail without async_post_call_streaming_iterator_hook.

Latest summary by full · View this review · Reviewed 8f532fd9a93f

[yucheng-berri]

Same accepted risk the owner signed off on 2026-10-14, unchanged in this commit, and sampled chat, messages and responses streams drop these masks too

[litereview]

Finding still applies

The reply cites the owner's sign-off and consistency with other streaming routes. Neither is technical evidence that the defect is wrong or fixed. At head, the routing logic and the write-back-only-at-stream-end behaviour are unchanged. A guardrail that defines apply_guardrail but no leaf-class async_post_call_streaming_iterator_hook (for example enterprise secret detection) is still relayed in sampled mode on Bedrock converse-stream. Its rewrites never reach the client, whereas the base buffered path applied them. This remains an accepted but real backwards-incompatible regression.

Evidence

litellm/proxy/common_request_processing.py:3466: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs is unchanged.

litellm/proxy/common_request_processing.py:3467-3474: the buffered fallback triggers only on a missing apply_guardrail, use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan. Nothing routes a text-rewriting guardrail to buffered or held mode.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py:745-746: if self._check_streaming_has_ended(frames): _write_back_stream_texts(...). Rewrites apply only after every frame has been relayed in sampled mode.

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py:592-606: apply_guardrail returns redacted texts, and the file defines no async_post_call_streaming_iterator_hook.

The reply (comment 4286828014) states the risk is accepted and the code is unchanged in this commit. That confirms the defect remains.

Reviewed 8f532fd9a93f · View follow-up

[yucheng-berri]

@litereview full

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litereview]

Medium · Default sampled relay drops guardrail masks on Bedrock converse-stream that the buffered path used to apply (PII leak)

Still open. The code is unchanged since the last two decisions upheld this finding.

How it happens

_passthrough_stream_guardrails sends every guardrail with a leaf-class apply_guardrail down the new frame relay. If the class has no leaf async_post_call_streaming_iterator_hook, it gets holds_until_moderated = False, so the unified guardrail runs it in sampled mode and forwards each frame as it arrives.

BedrockPassthroughGuardrailHandler.process_output_streaming_response writes rewritten texts back into the frames only once _check_streaming_has_ended(frames) is true. By then every content frame has already reached the client.

On base, the same request was buffered. It went through de_anonymize_event_stream and then post_call_success_hook, and the rewritten text was re-encoded into the frames before anything was sent.

Failure scenario

1. Attach enterprise secret detection to a key as a post_call guardrail. Its apply_guardrail returns redact_text(text), and the class has no streaming iterator override.
2. Call /bedrock/model/<id>/converse-stream and get the model to echo an API key.
3. Base: the client receives [REDACTED] inside the contentBlockDelta frames.
4. Head: the client receives the raw secret. Nothing is logged and no config changed.

Noma v2 anonymization, the Generic Guardrail API and custom-code guardrails fail the same way. The PR text says Noma v2 and custom code ignore streaming_buffer_until_moderated, so the documented opt-out doesn't restore masking for them.

Status

The PR title has ! and the body names an approver, so shipping this is a recorded decision. It is still a client-observable data-exposure regression, not a fix.

Possible fix

Keep guardrails whose apply_guardrail can rewrite output on the buffered path, or default them to held mode (holds_until_moderated=True) on this route.

Evidence

litellm/proxy/common_request_processing.py:3490: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

litellm/proxy/common_request_processing.py:3491-3498: the buffered fallback (return None) triggers only on a missing leaf apply_guardrail, use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py:775-776: if ended: _write_back_stream_texts(frames, groups, ...). Rewrites land only after the stream has ended.

unified_guardrail.py sampled branch: when not buffer_until_moderated, original_items = (copy.deepcopy(item),) are yielded right after each scan, so the original text is sent.

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py:592: defines apply_guardrail, and a grep finds no async_post_call_streaming_iterator_hook in the file.

Latest summary by full · View this review · Reviewed c694684a0ae4

[yucheng-berri]

Matches chat, messages and responses streaming, accepted by the ticket owner. Held mode still masks. I added secret detection to the PR body's list.

[litereview]

Finding still applies

The reply accepts the behaviour; it doesn't change it. At head, the guardrail routing and the stream-end-only write-back are unchanged.

Guardrails that rewrite text in apply_guardrail but have no streaming iterator override run in sampled mode by default on /bedrock/.../converse-stream. In that mode the original frames reach the client before any rewrite is applied. Before this PR, the same request was buffered and masked.

Held mode does mask, but it is opt-in. The PR says Noma v2 and custom code ignore that setting, so for them the opt-out doesn't help. Matching other routes and getting product sign-off is a recorded decision to ship the regression, not technical evidence that the regression is gone.

Evidence

litellm/proxy/common_request_processing.py:3489-3490: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs, so a guardrail that defines only apply_guardrail defaults to sampled relay.

litellm/proxy/common_request_processing.py:3491-3498: the buffered fallback still triggers only on a missing leaf apply_guardrail, use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan. Nothing keeps a rewriting guardrail on the buffered or held path.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py:816 and :832-833: ended = self._check_streaming_has_ended(frames) and then if ended: _write_back_stream_texts(...). The docstring at :806-808 says rewrites reach the caller only when the stream is held until moderated.

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py:592 defines apply_guardrail, and grep finds no async_post_call_streaming_iterator_hook in that file. Secret detection therefore takes the default sampled relay and its redactions don't reach the wire.

Reviewed fde82b354b3b · View follow-up

[yucheng-berri]

@litereview full

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litereview]

Medium · Default sampled relay on Bedrock converse-stream drops masks from rewriting guardrails that the buffered path used to apply

This finding is unchanged from earlier rounds. I re-checked it at head fde82b35, and none of the code involved has changed.

What happens

_passthrough_stream_guardrails sets holds_until_moderated only when the leaf class defines async_post_call_streaming_iterator_hook. It falls back to the buffered path only for these cases:

• the leaf class has no apply_guardrail
• use_native_lifecycle_hooks
• mask_response_content
• run_in_parallel
• release-on-scan
• policy pipelines

Any other guardrail that defines apply_guardrail is relayed in sampled mode. In that mode, BedrockPassthroughGuardrailHandler.process_output_streaming_response calls _write_back_stream_texts only if ended. By then the unified guardrail has already sent the original frames to the client.

Failure scenario

1. Attach a post_call guardrail that rewrites text (secret detection, Noma v2 anonymization, the Generic Guardrail API, or custom code) to a key.
2. Call /bedrock/model/<id>/converse-stream with a prompt that makes the model echo a secret.
3. Base: the stream is buffered, and de_anonymize_event_stream re-encodes the masked text into the contentBlockDelta frames.
4. Head: the frames reach the client with the raw secret. Nothing is logged and no config changed.

The documented opt-out is streaming_buffer_until_moderated: true. The PR says Noma v2 and custom code ignore it, so those users have no way to get masking back.

Status

The author replied with sign-off and the argument that other routes behave the same way. Neither changes the code, so this is still a client-visible data-exposure regression.

Possible fix

Keep guardrails that can rewrite output on the buffered or held path for this route, or default them to holds_until_moderated=True.

Evidence

litellm/proxy/common_request_processing.py:3489-3490: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs

litellm/proxy/common_request_processing.py:3491-3498: the only conditions that return None (buffered path) are a missing apply_guardrail, native hooks, mask_response_content, run_in_parallel and release-on-scan

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py process_output_streaming_response: if ended: _write_back_stream_texts(...), and its docstring says rewrites reach the caller only when the stream is held until moderated

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py defines apply_guardrail and no async_post_call_streaming_iterator_hook, so it takes the sampled relay by default

Latest summary by full · View this review · Reviewed fde82b354b3b

[yucheng-berri]

@litereview full

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[litereview]

Medium · Default sampled relay on Bedrock converse-stream sends rewriting guardrails' unmasked text to the client (redaction regression)

Still open. The code is unchanged since the earlier decisions, and I reran the reproducer at head 11bf9185.

How it happens

1. _passthrough_stream_guardrails sends any guardrail with a leaf-class apply_guardrail down the new frame relay.
2. If the class has no leaf async_post_call_streaming_iterator_hook, it gets holds_until_moderated=False.
3. unified_guardrail therefore runs it in sampled mode and forwards the original frames right after each scan.
4. BedrockPassthroughGuardrailHandler.process_output_streaming_response writes the guardrail's rewritten texts back into the frames only when ended is true. By then every contentBlockDelta has already gone to the client.

On base, the same request was buffered whole and went through _handle_event_stream_allm_passthrough_route, which re-encoded the rewritten text into the frames.

Failure scenario

1. Attach a post_call guardrail whose apply_guardrail returns rewritten text. Examples: enterprise secret detection (redact_text), Noma v2 anonymization, the Generic Guardrail API, custom code.
2. Call /bedrock/model/<id>/converse-stream and have the model echo a secret.
3. Base: the client gets [REDACTED].
4. Head: the client gets the raw secret. No config changed and nothing is logged.

The documented opt-out (streaming_buffer_until_moderated: true) has to be turned on by each operator. Per the PR text, Noma v2 and custom code ignore it.

Status

The ! marker and the named approver record that the risk was accepted. They are not a fix.

Possible fix

On this route, keep guardrails whose apply_guardrail can rewrite output on the buffered path, or make held mode their default.

Evidence

litellm/proxy/common_request_processing.py:3489-3490: leaf_attrs = type(cb).__dict__; holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

litellm/proxy/common_request_processing.py:3491-3498: the buffered fallback runs only when the leaf has no apply_guardrail, or on use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan. Nothing covers guardrails that rewrite text.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py:852-853: if ended: _write_back_stream_texts(...). The docstring at :819-820 says the rewrites reach the caller only when the stream is held until moderated.

Local reproducer /tmp/repro.py (real ProxyLogging and the real Bedrock framer/encoder, a guardrail that replaces sk-SECRET in apply_guardrail, the secret split across three deltas). Output at head: selected [('Redact', False)] then raw secret on wire: True redacted: False.

Latest summary by full · View this review · Reviewed 11bf9185a4a1

[yucheng-berri]

Kept on purpose to match chat, messages and responses, per the ticket owner. Held mode still masks, and the PR body marks it breaking.

[litereview]

Finding still applies

The reply says the behaviour was kept on purpose, that held mode still masks, and that the PR is marked breaking. All three are true, but none shows the defect is wrong or fixed.

At head, the code still works like this:

• _passthrough_stream_guardrails sends a guardrail down the frame relay when its class defines apply_guardrail. If the class doesn't also define async_post_call_streaming_iterator_hook, the guardrail gets holds_until_moderated=False.
• None of the buffered-path exclusions covers guardrails that rewrite text.
• In sampled mode, process_output_streaming_response writes rewrites back only once ended is true. By then the original contentBlockDelta frames have already reached the client.

On base, the same request was buffered and the rewrite (for example [REDACTED]) reached the client. With no config change, the raw text now does. Held mode is opt-in, and the PR itself says Noma v2 and custom code ignore that setting, so for them the opt-in doesn't help.

The ! marker and the owner's sign-off record that the risk was accepted. They don't remove it, so I'm keeping the regression open.

Evidence

litellm/proxy/common_request_processing.py:3489-3490: leaf_attrs = type(cb).__dict__; holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

litellm/proxy/common_request_processing.py:3491-3499: the buffered fallback runs only when there is no leaf apply_guardrail, or with use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan. Otherwise the guardrail is appended to relayed.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py:852-853: if ended: _write_back_stream_texts(...). The docstring at :817-819 says rewrites "reach the caller only when the stream is held until moderated".

The reply doesn't point to a code change, a test or a different observed result. It only acknowledges the behaviour and says it was approved.

Reviewed 11bf9185a4a1 · View follow-up

[litellm-stampbot]

@greptileai

[greptile-apps]

Confidence Score: 3/5

[High impact] Refactors streaming response handling for guardrails across multiple providers.

This PR should not merge yet because a stacked held guardrail can still release text without its masks

Findings

1.   Held masks disappear after blocking ▶
2.  New locals break coding rules ▶

Summary

This PR relays eligible Bedrock converse-stream replies through post-call guardrails instead of buffering the whole reply. It adds AWS error frames, Noma stream marks, disconnect cleanup, and prompt-token estimates for cut streams

yucheng-berri explicitly accepts the sampled-mode loss of masks and delivery before the verdict to match chat. The description also accepts added scan latency and vendor volume, skipped metrics and non-guardrail success hooks, approximate partial usage, late end-scan errors, delayed detection of sampled disconnects, and unchanged tool-only intervention output

New code also needs the repository’s required variable annotations

• Eligible guardrails scan Bedrock Converse frames as they arrive.
• Noma v2 scans get streaming marks and the finished Converse answer.
• A caller who leaves a held stream stops its upstream read.
• Cut Bedrock streams keep a prompt-token estimate in spend logs.

_{Reviews (2) · Last reviewed commit: "test(bedrock): expect the ValidationExce..." · Reviewed by Greptile}

[greptile-apps]

Held masks disappear after blocking

When a sampled Bedrock guardrail with disable_exception_on_block: true blocks mid-answer, these replacement frames reach the next guardrail as raw bytes. A later held masking guardrail ignores those bytes when checking whether the stream ended. It scans the earlier text but skips writing its masks back, then releases that text unmasked. Convert the replacement bytes back into ConverseStreamFrame items before passing them to later guardrails.

How this was verified: Later guardrails exclude raw bytes from their frame list, and mask write-back requires a typed messageStop or ends_stream frame.

[yucheng-berri]

bugbot run

[yucheng-berri]

@litereview full

[litereview]

High · Default sampled relay on Bedrock converse-stream sends rewriting guardrails' unmasked text to the client (still present at 8ab2027)

This finding is unchanged from the open, upheld history entry. I re-checked it at this head.

What happens

_passthrough_stream_guardrails sends any guardrail that defines apply_guardrail and isn't mask_response_content, run_in_parallel, native-hooks or release-on-scan to the frame-by-frame relay. Unless the guardrail class itself defines async_post_call_streaming_iterator_hook, it runs in sampled mode: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

In sampled mode, the unified guardrail yields copy.deepcopy(item) copies of the original frames. BedrockPassthroughGuardrailHandler.process_output_streaming_response writes rewrites back only if ended: (handler.py around line 852), after those frames have already gone out.

On base, the same request went through de_anonymize_event_stream, which calls post_call_success_hook on the buffered body and re-encodes the rewritten text. The client got masked output.

Failure scenario

1. Attach a post_call guardrail on a key whose apply_guardrail redacts text. Examples: enterprise secret detection (secret_detection.py defines apply_guardrail with no streaming iterator override), Noma v2 anonymization, Generic Guardrail API, or custom code.
2. Call /bedrock/model/<id>/converse-stream with a prompt that makes the model echo a card number or secret.
3. Base: the client's contentBlockDelta frames carry the redacted text.
4. Head: the client gets the raw text. Nothing is logged, and no config changed.

The documented opt-out, streaming_buffer_until_moderated: true, doesn't help secret detection or custom code, because the PR says those ignore the setting.

Possible fix

Keep guardrails whose apply_guardrail can rewrite output on the buffered path for this route, or default them to held mode (holds_until_moderated=True).

Evidence

Reproduced at head 8ab2027 with a real ProxyLogging.async_post_call_event_stream_iterator_hook over BedrockPassthroughGuardrailHandler.event_stream_items (the PR's own _relay_through_post_call_guardrail helper). The guardrail's apply_guardrail replaces 4111 with [X]. Output: CLIENT GOT: ['Card: 4111 1111 ', '1111 1111.', 'Thanks!'], so the redaction never reached the wire.

litellm/proxy/common_request_processing.py:3490: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs.

litellm/proxy/common_request_processing.py:3491-3498: the buffered fallback triggers only on a missing leaf apply_guardrail, use_native_lifecycle_hooks, mask_response_content, run_in_parallel or release-on-scan.

litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py:1324-1326: original_items = (... else (copy.deepcopy(item),)) holds the copies that are released after each sampled scan.

litellm/llms/bedrock/passthrough/guardrail_translation/handler.py process_output_streaming_response: if ended: _write_back_stream_texts(...), so rewrites land only after the stream ends.

The route-level test test_bedrock_converse_stream_relays_frames_for_an_apply_guardrail_guardrail uses _apply_guardrail, whose apply_guardrail returns inputs unchanged, so no test covers a rewriting guardrail on the relay.

Latest summary by full · View this review · Reviewed 8ab2027da160

[yucheng-berri]

Kept on purpose to match chat, per the ticket owner. Held mode still masks, and Noma v2 now honors it after main's #46606.

[litereview]

Finding still applies

The reply says the change is intentional and that held mode still masks. That describes a known limitation; it isn't evidence that the defect is gone. At head, a rewriting guardrail that defines apply_guardrail without a leaf streaming-iterator hook, such as CustomCodeGuardrail, still goes to the sampled relay with holds_until_moderated=False. Unredacted frames reach the client, where base buffered the response and re-encoded the masked text. Custom code can't opt into held mode per guardrail: its initializer never forwards streaming_buffer_until_moderated, and the class has no such attribute. The secret-detection example in the original finding was wrong, because that class sets use_native_lifecycle_hooks=True and stays buffered. That correction narrows the affected set but doesn't remove the defect.

Evidence

litellm/proxy/common_request_processing.py:3489-3499: holds_until_moderated = "async_post_call_streaming_iterator_hook" in leaf_attrs. The relay falls back to buffering only when the leaf lacks apply_guardrail, or when native hooks, mask_response_content, run_in_parallel or release-on-scan apply.

I ran this at head: CustomCodeGuardrail(custom_code=<modify replacing 4111>, event_hook='post_call') reports 'apply_guardrail' in leaf=True, 'async_post_call_streaming_iterator_hook' in leaf=False, use_native_lifecycle_hooks=False, mask_response_content=False, run_in_parallel=False, and streaming_buffer_until_moderated is not set. So it is relayed in sampled mode.

litellm/proxy/guardrails/guardrail_hooks/custom_code/__init__.py: initialize_guardrail passes only name, code, timeout, event_hook and default_on, never streaming_buffer_until_moderated.

litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py:1062-1068: resolve_streaming_flag reads the guardrail attribute, then guardrail_config, then the global optional params. Custom code has no attribute or config, so the default False applies.

litellm/proxy/common_request_processing.py:3748: the buffered de_anonymize_event_stream path, which re-encodes rewritten text, is now used only when _passthrough_stream_guardrails returns None.

Correction: enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py:538 sets use_native_lifecycle_hooks = True, so secret detection stays buffered and was a wrong example in the original finding.

Reviewed 8ab2027da160 · View follow-up

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8ab2027. Configure here.}

[litellm-stampbot]

@greptileai

[greptile-apps]

New locals break coding rules

AGENTS.md requires Final annotations and reasons for reassignment. These locals lack both. Satisfy that requirement before merging

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!