fix(presidio): reverting stream SSE output incrementally instead of buffering the whole response

[tin-berri]

Reverts #31503

[linear-code]

Issue reopened: LIT-3222 Presidio guardrails buffer SSE streaming responses

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing revert-31503-litellm_presidio_sse_streaming (d20413b) with litellm_internal_staging (8beb68a)}

[greptile-apps]

Greptile Summary

This PR reverts #31503, which had replaced the full-response-buffering approach in Presidio's streaming PII guardrail with an incremental per-sentence masking strategy. The revert restores buffering all ModelResponseStream chunks via stream_chunk_builder before calling _process_response_for_pii, then re-emitting the result as a single synthetic stream.

• presidio.py: Removes ~270 lines of incremental streaming helpers (_mask_emit_decision, _rewrite_chat_chunk, _build_tail_chunk, _accumulate_tool_calls, etc.) and replaces both _stream_apply_output_masking and _stream_pii_unmasking with full-buffering implementations; adds _preserve_usage_from_last_chunk to copy usage metadata the assembler misses.
• test_presidio.py: Deletes 678 lines of incremental-streaming tests and updates one mixed-stream test to match the new buffering behavior; removes the BlockedPiiEntityError import that was used in the deleted guardrail-blocking propagation test.

Confidence Score: 2/5

Not safe to merge as-is: a guardrail blocking exception raised during streaming masking is swallowed and unmasked content is forwarded to the client.

Both _stream_apply_output_masking and _stream_pii_unmasking now wrap _process_response_for_pii in a bare except Exception that logs and yields buffered chunks on any error. BlockedPiiEntityError and GuardrailRaisedException fall into that handler, so a streaming request that should be blocked instead delivers unmasked content silently. The deleted test_mask_streaming_propagates_guardrail_interventions test was the only CI guard for this contract.

Both changed files need attention: presidio.py for the exception-handling regression in the two streaming methods, and test_presidio.py for the missing guardrail-blocking propagation coverage.

Security Review

• Guardrail bypass on masking error (_stream_apply_output_masking): The bare except Exception handler wraps _process_response_for_pii, which raises BlockedPiiEntityError or GuardrailRaisedException when the guardrail action is BLOCK. These exceptions are silently caught, logged, and the original unmasked chunks are yielded to the client.
• Same bypass in unmasking path (_stream_pii_unmasking): Identical broad handler absorbs guardrail intervention exceptions without re-raising.
• Deleted coverage: test_mask_streaming_propagates_guardrail_interventions was the sole automated regression test for the blocking contract.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/presidio.py	Reverts incremental streaming to full-response buffering; introduces a broad exception handler that swallows guardrail blocking exceptions (BlockedPiiEntityError / GuardrailRaisedException), causing unmasked PII to pass through to the client instead of blocking the request.
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_presidio.py	Removes 678 lines of incremental-streaming tests including the only test that verified guardrail blocking exceptions propagate during streaming; the deletion leaves the regression undetected.

Comments Outside Diff (1)

1. tests/test_litellm/proxy/guardrails/guardrail_hooks/test_presidio.py, line 612 (link)

   Removal of guardrail-intervention propagation test

   BlockedPiiEntityError is no longer imported here because test_mask_streaming_propagates_guardrail_interventions was deleted as part of this revert. That test was the only coverage verifying that BlockedPiiEntityError raised by Presidio during streaming causes the request to be blocked rather than passing unmasked content through. Without it, the regression described in _stream_apply_output_masking has no automated detector.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "Revert "fix(presidio): stream SSE output..." | Re-trigger Greptile}

[greptile-apps]

Guardrail blocking exceptions silently swallowed

The broad except Exception catch covers _process_response_for_pii (line 1112), which raises BlockedPiiEntityError or GuardrailRaisedException when the guardrail's action is BLOCK. With this handler, those exceptions are logged and discarded, and the original unmasked chunks buffered in all_chunks are then yielded directly to the client. The removed test test_mask_streaming_propagates_guardrail_interventions specifically validated this contract. A minimal fix is to check self._is_guardrail_intervention(e) and re-raise before falling back to passthrough.

[greptile-apps]

Same guardrail-intervention bypass in unmasking path

_stream_pii_unmasking has the same broad except Exception wrapping _process_response_for_pii (mode="unmask"). A GuardrailRaisedException raised there is caught, logged, and the loop exits via for chunk in remaining_chunks: yield chunk, silently passing through content without re-raising. Guardrail interventions from the unmask path should propagate just like those from the mask path.

[codecov]

Codecov Report

❌ Patch coverage is 68.00000% with 24 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/proxy/guardrails/guardrail_hooks/presidio.py	68.00%	24 Missing ⚠️

📢 Thoughts on this report? Let us know!

[veria-ai]

Medium: Blocked PII is replayed on streaming masking errors

A user can now receive streamed content that Presidio was configured to block: _process_response_for_pii raises BlockedPiiEntityError/GuardrailRaisedException, but this catch logs the error and replays all_chunks unmodified. Re-raise guardrail interventions here and avoid falling back to the original chunks when output masking fails.

[veria-ai]

Medium: Unbounded buffering of streaming output

An authenticated user can keep the proxy accumulating every ModelResponseStream chunk in memory by requesting a large streaming completion, because this path appends until the upstream stream ends before masking or yielding. Keep a bounded buffer and process incrementally, or enforce a hard cap that fails closed instead of holding the entire streamed response.

[veria-ai]

PR overview

This PR changes the Presidio guardrail streaming response path for SSE/model stream output in the LiteLLM proxy, aiming to handle streamed chunks incrementally instead of waiting for the full response before returning data.

There are still two open issues in the streaming Presidio handling. The current path can fall back to replaying unmodified streamed chunks when masking or blocking fails, which can expose content that the guardrail was configured to stop. It also still buffers streamed output without a hard bound, allowing an authenticated caller to drive excessive memory use with a large streaming response. No issues have been fixed or addressed yet, so the PR still carries meaningful security risk.

Open issues (2)

• Medium: Blocked PII is replayed on streaming masking errors — litellm/proxy/guardrails/guardrail_hooks/presidio.py:1121
• Medium: Unbounded buffering of streaming output — litellm/proxy/guardrails/guardrail_hooks/presidio.py:1071

Fixed/addressed: 0 · PR risk: 7/10