fix(presidio): stream SSE output incrementally instead of buffering the whole response

[yassin-berriai]

Relevant issues

Resolves LIT-3222

Linear ticket

LIT-3222

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

With Presidio output handling enabled, the streaming post-call hooks buffered the entire upstream stream, ran Presidio over the reassembled completion, then emitted one reconstructed SSE chunk. Time-to-first-token collapsed to the full generation time and token-by-token streaming was lost. Note that the default presidio_filter_scope: both always instantiates an apply_to_output masking callback, so even an output_parse_pii configuration buffered the stream through that path. Both output paths now transform and forward chunks as they arrive.

Repro is a live proxy on real gpt-4o-mini streaming with the official Presidio analyzer + anonymizer docker images, swapping only presidio.py between the buffered and incremental versions. The client records the wall-clock arrival of every SSE chunk that carries delta.content. The prompt asks the model to generate a name, email, and city so the apply_to_output masking path (the one that was buffering under the default scope) is exercised end to end.

[BEFORE fix (buffered)]
  content_chunks      : 1
  TTFT (s)            : 2.749        # equals total stream time
  total stream (s)    : 2.749
  arrival timeline(s) : [2.75]       # everything arrives in one chunk at the end
  masked PII tokens   : ['<EMAIL_ADDRESS>', '<LOCATION>', '<PERSON>']

[AFTER fix (incremental)]
  content_chunks      : 9
  TTFT (s)            : 1.744
  total stream (s)    : 2.644
  arrival timeline(s) : [1.74, 1.79, 1.8, 2.11, 2.14, 2.18, 2.46, 2.46, 2.64]
  masked PII tokens   : ['<EMAIL_ADDRESS>', '<PERSON>']

Content arrives progressively after the fix instead of in a single delta at end-of-stream, while the model-generated name and email stay masked (<PERSON>, <EMAIL_ADDRESS>) with no raw PII forwarded. A guardrail-free request on the same proxy streams 22 progressive chunks, confirming the upstream model streams normally and the buffering was entirely in the Presidio hooks.

To reproduce locally

docker run -d --name presidio-analyzer  -p 5610:3000 mcr.microsoft.com/presidio-analyzer:latest
docker run -d --name presidio-anonymizer -p 5611:3000 mcr.microsoft.com/presidio-anonymizer:latest
# config.yaml: one gpt-4o-mini model + a presidio guardrail (mode pre_call,
# default_on true, output_parse_pii true, analyzer/anonymizer bases above);
# presidio_filter_scope defaults to both, so an apply_to_output instance is created
python litellm/proxy/proxy_cli.py --config config.yaml --port 4222
curl -N http://127.0.0.1:4222/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Write an 80-word fictional press release announcing that engineer Marcus Bennett, reachable at marcus.bennett@example.com in Denver, joined the team. Mention his name, email, and city at least twice."}]}'

Type

🐛 Bug Fix

Changes

_stream_apply_output_masking (apply_to_output) and _stream_pii_unmasking (output_parse_pii) in litellm/proxy/guardrails/guardrail_hooks/presidio.py no longer call stream_chunk_builder to reassemble the whole completion. They share a small incremental rewriter that mutates each chat chunk in place:

• the unmask path replaces placeholder tokens per chunk and holds back only the trailing run that could still complete into a token, so a placeholder split across SSE chunks (<PER + SON_1>) is still rewritten atomically
• the mask path detects model-generated PII, which an incremental flush could split across a boundary and leak, so it emits a prefix only when masking that prefix in isolation matches the corresponding prefix of masking the whole buffer, with at least _PRESIDIO_STREAM_MARGIN characters of lookahead still buffered past the cut; any entity straddling the cut makes the two maskings differ, so the cut is held until the entity completes and is masked as one unit. Past _PRESIDIO_STREAM_MAX_BUFFER with no safe cut the run is bounded without splitting an entity
• tool-call and legacy function-call argument fragments are accumulated per choice and transformed once the choice finishes, so masking stays correct without ever forwarding partial PII
• content is buffered independently per choice index, so n>1 streams stay correct
• raw Anthropic SSE bytes and /v1/responses events pass through untouched; any buffered masked or held content is flushed before such an event so a client never observes a later event ahead of earlier transformed text
• if Presidio fails while masking a chunk, that chunk is redacted in place (fail closed, keeping its finish_reason) and the stream keeps flowing to termination rather than truncating the whole response; intentional guardrail interventions (BlockedPiiEntityError, GuardrailRaisedException) still propagate

Tests extend the mapped file tests/test_litellm/proxy/guardrails/guardrail_hooks/test_presidio.py with regressions for incremental emission, cross-chunk token splits, per-choice independence, tool-call argument masking/unmasking, the no-split guarantee for an entity straddling a boundary, the buffer cap masking a runaway run without splitting an entity, flush-before-bytes ordering, terminal-chunk and tail masking-error survival (finish_reason preserved, fail closed), and guardrail-intervention propagation. Each fails on the unhardened implementation.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR changes Presidio streaming output handling to rewrite chunks incrementally. The main changes are:

• Incremental masking for apply_to_output chat streams without buffering the full response
• Incremental PII unmasking for output_parse_pii streams with split-token carry handling
• Per-choice buffering for multi-choice streams
• Tool-call and legacy function-call argument accumulation before masking or unmasking
• Ordered flushing before raw bytes and /v1/responses passthrough events
• Fail-closed handling for transient masking errors while preserving stream completion signals
• Tests for streaming behavior, split entities, passthrough ordering, tool calls, buffer caps, and guardrail interventions

Confidence Score: 5/5

The streaming rewrite appears merge-safe with focused test coverage around incremental Presidio masking, unmasking, ordering, and error handling.

The change is scoped to Presidio streaming output handling and includes targeted regressions for the main edge cases introduced by incremental rewriting, including split tokens, multi-choice streams, tool/function arguments, passthrough ordering, buffer caps, and failure behavior.

T-Rex Logs

What T-Rex did

• Ran the before/base incremental scenario and verified the output was buffered into a single chunk with PII masked.
• Ran the before/base error and intervention scenarios and confirmed that raw emails leaked and the runner completed.
• Ran the after/head incremental scenario and observed six timeline entries with no raw PII; non-guardrail errors produced redacted chunks and the guardrail intervention raised BlockedPiiEntityError after prior safe empty yields.
• Captured artifacts including logs and a Python artifact to document the run results.
• Compared base and head runs and confirmed base buffered all output until the final stop chunk, while head emitted progressively across multiple chunks with timing metrics.
• Confirmed both runs used the same command shape and harness, and that artifacts included the command, working directory, verbose arrivals, a summary JSON, and an exit code.
• Opened unmasking artifacts and confirmed five yielded chat chunks with no partial placeholders and unmasked output such as Hello Alice Smith, mail alice@example.com.
• Examined the head unmask artifact and observed twelve items in order, including chat text, raw SSE bytes, tool-call terminal args transformed to JSON, legacy function args transformed to JSON, a completion event, and that a partial placeholder was observed.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   Mixed non-chat boundary can still expose a split placeholder suffix

   • Bug
     • When a chat content chunk ending with a possible placeholder prefix is followed by a non-chat /v1/responses event and then another chat chunk containing the remainder of the placeholder, head flushes state before the event but later emits the suffix (_1>) as raw content. The after artifact records PARTIAL_PLACEHOLDER_OBSERVED True and joined content ending in alice@example.com_1>, violating the requested atomic split-token/no-partial-placeholder contract for mixed stream ordering.
   • Cause
     • In litellm/proxy/guardrails/guardrail_hooks/presidio.py, _stream_pii_unmasking flushes and clears content_buffers, tool_acc, and func_acc before yielding bytes or other non-ModelResponseStream events (around lines 1501-1518). That preserves event ordering, but it also discards the cross-chunk token-prefix context needed to recognize a placeholder whose suffix arrives after the non-chat event.
   • Fix
     • Either document that placeholder tokens cannot span non-chat event boundaries, or keep enough per-choice token-prefix context across non-chat passthrough events to suppress/repair a following suffix without reordering the non-chat event. Add a regression test matching the captured sequence: chat Tail <EMAIL, response.completed, chat _1> should not emit _1> raw.

   _{Ran code and verified through T-Rex}

_{Reviews (11): Last reviewed commit: "fix(presidio): stream SSE output increme..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 90.14778% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/proxy/guardrails/guardrail_hooks/presidio.py	90.14%	20 Missing ⚠️

📢 Thoughts on this report? Let us know!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 4 · PR risk: 0/10

[yassin-berriai]

Addressed the 4/5 finding: in the apply_to_output path, buffered masked content is now flushed before any non-chat passthrough event (raw bytes or a /v1/responses completion), so a client never sees stream completion ahead of the final masked text. This mirrors the unmask path, which already flushed-before-passthrough. Added a regression test (test_mask_streaming_flushes_buffered_content_before_passthrough_event) that fails without the flush.

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

Pushed fixes for the three open streaming findings.

The forced-flush window (_PRESIDIO_STREAM_MAX_BUFFER) that could split a space-bearing entity across two analyze calls is removed; content now holds until a real sentence boundary or end-of-stream, so an entity is never analyzed in halves and leaked unmasked. The unmask path flushes held chat content before yielding a raw byte chunk, matching the mask path. A masking error mid-stream now drops only the offending chunk and keeps streaming, failing closed so no unmasked content reaches the client.

Each fix has a regression that fails on the prior code: test_mask_streaming_does_not_split_entity_on_long_unpunctuated_run, test_unmask_streaming_flushes_held_content_before_bytes, test_mask_streaming_preserves_stream_on_check_pii_error. Live proof of progressive masked streaming is in the PR description.

@greptileai

[yassin-berriai]

Addressed the terminal-chunk follow-up: a masking error now redacts the chunk in place (empty content, original finish_reason and indices preserved) instead of dropping it, so the completion signal is never lost while still failing closed. Added test_mask_streaming_preserves_finish_reason_when_terminal_chunk_fails.

@greptileai

[yassin-berriai]

Fixed the boundary-at-EOF case: a sentence terminator at the end of a chunk is no longer flushed until the following whitespace arrives, so an email split across the chunk seam (Contact jane. + doe@example.com) is analyzed whole instead of leaking the jane. fragment. Added test_mask_streaming_holds_terminator_at_chunk_end_until_whitespace.

@greptileai

[yassin-berriai]

Reworked the output-masking path to stream incrementally while staying leak-safe, addressing the latest findings.

Instead of trusting a sentence boundary (which can fall inside a name with an initial or an address across a newline), the path emits a prefix only when masking it in isolation matches the prefix of masking the whole buffer, with a lookahead margin still buffered past the cut; a straddling entity makes those differ and is held until complete. Buffering is bounded by a cap that masks a runaway punctuation-free run as one unit, and a masking error redacts only the affected chunk (or drops only the held tail) while the stream continues. Verified on a live proxy: the apply_to_output path now streams progressively (five chunks) with name, email and city all masked.

@greptileai

[yassin-berriai]

Addressed the cap finding: the forced flush at the buffer cap now cuts a margin back from the end under the same stability check, so a PII value split exactly at the cap stays buffered and is masked whole rather than analyzed in halves. The other two new comments are a no-op interleaving (parsed chat chunks and raw Anthropic byte chunks never mix mid-token) and a passing assertion (the held placeholder prefix unmasks via the truncated-token fallback); details in the inline replies.

@greptileai

[greptile-apps]

All three are solid:

Cap fix (buffer bypass): The stability-check cut with a trailing margin is the right approach. Emitting buffer[:cut_point] while keeping buffer[cut_point:] (≥ _PRESIDIO_STREAM_MARGIN chars) in the hold means the Presidio call at the forced flush never sees an incomplete entity — any value that straddles the cut boundary will make the stability check differ and hold the cut until the entity resolves. Memory is still bounded at _PRESIDIO_STREAM_MAX_BUFFER. Acknowledged.

Carry split tokens (bytes interleaving): Already acknowledged in the thread — the Anthropic-native byte path and the parsed ModelResponseStream path are mutually exclusive by construction, so content_buffers is empty at the bytes branch. No issue.

Impossible assertion (<PERSON → Jane): The truncated-token fallback is valid here. min(20, len("<PERSON_1>") // 2) = min(20, 5) = 5, and the held prefix <PERSON has 7 chars of overlap — so 7 ≥ 5 satisfies the threshold and the partial is substituted with the original PII value. The test is covering real behavior (flush-before-bytes ordering + fallback unmasking of the incomplete prefix), not an impossible state. Acknowledged.

All open findings resolved. The incremental streaming approach is sound, PII safety invariants are maintained across the boundary cases, and the regression tests pin the guarantees. This looks good to merge.

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Masking error drops tool calls
  • Streaming masking errors no longer clear accumulated tool/function state globally, and terminal tool/function buffers are only popped after successful rebuild.
• ✅ Fixed: Buffer cap not enforced
  • The stream masking buffer now bounds held state even when stability checks fail by retaining only the trailing safety margin after forced cap handling.
• ✅ Fixed: Swallows intentional guardrail blocks
  • Intentional guardrail interventions are re-raised from chunk, tail, and outer streaming masking error handlers instead of being redacted and swallowed.

_{You can send follow-ups to the cloud agent here.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 32d15d4. Configure here.}

[yassin-berriai]

Rebased onto the current litellm_internal_staging and squashed to a single commit. Reformatted presidio.py with ruff format (line-length 120 per ruff.toml), which dropped roughly 220 lines of stale 88-column reflow churn that had inflated the diff; the logic is unchanged from the last reviewed commit and all prior findings stay addressed (flush-before-passthrough on both paths, fail-closed per-chunk masking that preserves finish_reason, the boundary stability check with lookahead margin, the bounded buffer cap that never splits an entity, and propagation of intentional guardrail interventions).

Both output paths are retained on purpose. The default presidio_filter_scope: both always instantiates an apply_to_output masking callback, so that path has to stream too or the response stays buffered even for an output_parse_pii setup; the updated live proof in the description shows the apply_to_output path going from 1 buffered chunk to 9 progressive chunks with the model-generated name and email still masked.

@greptileai

[yassin-berriai]

@greptileai please re-review the latest commit (ad59bfece0)

[yassin-berriai]

CI note: the three red checks are flaky external-API e2e jobs, not this diff.

`ci/circleci: proxy_logging_guardrails_model_info_tests`, `ci/circleci: proxy_multi_instance_tests`, and `ci/circleci: proxy_pass_through_endpoint_tests` are all green on the `litellm_internal_staging` base commit and on several other open PRs at the same time. Their proxy configs (e.g. `otel_test_config.yaml`) load only bedrock and custom guardrails, no Presidio, so this change cannot reach them; the proxy also boots cleanly with Presidio active. Every job that does exercise this change is green: `lint`, `ci/circleci: guardrails_testing`, `guardrails-hooks / Run tests` (the mapped unit tests for this file), and all of the unit-test shards. The three failing jobs hit real OpenAI/Bedrock/Cohere/Datadog/GCS endpoints, which is the usual flaky surface, and a Rerun failed should clear them

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_presidio_sse_streaming (d4ed88c) with litellm_internal_staging (59f51b2)}