test(guardrails): e2e validation of streaming_transform_mode: incremental_diff (PR #32084)

[mateo-berri]

Relevant issues

Validates PR #32084 (generic_guardrail_api streaming text transform + ComplianceChecker mode-match fix)

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem

Screenshots / Proof of Fix

Live e2e validation of PR #32084 against a real LiteLLM proxy (gpt-4o-mini, real OpenAI API, real $$$). All 6 cases below.

Setup: mock guardrail server on port 8888, proxy on port 4000 with generic_guardrail_api configured, streaming_transform_mode: incremental_diff, streaming_sampling_rate: 1.

pr32084_streaming_guardrail_e2e_validation.mp4

---

Test 1: Basic streaming rewrite

Guardrail mode: incremental_diff. Server: replace SENSITIVEVALUE_123 -> [REDACTED_123] (with stream_holdback_chars to prevent partial prefix exposure).

Command:

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"rewrite_sensitive"}'
curl -sN -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-test-guardrail-1234" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Output exactly: \"The secret code is SENSITIVEVALUE_123 and nothing else.\""}],"stream":true,"temperature":0,"max_tokens":30,"metadata":{"guardrails":["incremental-diff-guardrail"]}}'

Raw chunks received by client:

data: {"choices":[{"delta":{"role":"assistant","content":"The"}}]}
data: {"choices":[{"delta":{"content":" secret"}}]}
data: {"choices":[{"delta":{"content":" code"}}]}
data: {"choices":[{"delta":{"content":" is"}}]}
data: {"choices":[{"delta":{"content":" "}}]}
data: {"choices":[{"delta":{"content":"[REDACTED_123]"}}]}
data: {"choices":[{"delta":{"content":" and"}}]}
data: {"choices":[{"delta":{"content":" nothing"}}]}
data: {"choices":[{"delta":{"content":" else"}}]}
data: {"choices":[{"delta":{"content":"."}}]}
data: {"choices":[{"delta":{"content":""},"finish_reason":"stop"}]}
data: [DONE]

Assembled client text: The secret code is [REDACTED_123] and nothing else.
Raw SENSITIVEVALUE_123 in any chunk? No

Test 1 - basic streaming rewrite showing [REDACTED_123]

---

Test 2: Holdback behavior

Server holds back partial prefix of "Thomas Weber" across chunk boundaries, replaces once full name arrives.

Command:

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"holdback_name"}'
curl -sN ... "Output exactly: \"The patient Thomas Weber had condition XYZ.\"" --guardrails incremental-diff-guardrail

Raw chunks: The / patient / / [NAME_REDACTED] / had / condition / XYZ / .

Assembled: The patient [NAME_REDACTED] had condition XYZ.
Raw Thomas Weber in any chunk? No. Partial Thomas W in any chunk? No

Test 2 - holdback preventing Thomas Weber from reaching client

---

Test 3: Prefix-rewrite / underflow failure

Server returns NONE for partial "John" (no holdback - this is the misconfiguration). When "John Smith" arrives at end of stream, tries to return "The patient is [NAME]." - but "John" was already emitted. Underflow detected.

Command:

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"underflow_trigger"}'
curl -vN ... "Output exactly: \"The patient is John Smith.\"" --guardrails incremental-diff-guardrail

Client received: The / patient / is / John then:

* transfer closed with outstanding read data remaining
curl: (18) transfer closed with outstanding read data remaining

No data: [DONE]. Stream terminated abruptly (fail-closed). For non-A2A endpoints, the underflow raises HTTPException(400, stream_transform_underflow) which closes the HTTP connection. For A2A (JSON-RPC), it emits an in-stream JSON-RPC error instead.

Test 3 - underflow causes abrupt stream termination

---

Test 4: Tool-call stream behavior

Tool call chunks pass through raw (before inspection). End-of-stream block-only check inspects tool calls and blocks.

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"block_tool_call"}'
curl -sN ... --tools get_weather --tool_choice required --guardrails incremental-diff-guardrail

Tool call chunks delivered raw to client first:

data: {"choices":[{"delta":{"tool_calls":[{"id":"call_hEPOUzLQ...","function":{"name":"get_weather"},...}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"{\"city\":\"Seattle\"}"}}]}}]}

Then after all chunks accumulated, guardrail ran block-only check over assembled tool calls and fired:

data: {"error": {"message": "Tool call blocked by policy: sensitive arguments detected", "code": "400"}}

Tool call chunks: delivered before inspection (end-of-stream inspection only, not block-before-delivery).

---

Test 5: Mixed text + tool-call

Text portion rewritten via incremental_diff. Tool call chunks passed through raw (identifiable by system_fingerprint/obfuscation fields from raw OpenAI response; synthetic text chunks lack those).

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"rewrite_sensitive"}'
curl -sN ... "First say 'The code is SENSITIVEVALUE_123.' Then call get_info" --tools get_info --guardrails incremental-diff-guardrail

Assembled text: The code is [REDACTED_123]. Now, I will call the get_info function...
Raw SENSITIVEVALUE_123? No. Tool call chunks present? Yes (get_info). Synthetic text chunks mixed with raw tool call chunks in same stream.

---

Test 6: Baseline compatibility (block_only)

Same guardrail server (rewrite_sensitive) but using block-only-guardrail (streaming_transform_mode: block_only). Rewrites dropped; raw value reaches client.

curl -s -X POST http://127.0.0.1:8888/set_mode -d '{"mode":"rewrite_sensitive"}'
curl -sN ... "The secret code is SENSITIVEVALUE_123..." --guardrails block-only-guardrail

Raw chunks: S / ENSITIVE / VALUE / _ / 123 all visible to client.
Assembled: The secret code is SENSITIVEVALUE_123 and nothing else.
incremental_diff behavior: not activated (backward compatible; opt-in only)

Test 6 - block_only mode drops rewrites, raw value visible

---

Potential regressions / backward incompatible changes

1. No change for existing users: streaming_transform_mode defaults to block_only (exact pre-PR behavior).

2. Tool calls in incremental_diff mode: tool call argument chunks are delivered to the client before the end-of-stream block check fires. This was always true for block_only with streaming_end_of_stream_only: false too (sampled block can fire mid-stream but tool call chunks always pass raw). No regression.

3. Underflow for OpenAI chat completions: HTTP connection close rather than explicit in-stream SSE error event. For A2A the PR correctly emits a JSON-RPC error. Non-A2A clients need to handle connection close as an error indicator (no data: [DONE]).

4. ComplianceChecker mode-match fix: guardrails configured with guardrail_mode: [list] were never matching before this PR. This is a bug fix but changes behavior for list-mode guardrails. Previously-skipped compliance checks will now run.

Type

✅ Test

Changes

No code changes; this branch merges PR #32084 from the contributor fork onto litellm_internal_staging to provide live proxy validation.

_{To show artifacts inline, enable in settings.}

  

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ schneidermr
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 86.53846% with 28 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...drail_hooks/unified_guardrail/unified_guardrail.py	86.42%	19 Missing ⚠️
.../llms/openai/chat/guardrail_translation/handler.py	81.63%	9 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_guardrail_streaming_test-2497 (c50a05f) with litellm_internal_staging (68a4ca7)}