feat(guardrails): streaming text transformation in generic_guardrail_api

[yucheng-berri]

Title

feat(guardrails): streaming text transformation in generic_guardrail_api

Relevant issues

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks
• My PR's scope is as isolated as possible
• I have received a Greptile Confidence Score of at least 4/5

Type

New Feature

Changes

Adds an opt-in streaming_transform_mode: incremental_diff config on generic_guardrail_api so an HTTP guardrail can rewrite streamed response text (PII masking, pseudonym reversal, redaction) instead of only blocking on it.

Default block_only preserves historical behavior: the guardrail can terminate a stream via a BLOCKED action, but GUARDRAIL_INTERVENED text rewrites are silently dropped. incremental_diff withholds the raw upstream chunks, runs the guardrail every N chunks over the accumulated text, and emits the guardrail's rewritten text as synthetic deltas by diffing the mutated accumulated text against what has already been streamed.

Companion stream_holdback_chars on the guardrail response contract lets a guardrail server request the framework withhold N trailing chars per choice (word-boundary safety across rounds).

v1 scope:

• Only the OpenAI chat completions streaming path supports incremental_diff; other routes silently fall back to block_only.
• Only delta.content text is transformed. delta.tool_calls chunks pass through with content stripped and are inspected for a block decision at end of stream (matching block_only), but never text-rewritten.
• A rewrite that is not a forward extension of what has already been streamed is failed closed with HTTP 400 stream_transform_underflow; a guardrail that needs to rewrite recent output must withhold it first via stream_holdback_chars.

Credit

Adopted from #32084 by @schneidermr (PalenaAI). Mirrored onto a litellm_ branch so CircleCI and the internal lint workflow run against the current HEAD.

This is a fresh replacement for #32915 (closed due to accumulated force-push noise); the branch is now clean: OSS PR head → staging merge → one fix commit addressing five correctness bugs surfaced during review.

Original PR: #32084
Prior adoption PR (closed): #32915

Five correctness fixes on top of the OSS PR

Single commit addressing:

1. Mixed content+tool_call finish_reason deferral — _tool_call_passthrough_chunk defers finish_reason for choices carrying both text and tool_calls so a terminator chunk delivers it. Without this, SSE-compliant clients stopping at finish_reason silently drop the guardrailed text. (Greptile P1 twice, Veria.)

2. Choice-index sort in _process_streaming_transform — indices/texts_to_check sorted by StreamingChoices.index so n>1 streams don't map guardrail-returned texts to the wrong choice indices. (Bugbot Medium.)

3. Cross-chunk pre-tool-call text flush — text chunks trailed by a pure tool-call chunk carrying finish_reason="tool_calls" flush any pending transformed text via _round(is_final=False) before yielding the passthrough. Same class as Give me consistent exceptions  #1 but cross-chunk. (Greptile P1.)

4. Terminator chunk on empty mutated_text_per_choice — _build_transform_chunk emits a terminator chunk on the final flush when the guardrail returned empty texts, so a deferred finish_reason (from Give me consistent exceptions  #1) is delivered even when the guardrail suppresses the text. (Bugbot High.)

5. Deep-copy responses_so_far before end-of-stream block inspection — _inspect_full_response_for_block now receives copy.deepcopy(responses_so_far). Without this, the block path's in-place mutation of delta.content corrupts the shared accumulator, causing the subsequent final _round to see already-guardrailed text and re-apply the guardrail (silent double-application for non-idempotent guardrails, or stream_transform_underflow from mismatched prefixes). (Greptile P1.)

All five fixes are scoped to the incremental_diff code paths (reachable only behind streaming_transform_mode == "incremental_diff" or stream_transform_sink is not None). No changes to block_only, non-streaming, pre_call, moderation, or sibling guardrails.

Also normalizes Optional[X] → X | None across the OSS PR's added surface via ruff UP045 autofix + F401 unused-import cleanup (required to pass the strict-rule budget gate; mechanical typing style change, no semantic effect).

Regression tests added for all five fixes.

Follow-ups (opt-in path only, no correctness impact — will be separate PRs)

#	Limitation	Suggested fix
5	Terminal chunk guardrailed twice when sampling boundary lands on it	Add not _chunk_has_finish_reason(item) gate on the sampled round
6	Every tool-call chunk after text fires redundant guardrail HTTP call producing no delta (perf)	Dirty-flag pattern (text_pending_flush set on text, cleared after any _round)
7	stream_options.include_usage chunks dropped on incremental_diff	Detect via _is_usage_only_chunk, pass through raw
8	n>1 streams where text choice finishes in a separate chunk from tool-call: text choice's finish_reason can be dropped	Extend final-round guard to fire when finish_reason_per_choice has entries
9	SSE mid-stream raise exc on non-A2A: re-raises rather than emitting clean SSE error frame	Add build_block_sse_chunks override on OpenAIChatCompletionsHandler (fixes framework-wide, not just this PR)
10	Empty texts: [] in GUARDRAIL_INTERVENED treated as no-op across all modes	Introduce distinct SUPPRESSED action code
11	Tool-call arguments not text-rewritten in v1 — only inspected at EOS	v2: add tool_call_arguments_texts to inputs; per-chunk redaction; diff-emit like text deltas
12	streaming_buffer_until_moderated ignored under incremental_diff	Runtime warning when both flags set
13	has_stream_ended inspects only choices[0] (pre-existing block_only limitation)	any(c.finish_reason is not None for c in chunk.choices)
14	docs.litellm.ai streaming section missing	External docs repo — add "Streaming text transformations" section
15	stream_transform_underflow manifests as ChunkedEncodingError on SSE clients (tied to #9)	Same fix as #9
16	streaming_end_of_stream_only=True silently violated by the cross-chunk pre-tool-call text flush (fix #3 above)	Skip the pre-flush when end_of_stream_only=True (accepts mixed-chunk finish_reason re-break under that config), or add runtime warning that the flag is best-effort under incremental_diff
17	n>1 mixed-chunk edge case: when text arrives on choice 0 (no finish_reason) and tool_calls with finish_reason on choice 1 in the same chunk, the pre-tool-call flush is skipped because saw_text_content=False at entry. Choice 1's passthrough emits finish_reason immediately, SSE clients stop reading, and choice 0's transformed text is lost. Fix #1's _tool_call_passthrough_chunk defer only fires when the same choice has both content and tool_calls, not the cross-choice n>1 form. (Bugbot Medium)	Extend the pre-flush trigger to also fire when the current chunk carries text on any choice (inspect the chunk with _chunk_carries_text before the tool-call branch), OR strip finish_reason from all passthrough choices when any choice in the chunk has text pending
18	Pre-tool-call _round(is_final=False) honors stream_holdback_chars, so trailing text can be withheld. The subsequent tool-call passthrough emits finish_reason, SSE clients stop reading, and the held-back text is dropped when the EOS flush runs. Direct interaction with Fix #3. (Bugbot High)	Have the pre-tool-call flush call a variant of _round that forces holdback to 0 (equivalent to is_final=True semantics for holdback purposes only, without triggering the terminator branch), OR emit an additional "final-holdback-release" round before the passthrough

---

Note

High Risk
Large new streaming proxy path touches guardrail output delivery, SSE ordering, and fail-closed redaction semantics; mistakes could leak raw text or break tool-calling streams, though default block_only is unchanged.

Overview
Adds opt-in streaming_transform_mode: incremental_diff on generic_guardrail_api so HTTP guardrails can rewrite streamed chat text (masking, redaction) instead of only blocking. Default block_only is unchanged: upstream chunks still stream and GUARDRAIL_INTERVENED text edits are dropped.

With incremental_diff, the unified streaming hook withholds raw chunks, runs the guardrail on accumulated string delta.content on a sampling schedule (and at EOS), and emits synthetic deltas by diffing guardrailed text against what was already sent. The API contract gains stream_holdback_chars (with coercion) so the server can withhold trailing chars across rounds; non-forward rewrites fail closed with stream_transform_underflow.

StreamTransformSink and OpenAI chat _process_streaming_transform keep the raw accumulator immutable while reporting per-choice mutated text and holdback. The incremental path adds tool-call passthrough (text stripped), EOS block inspection on a deep-copied accumulator, deferred finish_reason for mixed text+tool chunks, pre-tool-call text flush, sorted n>1 choice indices, and terminator chunks when the guardrail returns no text.

v1 is OpenAI chat completions streaming with string content only; other routes fall back to block_only. Extensive regression tests cover the new mode and the five correctness fixes.

^{Reviewed by Cursor Bugbot for commit f980922. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yucheng-berri]

@greptileai

[greptile-apps]

Greptile Summary

This PR adds an opt-in streaming_transform_mode: incremental_diff to generic_guardrail_api that allows an HTTP guardrail to rewrite streamed response text (PII masking, redaction) by withholding raw chunks, running the guardrail over accumulated text every N chunks, and emitting synthetic deltas computed by diffing the mutated text against what has already been sent to the client. The default block_only mode is fully preserved.

• The new incremental_diff path lives entirely behind a feature flag and is scoped to the OpenAI chat completions streaming surface; other routes silently fall back to block_only with a warning log.
• Five correctness bugs from the original OSS PR are addressed: finish_reason deferral for mixed content+tool_call chunks (Fix Give me consistent exceptions  #1), choice-index sort for n>1 streams (Fix Enable model / call timeouts #2), pre-tool-call text flush (Fix Guarantee format of exceptions #3), terminator chunk on empty guardrail text (Fix Add support for replicate [code in ticket] #4), and copy.deepcopy(responses_so_far) before the end-of-stream block inspection to prevent the block path's in-place content mutation from corrupting the raw accumulator (Fix Update main.py #5).
• All new tests are mock-based with no real network calls.

Confidence Score: 5/5

Safe to merge; the feature is entirely opt-in behind streaming_transform_mode=incremental_diff and the default block_only path is unchanged.

Fix #5 (copy.deepcopy(responses_so_far) at the _inspect_full_response_for_block call site) correctly addresses the shared-accumulator mutation concern from the prior review. The StreamTransformSink out-parameter protocol ensures the transform path never mutates responses_so_far. All five correctness fixes are implemented correctly and each has a dedicated regression test. The tests use only mocks with no real network calls. No new issues were found on this HEAD.

No files require special attention; the core transform logic in unified_guardrail.py and handler.py is well-covered by the added tests.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py	Core of the PR: adds the _run_incremental_transform_stream path and five correctness fixes; Fix #5 (deepcopy before block inspection) properly addresses the previous P1 about shared-accumulator corruption.
litellm/llms/openai/chat/guardrail_translation/handler.py	Adds _process_streaming_transform / _process_streaming_block_only split and the StreamTransformSink out-parameter protocol; does not mutate responses_so_far, preserving it as a correct raw accumulator across rounds.
litellm/llms/base_llm/guardrail_translation/base_translation.py	Introduces StreamTransformSink dataclass as a write-once out-parameter; base-class process_output_streaming_response accepts it but ignores it (backwards-compatible).
litellm/types/proxy/guardrails/guardrail_hooks/generic_guardrail_api.py	Adds streaming_transform_mode config field, stream_holdback_chars to the response model and coerce_stream_holdback_value helper; from_dict gracefully degrades malformed holdback values to 0.
tests/test_litellm/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py	Adds 17 regression tests covering all five correctness fixes and the main incremental_diff path; all mock-based, comprehensive coverage of n>1, holdback, underflow, mixed content+tool_call, and finish_reason deferral scenarios.

_{Reviews (5): Last reviewed commit: "fix(guardrails): four correctness fixes ..." | Re-trigger Greptile}

[greptile-apps]

Silent bypass of streaming_buffer_until_moderated

When both streaming_transform_mode="incremental_diff" and streaming_buffer_until_moderated=True are configured, the incremental_diff branch returns early (line 899) and the buffer logic never runs. A user who sets both flags expecting that all chunks are withheld until end-of-stream moderation passes will instead silently get incremental_diff behavior — raw chunks are withheld by a different mechanism, but without the usual "release all original chunks on pass / emit block message on fail" semantics. No warning is logged. Follow-up #12 identifies this gap but does not add the missing warning.

[greptile-apps]

end_of_stream_only=True is silently violated for text-then-tool-call streams

The pre-tool-call flush (Fix #3) calls _round(item, is_final=False) without checking end_of_stream_only. Any stream where text content is followed by a tool-call chunk will trigger a mid-stream guardrail HTTP call regardless of end_of_stream_only=True. For PII masking use-cases where the user sets end_of_stream_only=True specifically to send only complete text to the guardrail, partial text up to the tool-call boundary is sent mid-stream, potentially degrading detection quality. There is no warning and no documentation of this override.

[greptile-apps]

raise _StreamTerminated() is dead code for non-A2A underflow paths

For non-A2A call types, _emit_streaming_http_error re-raises the HTTPException from within its async generator body, so the async for error_item in self._emit_streaming_http_error(...) loop propagates the exception rather than completing normally. The raise _StreamTerminated() on the line after the loop is therefore never reached for non-A2A paths, and the outer except _StreamTerminated: return in _run_incremental_transform_stream also never fires. The stream terminates correctly (via exception propagation), but through an uncontrolled path rather than the designed sentinel path, which can mislead future maintainers about the control flow. This is the mechanism behind the ChunkedEncodingError described in follow-up #9/#15.

[greptile-apps]

Greptile Summary

This PR adds an opt-in streaming_transform_mode: incremental_diff mode to generic_guardrail_api that enables HTTP guardrails to rewrite streaming response text (PII masking, redaction) in real time, rather than only blocking. Raw upstream chunks are withheld, the guardrail is sampled over accumulated text every N chunks, and synthetic delta chunks carrying only the newly-guardrailed portion are emitted to the client.

• Adds StreamTransformSink as a write-once out-parameter so the streaming transform path never mutates the raw accumulator, and threads it through process_output_streaming_response.
• Implements _run_incremental_transform_stream in UnifiedLLMGuardrails with four correctness fixes on top of the OSS PR: mixed-chunk finish_reason deferral, choice-index sort for n>1 streams, cross-chunk text flush before tool-call passthroughs, and a terminator chunk when the guardrail suppresses all text.
• Adds stream_holdback_chars to the guardrail response contract so a guardrail server can request the framework withhold trailing characters between rounds (word-boundary safety).

Confidence Score: 3/5

The new incremental_diff path is well-structured and the four correctness fixes are sound, but a mutation bug in the block-inspection helper can corrupt the raw text accumulator for n>1 mixed content+tool_call streams, producing doubled or mis-transformed output for non-idempotent guardrails.

The core issue is that _inspect_full_response_for_block passes the live responses_so_far list to _process_streaming_block_only. For n>1 chunks where a text choice (choice 0, no finish_reason) and a tool-call choice (choice 1, with finish_reason) appear in the same chunk, the block path's has_stream_ended check reads choices[0].finish_reason=None and falls through to apply in-place text rewrites on the shared accumulator. The existing tests all use an idempotent uppercase guardrail and therefore pass even with the corrupted accumulator, meaning this failure mode has no test coverage today.

litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py — specifically _inspect_full_response_for_block and its interaction with _run_incremental_transform_stream's final _round call.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py	Core streaming transform implementation. Contains a correctness bug where _inspect_full_response_for_block passes the live responses_so_far to _process_streaming_block_only; for n>1 mixed chunks the block-only path can mutate the raw accumulator before the final transform round reads it.
litellm/llms/openai/chat/guardrail_translation/handler.py	Adds _process_streaming_transform and _accumulate_string_content_by_choice_index for the incremental_diff path; correctly avoids mutating responses_so_far in the transform path.
litellm/types/proxy/guardrails/guardrail_hooks/generic_guardrail_api.py	Adds streaming_transform_mode to the config model, stream_holdback_chars to the response/inputs types, and coerce_stream_holdback_value for safe int coercion.
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/generic_guardrail_api.py	Adds streaming_transform_mode init parameter; stream_holdback_chars is now forwarded from the guardrail response to the returned inputs.
litellm/llms/base_llm/guardrail_translation/base_translation.py	Adds StreamTransformSink dataclass and threads it through process_output_streaming_response.
tests/test_litellm/proxy/guardrails/guardrail_hooks/unified_guardrails/test_unified_guardrail.py	Comprehensive integration tests for the incremental_diff path; idempotent test guardrail (uppercasing) masks the n>1 mixed-chunk accumulator corruption bug.
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_generic_guardrail_api.py	New unit tests for stream_holdback_chars parsing and streaming_transform_mode config. All tests are mock-only.
litellm/types/utils.py	Adds stream_holdback_chars field to GenericGuardrailAPIInputs TypedDict.

Comments Outside Diff (1)

1. litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py, line 1036-1056 (link)

   _inspect_full_response_for_block can corrupt the raw accumulator for n>1 mixed-chunk streams

   _inspect_full_response_for_block calls process_output_streaming_response with stream_transform_sink=None, routing through _process_streaming_block_only. That path calls _apply_guardrail_responses_to_output_streaming(responses_so_far, ...) which mutates delta.content in-place on the chunks. The mutation is guarded by has_stream_ended, which is determined by checking only chunk.choices[0].finish_reason.

   For an n>1 chunk that carries text on choice 0 (no finish_reason) and tool_calls on choice 1 (with finish_reason="tool_calls"), choices[0].finish_reason is None → has_stream_ended=False → the block path processes the text AND rewrites it in-place on the shared responses_so_far list.

   The subsequent _round(last_chunk, is_final=True) then calls _accumulate_string_content_by_choice_index(responses_so_far) and reads the already-guardrailed text. For the common uppercasing test guardrail the result is still correct (idempotent), but a non-idempotent guardrail (e.g., "John" → "[REDACTED]") would see "[REDACTED]" on the second pass. Worse, if prior text chunks were also in the accumulator, the block path writes the full guardrailed string into the first text chunk only (the already_set guard), so the accumulator becomes "GUARDRAILED_PREFIX" + the remaining raw text — a concatenation that will either produce duplicate output or trigger a stream_transform_underflow 400.

   Pass list(responses_so_far) as a shallow copy to _inspect_full_response_for_block to keep the raw accumulator untouched by the block inspection.

_{Reviews (2): Last reviewed commit: "fix(guardrails): four correctness fixes ..." | Re-trigger Greptile}

[greptile-apps]

When streaming_transform_mode=incremental_diff takes the early-return branch, buffer_until_moderated is computed but then silently dropped. Unlike the mask_response_content + buffer_until_moderated combination (which emits a warning and disables buffering), this combination gives the user no feedback that their streaming_buffer_until_moderated=True config is being ignored. The PR description acknowledges this as follow-up #12, but even a one-liner warning would prevent silent misconfiguration in production.

Suggested change

	if streaming_transform_mode == "incremental_diff":
	transform_call_type = self._resolve_transform_call_type(
	user_api_key_dict=user_api_key_dict,
	mappings=endpoint_guardrail_translation_mappings,
	)
	if streaming_transform_mode == "incremental_diff":
	if buffer_until_moderated:
	verbose_proxy_logger.warning(
	"UnifiedLLMGuardrails: streaming_buffer_until_moderated is ignored for %s "
	"because streaming_transform_mode=incremental_diff -- the transform path "
	"withholds chunks itself; set streaming_buffer_until_moderated=False to "
	"suppress this warning.",
	getattr(guardrail_to_apply, "guardrail_name", None),
	)
	transform_call_type = self._resolve_transform_call_type(
	user_api_key_dict=user_api_key_dict,
	mappings=endpoint_guardrail_translation_mappings,
	)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

❌ Patch coverage is 89.92537% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...drail_hooks/unified_guardrail/unified_guardrail.py	89.50%	19 Missing ⚠️
.../llms/openai/chat/guardrail_translation/handler.py	87.87%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_guardrail_streaming_text_transform_v2 (f980922) with litellm_internal_staging (3a42011)}

[yucheng-berri]

Update to follow-up list

Greptile scored 4/5 on d112d44c88 — all 4 findings are P2 style/warning items, no new correctness bugs. Three overlap with existing follow-ups (#9/#12/#15); one is new. Adding it to the table:

#	Limitation	Suggested fix
16	streaming_end_of_stream_only=True silently violated by the pre-tool-call text flush (Fix #3 in this PR) — a mid-stream _round(is_final=False) fires when text is followed by a tool-call chunk, regardless of the config	Either (a) skip the pre-tool-call flush when end_of_stream_only=True and defer the transformed text to the EOS flush (accepts that mixed text+tool_call finish_reason ordering will re-break for the end_of_stream_only=True config), or (b) add a runtime warning at config-parse time noting the flag is best-effort under incremental_diff

Rebuttal on the other three P2s (all pre-existing scope):

• Sentinel dead-code path for non-A2A underflow (line 448) — mechanism behind the ChunkedEncodingError already tracked as follow-up Add state handlers / observability #9/Add an example on how to deploy litellm on a render server  #15. Same fix (build_block_sse_chunks override) closes all three.
• Silent bypass of streaming_buffer_until_moderated (line 899) — tracked as follow-up Pushing data to posthog, sentry, slack, etc.  #12. Greptile's suggested 5-line verbose_proxy_logger.warning patch is the intended fix; keeping it out of this PR to preserve the "only fixes 1-4" scope decision.
• Silent drop of buffer_until_moderated in the early-return branch (line 886) — same finding as Pushing data to posthog, sentry, slack, etc.  #12 stated a second way. Same fix.

[yucheng-berri]

@greptileai

[yucheng-berri]

Fifth correctness fix landed

Greptile flagged a P1 in this pass's review (d112d44c88): _inspect_full_response_for_block passed the live responses_so_far list to _process_streaming_block_only, which mutates delta.content in-place on the chunk objects. For n>1 mixed chunks (text on choice 0, tool_calls with finish_reason on choice 1), has_stream_ended reads choices[0] alone and misses the terminal signal, so the block path proceeds to rewrite the shared accumulator. The subsequent _round(is_final=True) then reads the already-guardrailed text — silent double-application for non-idempotent guardrails (e.g. "John" → "[REDACTED]") or stream_transform_underflow from mismatched prefixes.

Fix in f9809229a7: pass copy.deepcopy(responses_so_far) into _inspect_full_response_for_block. The block path's in-place mutation now hits only the copy; the raw accumulator stays untouched for the final _round.

Regression test test_non_idempotent_guardrail_not_double_applied_on_tool_call_streams added — uses a "John" → "[REDACTED]" guardrail (non-idempotent for detection purposes) and asserts every input the guardrail sees is raw text, never a re-guardrailed prefix. Verified: test fails on the unfixed HEAD, passes on f9809229a7.

This PR now fixes 5 correctness bugs from the OSS PR, not 4. Updated summary:

#	Fix	Bug class	Bot
1	Mixed content+tool_call finish_reason deferred to terminator	Data loss (SSE clients drop redacted text)	Greptile P1 x2, Veria
2	Choice-index sort in _process_streaming_transform	Data corruption (wrong text to wrong choice)	Bugbot Medium
3	Cross-chunk pre-tool-call text flush	Data loss (same class as #1, cross-chunk)	Greptile P1
4	Terminator on empty mutated_text_per_choice	Data loss (deferred finish_reason never delivered)	Bugbot High
5	Deep-copy responses_so_far before end-of-stream block inspection	Data corruption (non-idempotent guardrails re-applied to their own output)	Greptile P1 (this pass)

Follow-up list unchanged (16 items). Local tests: 42/42 unified + 66/66 generic pass. All still scoped to incremental_diff — no changes to block_only, non-streaming, pre_call, or sibling guardrails.

[yucheng-berri]

@greptileai

[yucheng-berri]

@greptileai please rescore on HEAD f980922 — a 5th correctness fix was pushed after your last review (copy.deepcopy(responses_so_far) before end-of-stream block inspection, addressing your P1 on _inspect_full_response_for_block shared-accumulator mutation)

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f980922. Configure here.}

[cursor]

Holdback text after tool finish

High Severity

When _round is called for a pre-tool-call text flush, is_final=False allows stream_holdback_chars to withhold trailing text. If a subsequent tool-call passthrough emits finish_reason, SSE clients stop reading, causing this held-back text to be dropped.

 

^{Reviewed by Cursor Bugbot for commit f980922. Configure here.}

[yucheng-berri]

Legit. Real bug: pre-tool-call _round(is_final=False) honors stream_holdback_chars, held-back text lost when passthrough's finish_reason terminates client read. Added as follow-up #18 in the PR body — deferred to keep this PR strictly to fixes 1-5. Not blocking merge because holdback under incremental_diff is only active when the guardrail server explicitly requests it via stream_holdback_chars; default configs are unaffected.

[cursor]

Tool block inspect can miss

High Severity

The _inspect_full_response_for_block method, intended to ensure guardrails inspect all tool calls, incorrectly relies on choices[0].finish_reason to determine stream completion. This means for n>1 responses where finish_reason or tool_calls are on a later choice, the guardrail's tool-call blocking logic is bypassed.

 

^{Reviewed by Cursor Bugbot for commit f980922. Configure here.}

[yucheng-berri]

Pre-existing block_only limitation (has_stream_ended inspects choices[0] alone). Already tracked as follow-up #13 in the PR body. Not introduced by this PR — same limitation existed under block_only for OpenAI chat streams; this PR inherits it because _inspect_full_response_for_block routes through the block_only path for the tool-call block check. Fix belongs in a separate PR that touches _process_streaming_block_only's has_stream_ended check directly (trivial 1-liner: any(c.finish_reason is not None for c in chunk.choices)).

[cursor]

Two tests merged into one

Low Severity

The test_non_idempotent_guardrail_not_double_applied_on_tool_call_streams method inadvertently combines two distinct test scenarios. The second, verifying raw-accumulator behavior, is embedded after an orphaned docstring, obscuring its intent and potentially masking failures. This weakens regression coverage, as the first scenario lacks client-visible output assertions and the second isn't a distinct test.

 

^{Reviewed by Cursor Bugbot for commit f980922. Configure here.}

[yucheng-berri]

Confirmed by my scope-audit agent as well — the pre-existing test_guardrail_always_sees_raw_accumulated_text was renamed and its body left as tail-statements in the new non-idempotent test. Assertions still execute (no runtime coverage loss) but the named test node is gone. Test-hygiene issue, no correctness impact. Will fold into a follow-up cleanup along with #16-#18.

[cursor]

Pre-flush misses first mixed text

Medium Severity

The pre-tool-call text flush only runs when saw_text_content is already true from an earlier text-only chunk. Text that first appears on a tool-classified chunk (including n>1 text on one choice and tool_calls with finish_reason on another) skips that flush, so the passthrough can emit finish_reason before any synthetic transformed text. SSE clients that stop at finish_reason then drop the guardrailed content Fix #3 aimed to preserve.

 

^{Reviewed by Cursor Bugbot for commit f980922. Configure here.}

[yucheng-berri]

Legit new edge case (n>1 same-chunk, text on choice 0 + tool_call+finish_reason on choice 1). Added as follow-up #17 in the PR body — deferred to a v1.1 patch to keep this PR strictly to fixes 1-5. Not blocking merge because activation requires n>1 + specific chunk shape + non-idempotent guardrail.

[mateo-berri]

LGTM; thanks!

For future reference, this was tested e2e here: #32915 (comment)