feat(guardrails): expose streaming knobs on generic_guardrail_api

[yucheng-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Live proxy smoke test (run the proxy with your usual dev_config.yaml and a generic_guardrail_api guardrail configured on post_call), exercise both modes:

# Default: mid-stream sampling (streaming_end_of_stream_only=false, sampling_rate=5)
curl -sS http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a short paragraph about the sky"}]
  }'

# End-of-stream-only: set streaming_end_of_stream_only=true on the guardrail config, reload, then the same curl
# A BLOCKED response from your guardrail API should surface as GuardrailRaisedException after the stream is assembled

Type

New Feature

Changes

Adds first-class streaming configuration to generic_guardrail_api so it can participate in the existing UnifiedLLMGuardrails post-call streaming path with the same knobs other guardrails already honor via getattr(guardrail_to_apply, "streaming_*", default)

streaming_end_of_stream_only (default false when unset) controls whether the guardrail runs incrementally on sampled chunks (an in-flight BLOCKED stops further chunks) or once at end of stream over the assembled response (cheaper/faster, but flagged content may already have reached the client). streaming_sampling_rate (default 5 when unset) sets the every-Nth-chunk cadence when incremental mode is on; ignored when end-of-stream-only is true

Plumbing is through GenericGuardrailAPIOptionalParams / GenericGuardrailAPIConfigModel (UI/config surface via get_config_model()), constructor attributes on GenericGuardrailAPI, and initialize_guardrail which reads either top-level litellm_params or nested optional_params so both config styles work. Optional-params fields default to None so unset nested values do not shadow top-level streaming flags; real defaults are applied in the constructor

Tests cover defaults/overrides, config model exposure, initialize_guardrail forwarding, mixed-config priority (top-level vs explicit optional_params for both model and dict shapes), safe streaming yield, mid-stream BLOCKED, sampling cadence (sampled + final aggregate), fail_open continuing the stream when the guardrail API is unreachable, non-positive streaming_sampling_rate rejection, and /v1/responses streaming through the unified hook (end-of-stream-only and BLOCKED)

Credit

Adopted from #30924 by @schneidermr (Marton Schneider). Mirrored onto BerriAI/litellm's litellm_generic_guardrail_streaming_config so CircleCI and the internal lint workflow trigger; original commits and authorship preserved

---

Note

Medium Risk
Changes post-call streaming safety behavior for generic guardrail users; end-of-stream-only mode can allow flagged content to reach clients before a block, which is an intentional tradeoff but security-relevant.

Overview
Adds streaming post-call guardrail settings to generic_guardrail_api so it can use the same UnifiedLLMGuardrails streaming path as other integrations.

streaming_end_of_stream_only (default false) chooses between sampled in-stream checks (a mid-stream BLOCKED can stop the stream) versus a single check on the assembled response at end of stream. streaming_sampling_rate (default 5) sets every-Nth-chunk cadence when incremental mode is on.

Config is exposed on GenericGuardrailAPIOptionalParams, wired through initialize_guardrail via _get_config_value (optional_params dict or model wins when set; unset nested None does not override top-level litellm_params), stored on GenericGuardrailAPI, and surfaced for UI via get_config_model(). Constructor applies defaults and rejects streaming_sampling_rate < 1.

Tests cover init/plumbing priority, unified-hook behavior for chat completions and /v1/responses (cadence, BLOCKED, end-of-stream-only, fail_open on unreachable).

^{Reviewed by Cursor Bugbot for commit 2b464fd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_generic_guardrail_streaming_config (2b464fd) with litellm_internal_staging (6ab3742)}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2b464fd. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds first-class streaming configuration (streaming_end_of_stream_only, streaming_sampling_rate) to GenericGuardrailAPI so it can participate in the UnifiedLLMGuardrails post-call streaming path on equal footing with other guardrail integrations.

• Types & config model: GenericGuardrailAPIOptionalParams gains both fields defaulting to None, with a Pydantic ge=1 constraint on the rate; GenericGuardrailAPIConfigModel exposes them to the UI via get_config_model().
• Constructor & plumbing: GenericGuardrailAPI.__init__ stores streaming_end_of_stream_only (default False) and streaming_sampling_rate (default 5), rejecting < 1 rates; initialize_guardrail reads both fields through the _get_config_value helper that correctly gives nested optional_params priority without letting None defaults shadow top-level litellm_params.
• Backward compatibility: The new defaults match exactly what UnifiedLLMGuardrails already applied via getattr(guardrail, "streaming_*", default), so existing deployments see no behavioral change.

Confidence Score: 5/5

Safe to merge; the change is additive and the new defaults are identical to what the unified guardrail hook already applied via getattr fallbacks.

The implementation is logically sound across all three layers (types, constructor, plumbing helper). Defaults are proven backward-compatible with the UnifiedLLMGuardrails source. Validation is correctly duplicated at the Pydantic and constructor levels. Test coverage is comprehensive, touching every config permutation, both streaming modes, BLOCKED escalation, fail-open, and both API endpoints.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/init.py	Adds _get_config_value helper and wires streaming_end_of_stream_only / streaming_sampling_rate through initialize_guardrail; also adds a return of the created callback (new, but backward-compatible). Logic is correct and defaults match what UnifiedLLMGuardrails already applied via getattr.
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/generic_guardrail_api.py	Adds streaming_end_of_stream_only and streaming_sampling_rate constructor parameters with validation (raises ValueError for < 1), stores them as typed instance attributes, and adds get_config_model() static method. Implementation is clean and consistent with other guardrail integrations.
litellm/types/proxy/guardrails/guardrail_hooks/generic_guardrail_api.py	Adds streaming_end_of_stream_only: Optional[bool] and streaming_sampling_rate: Optional[int] (with ge=1) to GenericGuardrailAPIOptionalParams. Both default to None to avoid shadowing top-level litellm_params values. Correct use of Pydantic Field constraints.
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_generic_guardrail_api.py	Adds ~700 lines of new tests covering defaults, overrides, invalid rate rejection, get_config_model, initialize_guardrail plumbing with all config shapes (model/dict/mixed priority), and end-to-end unified-hook behavior for both /chat/completions and /v1/responses. Existing test changes remove unused dead variables only.

_{Reviews (3): Last reviewed commit: "style(guardrails): ruff format generic_g..." | Re-trigger Greptile}

[greptile-apps]

Avoid broad Any
_get_config_value adds broad Any annotations for untyped config shapes, which increases the static typing budget this repo tracks. The project guidance prefers object for broad parameters because it still accepts None without adding Any; this helper can keep the same runtime behavior by typing the inputs and return as object.

Rule Used: In this repo, prefer object over Optional[Any]... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Greptile Summary

This PR exposes streaming configuration for generic_guardrail_api. The main changes are:

• Added streaming_end_of_stream_only and streaming_sampling_rate to the guardrail config model.
• Forwarded streaming settings through initialize_guardrail from top-level params or nested optional_params.
• Added constructor defaults and validation on GenericGuardrailAPI.
• Added mocked tests for streaming defaults, overrides, fail-open behavior, blocked streams, sampling cadence, and /v1/responses streaming.

Confidence Score: 4/5

The implementation is narrowly scoped and well covered by tests for the new streaming configuration paths.

The remaining issue is a typing cleanup in the helper surface rather than a functional streaming behavior concern.

litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/init.py

T-Rex Logs

What T-Rex did

• Inspected the pre-change test log and observed base exit code 1 with an AssertionError: optional params exposes streaming_end_of_stream_only.
• Inspected the post-change test log and observed head exit code 0 with all tests PASS, including defaults False/5, override forwarding, nested priority, nested None no-shadow behavior, and ValueError/ValidationError rejection for streaming_sampling_rate <= 0.
• Compared pre-change targeted checks and observed end_only_calls=3 (expected 1), sampling_2_calls=2 (expected 4), and responses_events_calls=(6, 2) (expected (6, 1)); other safety paths were exercised.
• Validated post-change targeted checks and observed default_sample_calls=3, end_only_calls=1, sampling_2_calls=4, fail_open_chunks=3, blocked_exception=GuardrailRaisedException, responses_events_calls=(6,1), EXIT_CODE=0.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "style(guardrails): ruff format generic_g..." | Re-trigger Greptile}

[greptile-apps]

Use object typing
This helper introduces broad Any annotations for config values, which adds to the static type checker’s Any budget. Since these parameters can be arbitrary objects or dicts and None is still accepted by object, typing them as object preserves the current runtime behavior while keeping the proxy module aligned with the repo’s typing guidance.

Rule Used: In this repo, prefer object over Optional[Any]... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[yucheng-berri]

@greptileai I think this finding is out of scope for this PR.

The mutation pattern described here is in UnifiedLLMGuardrails.async_post_call_streaming_iterator_hook / process_output_streaming_response, not in generic_guardrail_api. This PR does not modify unified_guardrail.py or the responses_so_far accumulation logic. If responses_so_far is being mutated in-place and corrupting later sampled scans, that would be a pre-existing unified streaming-hook issue affecting all sampling guardrails, not a regression introduced by this PR.

Also, I do not think the reported captured output comes from the new test. test_streaming_sampling_rate_override only asserts mock_post.await_count == 4, and the mock returns a fixed texts=["ABCDEF"]. It does not capture or assert mock_post.await_args_list[*].kwargs["json"]["texts"], and there is no "ABCDEFCD" value produced by the test.

I'm happy to file a follow-up against UnifiedLLMGuardrails to audit whether process_output_streaming_response should receive a deep-copied snapshot of responses_so_far for incremental scans, but I don't think this should block #30924.

[greptile-apps]

You're right on both counts, and I appreciate the detailed walkthrough.

1. Mutation concern is pre-existing in UnifiedLLMGuardrails — this PR doesn't touch unified_guardrail.py at all, so any responses_so_far accumulation issue is orthogonal to this change and would affect all sampling guardrails equally. A follow-up audit there makes sense.

2. The "ABCDEFCD" output claim was incorrect — looking at test_streaming_sampling_rate_override, it only asserts mock_post.await_count == 4 and the mock returns a fixed texts=["ABCDEF"] regardless of input. There's no assertion on the payload texts field in any call, and nothing in the test would produce a "ABCDEFCD" value. That was an error on my part.

The finding is withdrawn. The implementation and test coverage look correct for the scope of this PR.

[yucheng-berri]

@greptileai

[yucheng-berri]

Adopted from #30924 by @schneidermr — wires the existing streaming_sampling_rate / streaming_end_of_stream_only contract (already honored by UnifiedLLMGuardrails and exposed on grayswan / openai_moderation since #17400 and #27324) into generic_guardrail_api, opt-in with backwards-compatible defaults