feat(guardrails): expose streaming knobs on generic_guardrail_api

[schneidermr]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Unit coverage is in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_generic_guardrail_api.py for streaming defaults, optional_params plumbing via initialize_guardrail (including top-level knobs not shadowed by sibling optional_params defaults), sampled vs end-of-stream-only cadence through UnifiedLLMGuardrails.async_post_call_streaming_iterator_hook, mid-stream BLOCKED, fail_open on unreachable, /v1/responses streaming (end-of-stream-only + BLOCKED), and streaming_sampling_rate rejection of non-positive values

PR-scoped local run: streaming config + unified classes -> 18 passed on the latest commit. Greptile P1 (optional_params non-None defaults shadowing top-level streaming flags) addressed earlier. Sameerlite review nits addressed: streaming_sampling_rate validated >= 1 (constructor + Pydantic ge=1), /v1/responses streaming tests added; _get_config_value left local for now (same pattern as openai/grayswan/pillar)

For a live proxy smoke test (run the proxy with your usual dev_config.yaml and a generic_guardrail_api guardrail configured on post_call), exercise both modes:

# Default: mid-stream sampling (streaming_end_of_stream_only=false, sampling_rate=5)
curl -sS http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a short paragraph about the sky"}]
  }'

# End-of-stream-only: set streaming_end_of_stream_only=true on the guardrail config, reload, then the same curl
# A BLOCKED response from your guardrail API should surface as GuardrailRaisedException after the stream is assembled

Type

🆕 New Feature

Changes

Adds first-class streaming configuration to generic_guardrail_api so it can participate in the existing UnifiedLLMGuardrails post-call streaming path with the same knobs other guardrails already honor via getattr(guardrail_to_apply, "streaming_*", default)

streaming_end_of_stream_only (default false when unset) controls whether the guardrail runs incrementally on sampled chunks (an in-flight BLOCKED stops further chunks) or once at end of stream over the assembled response (cheaper/faster, but flagged content may already have reached the client). streaming_sampling_rate (default 5 when unset) sets the every-Nth-chunk cadence when incremental mode is on; ignored when end-of-stream-only is true

Plumbing is through GenericGuardrailAPIOptionalParams / GenericGuardrailAPIConfigModel (UI/config surface via get_config_model()), constructor attributes on GenericGuardrailAPI, and initialize_guardrail which reads either top-level litellm_params or nested optional_params so both config styles work. Optional-params fields default to None so unset nested values do not shadow top-level streaming flags; real defaults are applied in the constructor

Tests cover defaults/overrides, config model exposure, initialize_guardrail forwarding, mixed-config priority (top-level vs explicit optional_params), safe streaming yield, mid-stream BLOCKED, sampling cadence (sampled + final aggregate), fail_open continuing the stream when the guardrail API is unreachable, non-positive streaming_sampling_rate rejection, and /v1/responses streaming through the unified hook (end-of-stream-only and BLOCKED)

---

Note

Medium Risk
Changes guardrail behavior on streaming responses (when checks run and whether bad content may already reach the client); misconfiguration could weaken policy enforcement or increase guardrail API load.

Overview
Adds streaming post-call guardrail settings to generic_guardrail_api so it plugs into the existing UnifiedLLMGuardrails streaming hook via streaming_end_of_stream_only and streaming_sampling_rate on the guardrail instance.

streaming_end_of_stream_only (default false) chooses between sampled in-stream checks (mid-stream BLOCKED can stop the stream) versus a single check on the assembled response at end of stream. streaming_sampling_rate (default 5, must be ≥ 1) sets every-Nth-chunk sampling when incremental mode is on.

Config is exposed on GenericGuardrailAPIOptionalParams / GenericGuardrailAPIConfigModel, wired through initialize_guardrail with _get_config_value so nested optional_params (model or dict) wins when set but None defaults do not override top-level litellm_params. GenericGuardrailAPI.get_config_model() is added for the UI/config surface.

Tests cover init defaults, config resolution, unified streaming behavior (cadence, block, fail-open, /v1/responses), and validation of non-positive sampling rates.

^{Reviewed by Cursor Bugbot for commit 2b464fd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds first-class streaming configuration to GenericGuardrailAPI so it participates in the existing UnifiedLLMGuardrails post-call streaming path using the same getattr-based knob protocol as other guardrails. The previous reviewer's concern (non-None defaults causing optional_params to shadow top-level litellm_params) is correctly resolved — both new fields in GenericGuardrailAPIOptionalParams default to None, with real defaults applied only in the constructor.

• Adds streaming_end_of_stream_only (default False) and streaming_sampling_rate (default 5, must be ≥ 1) as constructor params on GenericGuardrailAPI, with a ValueError guard on invalid rates and a Pydantic ge=1 constraint in the type model.
• Introduces _get_config_value in initialize_guardrail to correctly prefer an explicit optional_params override over the top-level litellm_params value, falling through only when the nested value is None.
• All 18+ new tests are mock-only (no real network calls), covering defaults, overrides, sampling cadence, mid-stream BLOCKED, fail-open on unreachable endpoints, and /v1/responses streaming.

Confidence Score: 5/5

Safe to merge — the change is additive, all new fields are opt-in with backwards-compatible defaults, and no existing behaviour is altered.

The two new streaming knobs are wired cleanly through every layer (Pydantic model → constructor → initialize_guardrail), the None-sentinel pattern correctly prevents unset optional_params from shadowing top-level config, and the new tests verify every integration point with mocks only.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/init.py	Adds _get_config_value helper for priority resolution between top-level litellm_params and nested optional_params, and wires the two new streaming knobs into initialize_guardrail
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/generic_guardrail_api.py	Adds streaming_end_of_stream_only and streaming_sampling_rate constructor params with correct None-sentinel defaults; validates sampling rate >= 1; exposes get_config_model() static method
litellm/types/proxy/guardrails/guardrail_hooks/generic_guardrail_api.py	Adds streaming_end_of_stream_only and streaming_sampling_rate fields to GenericGuardrailAPIOptionalParams with default=None (correct sentinel pattern) and Pydantic ge=1 constraint on the rate field
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_generic_guardrail_api.py	Adds 718 lines of new mock-only tests covering streaming defaults, overrides, cadence sampling, mid-stream BLOCKED, fail-open, config model exposure, initialize_guardrail plumbing for both dict and model optional_params, and /v1/responses streaming

_{Reviews (3): Last reviewed commit: "fix(guardrails): read nested streaming c..." | Re-trigger Greptile}

[CLAassistant]

All committers have signed the CLA.

[schneidermr]

@greptileai

[Muhtasim-Munif-Fahim]

Clean feature addition. \streaming_end_of_stream_only\ and \streaming_sampling_rate\ knobs give users meaningful control over guardrail cost/latency tradeoffs in streaming mode. The _get_config_value\ priority logic (optional_params wins over litellm_params) is correct and well-documented. Thorough test coverage across defaults, overrides, sampling cadence, and fail-open scenarios. LGTM.

[Sameerlite]

Minor nits :
Duplicated _get_config_value — Same helper exists in openai/init.py. Fine for now; could be shared later.

No /v1/responses streaming test — All integration tests use /chat/completions. Acceptable since the unified hook is route-agnostic, but a responses-streaming test would match real-world usage (e.g. your gpt-5.5 curl).

No validation on streaming_sampling_rate — 0 or negative values aren’t guarded. Same gap as other guardrails; only worth fixing if you want parity with stricter validators elsewhere.

[schneidermr]

Thanks for the review @Sameerlite, good catches.

Pushed a follow-up that covers the actionable nits:

• streaming_sampling_rate is now validated as >= 1 both in GenericGuardrailAPI.__init__ (raises ValueError for 0/negative) and on the Pydantic optional params field (ge=1), with regression tests for the invalid values
• Added /v1/responses streaming coverage through UnifiedLLMGuardrails.async_post_call_streaming_iterator_hook (end-of-stream-only single call + mid-stream BLOCKED raises), so the Responses path is exercised alongside the chat completions tests

On the duplicated _get_config_value: left local for now, same pattern as openai/grayswan/pillar. Happy to extract to a shared helper in a follow-up if you want that cleaned up across guardrails in one pass

Mind taking another look when you get a chance?

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[Sameerlite]

@schneidermr Please check veria comment

[schneidermr]

@Sameerlite thanks for the ping, and thanks to Veria for catching this

Fixed the nested streaming config issue: when configs come through the guardrail API/UI, optional_params arrives as a plain dict, and getattr was silently skipping streaming_sampling_rate / streaming_end_of_stream_only. _get_config_value now handles both dict and model shapes, so nested streaming knobs are applied correctly at runtime

Also added regression tests for the dict path (explicit nested values win, and sibling-only dict keys still fall through to top-level knobs). All 52 tests in the generic guardrail suite pass

Mind taking another look when you get a chance?

[Sameerlite]

Thanks for your patience on this one! Your changes look solid — the Greptile review is just stale because of commits since the last review. Kicking off a fresh pass now.

@greptileai

[krrish-berri-2]

• @yucheng-berri

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2b464fd. Configure here.}

[yucheng-berri]

merged with #31730