fix(openai-moderation): wire streaming flags through to unified dispatcher

[michelligabriele]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Backend-only change — proof is in the new tests under tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/:

• test_openai_moderation_streaming_default_calls_moderation_once — asserts the OpenAI /moderations endpoint is awaited exactly once at end-of-stream across 10 streamed chunks under the default configuration.
• test_openai_moderation_streaming_sampled_when_end_of_stream_only_disabled — with streaming_end_of_stream_only=False, streaming_sampling_rate=2 over 6 chunks, asserts 4 calls (sampled at chunks 2 / 4 / 6 plus the final aggregate pass).
• test_openai_moderation_guardrail_streaming_defaults / ..._streaming_overrides / ..._initialize_guardrail_forwards_streaming_flags — verify the wire-through from YAML → LitellmParams → constructor → instance attribute end-to-end.

Type

🐛 Bug Fix

Changes

Before

OpenAIModerationGuardrail configured with mode: post_call over a streaming response calls the OpenAI /moderations endpoint every 5th streamed chunk. The unified streaming dispatcher at litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py reads streaming_end_of_stream_only and streaming_sampling_rate via getattr(...) on the configured guardrail instance, but neither attribute was declared on OpenAIModerationGuardrail, its config model, or its initializer. The flags were therefore unreachable from YAML for this integration — every deployment got the dispatcher's sampled default regardless of intent, which contradicts the publicly documented end-of-stream-only behavior for OpenAI Moderation.

This regressed in #20718, which introduced the unified streaming dispatcher's sampled mode and made it the implicit default for guardrails that did not opt into end-of-stream-only via the new flags. Integrations that ship their own wire-through (e.g. GraySwan, see litellm/proxy/guardrails/guardrail_hooks/grayswan/__init__.py) opted in; OpenAI Moderation did not.

After

OpenAIModerationGuardrailConfigModel declares streaming_end_of_stream_only (default True) and streaming_sampling_rate (default 5), so both knobs validate in YAML and surface in the auto-generated UI form. OpenAIModerationGuardrail.__init__ accepts both flags and stores them on self so the dispatcher's getattr reads return real values. initialize_guardrail forwards them from litellm_params and optional_params via a small _get_config_value helper, mirroring the existing GraySwan wire-through.

Default behavior now matches the public documentation: one /moderations call at end of stream over the assembled response. Deployments that prefer the post-#20718 cadence (lower time-to-first-token, multiple /moderations calls during the stream) keep that as a one-line YAML opt-out:

guardrails:
  - guardrail_name: openai-moderation
    litellm_params:
      guardrail: openai_moderation
      mode: post_call
      streaming_end_of_stream_only: false
      streaming_sampling_rate: 2

If the team would rather preserve the post-#20718 cadence as the default (and update the public docs to match) instead of restoring the pre-#20718 default, the change is a one-line literal flip in OpenAIModerationGuardrailConfigModel and OpenAIModerationGuardrail.__init__ — flagging here so you can pick the direction on review.

[greptile-apps]

Greptile Summary

This PR fixes a regression introduced in #20718 where OpenAIModerationGuardrail lacked the streaming_end_of_stream_only and streaming_sampling_rate instance attributes expected by the unified streaming dispatcher, causing all deployments to silently use the dispatcher's sampled default (every 5th chunk) regardless of YAML config.

• Adds both flags to OpenAIModerationGuardrailConfigModel, the __init__ signature, and stores them on self — the dispatcher's getattr calls now read real values.
• Adds a _get_config_value helper in initialize_guardrail (mirroring GraySwan) to forward the flags from litellm_params/optional_params into the constructor.
• Updates three existing tests whose metadata.guardrail_config was dead code (the dispatcher reads from the guardrail instance, not request metadata) and adds five new mock-only tests covering defaults, overrides, wire-through, and call-count behaviour in both modes.

Confidence Score: 4/5

Safe to merge after confirming the team's intended direction on the default behavior change.

The implementation is correct and well-tested. The one discussion point is that every existing openai_moderation streaming deployment will silently move from every-5th-chunk sampling to end-of-stream-only on the next deploy — the PR justifies this as restoring documented pre-#20718 behavior, but it is a runtime behavior change for anyone who adapted to the post-#20718 cadence. An opt-out flag is provided. No data-loss or security issues were identified.

No files require special attention beyond confirming the intended default direction discussed in the PR description.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/openai/init.py	Adds _get_config_value helper and wires streaming_end_of_stream_only/streaming_sampling_rate from litellm_params through initialize_guardrail. Logic is correct; optional_params takes priority, fallback is litellm_params, and None triggers constructor defaults.
litellm/proxy/guardrails/guardrail_hooks/openai/moderations.py	Adds two new optional constructor params (streaming_end_of_stream_only, streaming_sampling_rate) and stores them on self with sensible defaults (True and 5). The unified dispatcher's getattr(guardrail, "streaming_*", default) will now read real values from these attributes.
litellm/types/proxy/guardrails/guardrail_hooks/openai/openai_moderation.py	Adds streaming_end_of_stream_only (default True) and streaming_sampling_rate (default 5) to the Pydantic config model, enabling YAML validation and UI form generation for these knobs.
tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_openai_moderation_streaming.py	Removes now-dead guardrail_config: {"streaming_sampling_rate": 1} metadata from three tests (the dispatcher reads from the guardrail instance, not request metadata) and adds two new integration tests verifying call-count behaviour under the two modes.
tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py	Adds three targeted unit tests for default attributes, constructor overrides, and end-to-end YAML wire-through via initialize_guardrail. All use mocks and env-var patches; no real network calls.

_{Reviews (1): Last reviewed commit: "fix(openai-moderation): wire streaming f..." | Re-trigger Greptile}

[greptile-apps]

Silent default-flip for existing deployments

Before this PR the unified dispatcher fell back to its own local default of end_of_stream_only = False (line 314 of unified_guardrail.py) when streaming_end_of_stream_only was absent on the guardrail instance. After this PR, every existing openai_moderation deployment in post_call streaming mode will silently switch from every-5th-chunk sampling to end-of-stream-only on the next deploy — with no config change required by the operator.

The PR description acknowledges this trade-off and claims it restores pre-#20718 documented behavior, which is a reasonable position. The rule here is satisfied because an opt-out flag is provided. Worth confirming with the team that this is the intended direction before merging, since deployments that accepted the post-#20718 latency trade-off in exchange for faster flagging will be silently reverted.

[codecov]

Codecov Report

❌ Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roxy/guardrails/guardrail_hooks/openai/__init__.py	57.14%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Sameerlite]

LGTM, thanks!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10