fix: openai moderation guardrails

[Harshit28j]

Relevant issues

Fixes latency issues with OpenAI Moderation guardrail during streaming.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:
• CI run for the last commit
  Link:
• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix
✅ Test
🧹 Refactoring

Changes

• Enabled True Streaming for OpenAI Moderation: Refactored OpenAIModerationGuardrail to utilize UnifiedLLMGuardrails for streaming. This allows moderation checks to happen on sampled chunks in parallel with the stream, rather than buffering the entire response.
• Removed Buffering Hook: Deleted async_post_call_streaming_iterator_hook from moderations.py, which was responsible for delaying the stream until completion.
• Added Streaming Tests: Created tests/test_openai_moderation_streaming.py to verify low latency (time-to-first-token < 0.2s) and correct detection of harmful content during streaming.
• Updated Existing Tests: Modified tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py to correctly mock litellm.ModelResponse structures for the new unified guardrail flow.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 13, 2026 5:00am

[greptile-apps]

Greptile Overview

Greptile Summary

This change removes the OpenAI moderation guardrail’s old streaming hook that buffered the entire response before moderating, and instead relies on the UnifiedLLMGuardrails streaming hook to yield chunks immediately while running moderation checks on sampled content in parallel. Tests were updated to exercise the unified streaming path, and a new test module was added to validate time-to-first-token behavior and that harmful content is blocked during streaming.

Key issues to address before merge are in the test suite: one streaming test function name is duplicated (so half the intended coverage is silently skipped), and the new latency test uses wall-clock thresholds/sleeps that can be flaky and can also crash if no chunks are yielded.

Confidence Score: 3/5

• This PR is moderately safe to merge once the test issues are fixed.
• Core functional change is a straightforward removal of the buffering streaming hook in favor of the unified guardrail path, but the PR currently introduces definite test problems: a duplicate pytest test name that suppresses coverage, and a wall-clock latency test that is prone to nondeterministic failures (and can error if no chunks are yielded).
• tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py, tests/test_openai_moderation_streaming.py

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/openai/moderations.py	Removes the buffering streaming hook and routes moderation through the unified apply_guardrail path; code looks consistent, but runtime behavior depends on UnifiedLLMGuardrails sampling/stream assembly.
tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py	Updates moderation tests to use UnifiedLLMGuardrails streaming hook; introduces a duplicate test function name that prevents one test from running.
tests/test_openai_moderation_streaming.py	Adds new streaming latency/harmful-content tests; current latency assertion is wall-clock based and can be flaky, and the test can crash if no chunks are yielded.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Proxy as LiteLLM Proxy
    participant Unified as UnifiedLLMGuardrails
    participant Model as LLM Streaming Response
    participant Builder as stream_chunk_builder
    participant Mod as OpenAIModerationGuardrail
    participant OpenAI as OpenAI Moderations API

    Client->>Proxy: /chat/completions (stream=true)
    Proxy->>Model: initiate upstream stream
    Model-->>Proxy: stream chunks (delta.content)
    Proxy->>Unified: async_post_call_streaming_iterator_hook(response iterator, request_data)

    loop For each streamed chunk
        Unified-->>Client: yield chunk immediately (no buffering)
        Unified->>Unified: sample chunk(s) per streaming_sampling_rate
    end

    Note over Unified,Builder: When sampling triggers or stream ends
    Unified->>Builder: assemble sampled chunks into ModelResponse
    Builder-->>Unified: ModelResponse
    Unified->>Mod: apply_guardrail(inputs from assembled content, input_type="response")
    Mod->>OpenAI: POST /moderations {model,input}
    OpenAI-->>Mod: moderation result

    alt flagged
        Mod-->>Unified: raise HTTPException(400)
        Unified-->>Proxy: propagate error
        Proxy-->>Client: 400 Violated moderation policy
    else not flagged
        Mod-->>Unified: ok
    end

Loading

[greptile-apps]

_{3 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

[ghost]

@greptile please re-review this

[greptile-apps]

Greptile Overview

Greptile Summary

Refactors the OpenAI Moderation guardrail to delegate streaming to UnifiedLLMGuardrails instead of buffering the entire response before moderating, which fixes latency issues during streaming (time-to-first-token).

• Removed async_post_call_streaming_iterator_hook and _extract_response_text from OpenAIModerationGuardrail, delegating streaming to UnifiedLLMGuardrails which applies moderation on sampled chunks in parallel with the stream
• Updated existing streaming tests and added new deterministic streaming tests
• Critical bug: log_guardrail_information import was removed but the decorator is still used on apply_guardrail — this will cause a NameError at runtime

Confidence Score: 2/5

• This PR has a broken import that will cause a runtime NameError on the primary apply_guardrail method, blocking all moderation checks.
• The core logic change (delegating streaming to UnifiedLLMGuardrails) is sound, but the log_guardrail_information import was accidentally removed while the decorator is still in use on line 177. This will crash apply_guardrail at runtime, which is the primary entry point for moderation checks.
• litellm/proxy/guardrails/guardrail_hooks/openai/moderations.py — missing import for log_guardrail_information decorator

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/openai/moderations.py	Removed log_guardrail_information import but it's still used as a decorator on apply_guardrail — will cause NameError at runtime. Also removed buffering-based streaming hook in favor of UnifiedLLMGuardrails delegation.
tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py	Updated streaming tests to route through UnifiedLLMGuardrails instead of the removed direct streaming hook. Tests use proper mocks. Formatting cleanup throughout.
tests/test_openai_moderation_streaming.py	New streaming test file with deterministic assertions. Unused asyncio import. Tests verify chunk passthrough and harmful content detection via UnifiedLLMGuardrails.

_{Last reviewed commit: 5b732e3}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[Harshit28j]

@greptile re-review this PR please thanks
give suggestions for improving score if it sounds relevant, I'll update

[greptile-apps]

Greptile Overview

Greptile Summary

Refactored OpenAIModerationGuardrail to delegate streaming to UnifiedLLMGuardrails instead of buffering the entire response before running moderation. This removes the latency bottleneck where the old async_post_call_streaming_iterator_hook collected all chunks before yielding any, and instead allows chunks to be yielded to the client immediately while moderation runs on sampled chunks in parallel.

• Removed async_post_call_streaming_iterator_hook and _extract_response_text from OpenAIModerationGuardrail, relying on UnifiedLLMGuardrails for streaming orchestration
• Updated streaming tests to use UnifiedLLMGuardrails.async_post_call_streaming_iterator_hook with proper mocking of stream_chunk_builder
• New streaming test file (tests/test_openai_moderation_streaming.py) should be moved to tests/test_litellm/ per contribution guidelines

Confidence Score: 4/5

• This PR is safe to merge with minor issues; the core refactor is clean and well-tested.
• The main change (removing the buffering hook and delegating to UnifiedLLMGuardrails) is straightforward and reduces code. Tests properly mock all external calls. The previously reported missing log_guardrail_information import has been fixed. The only concern is the new test file location outside tests/test_litellm/.
• tests/test_openai_moderation_streaming.py - should be moved to tests/test_litellm/ directory structure

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/openai/moderations.py	Removed the buffering async_post_call_streaming_iterator_hook and _extract_response_text methods, delegating streaming to UnifiedLLMGuardrails. Cleaned up unused imports. The log_guardrail_information import (previously reported as missing) has been restored.
tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_moderations.py	Updated streaming tests to use UnifiedLLMGuardrails instead of the removed async_post_call_streaming_iterator_hook. Properly mocks stream_chunk_builder and uses real litellm types for isinstance checks. Minor formatting cleanup.
tests/test_openai_moderation_streaming.py	New test file for streaming latency and harmful content detection via UnifiedLLMGuardrails. Properly mocked, but placed outside tests/test_litellm/ directory (violates PR contribution requirements). Has unused asyncio import.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Proxy as LiteLLM Proxy
    participant UG as UnifiedLLMGuardrails
    participant LLM as LLM Provider
    participant Mod as OpenAI Moderation API

    Client->>Proxy: Chat Completion (stream=true)
    Proxy->>LLM: Forward request
    LLM-->>UG: Stream chunks
    loop For each chunk
        UG-->>Client: Yield chunk immediately
        Note over UG: Collect chunk in buffer
    end
    Note over UG: Stream ended (finish_reason=stop)
    UG->>UG: stream_chunk_builder (assemble response)
    UG->>Mod: apply_guardrail (full text)
    Mod-->>UG: Moderation result
    alt Content flagged
        UG->>Client: HTTPException 400
    end

Loading

_{Last reviewed commit: b6657d0}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}