fix(responses): monitor-mode output guardrails must not hold back or fail streaming closed

[jarvis9443]

Problem

AISIX-Cloud#1010: a customer running 0.3.1 with an aliyun output guardrail in monitor mode saw intermittent 422 rejections on Codex traffic — dashboard rows showing content_filter, 0 tokens, $0.0000, and 16–26 s latency. Monitor mode is documented to observe and never block.

Root cause: the streaming /v1/responses path (both the verbatim OpenAI forward and the cross-provider bridge) entered the whole-response hold-back branch whenever any output-hook guardrail was attached, ignoring the chain's resolved stream_output_policy:

• a monitor-only chain resolves to EndOfStreamCheck (can never block), yet its stream was fully buffered — the client saw nothing until the generation finished;
• past the buffer cap (DEFAULT_STREAM_OUTPUT_BUFFER_BYTES = 256 KiB, hit by exactly the long SSE streams Codex produces), the request was rejected 422 content_filter before any verdict ran — MonitorGuardrail never got the chance to downgrade.

chat.rs and messages.rs already gate hold-back on the policy (holds_back() / BufferFull-only); /v1/responses was the one deviating surface (family audit: completions/audio/passthrough/realtime have no streaming output-buffer path).

Fix

• Hold-back (and its fail-closed overflow) now engages only when the resolved policy actually holds back (Window/BufferFull — any block-capable chain). Block-mode behavior is unchanged, including the fail-closed cap.
• An EndOfStreamCheck chain (monitor-only) forwards the SSE live and runs the same two-phase scan (blob check + segment pass) at end-of-stream, so would_block/would_mask monitor hits still reach telemetry. The scan runs while the completion guard stays armed: SDK clients close the connection right after the terminal frame, and a disconnect mid-scan must fall back to the guard's Drop emit rather than lose the usage event for a fully-delivered stream. Scan input is bounded to the same 256 KiB on both paths so observation provider calls stay bounded.
• A Block verdict on the live path (only reachable via the documented mandatory-unavailability composition) is logged and, on the bridge path, signalled with a trailing error frame — mirroring chat's EndOfStreamCheck behavior.

Behavior change

Monitor-mode-only output chains on streaming /v1/responses: clients now receive tokens live (no whole-response buffering latency) and oversized responses are no longer rejected. Blocking chains: no change.

LiteLLM baseline

LiteLLM never withholds or fails a stream for logging-only / on_flagged: monitor guardrails, and has no scan-buffer size cap at all — live-forward + observe-at-end matches the baseline. Keeping the fail-closed cap for blocking chains is our stricter, OOM-bounding divergence (pre-existing, unchanged).

Tests

• Unit (responses.rs): oversized (300 KB) stream + monitor guardrail released with 200 on both paths (verbatim + cross-provider bridge) — both fail before the fix (mutation-verified); live-path would_block observation recorded on the usage event; disconnect-during-scan still emits the usage event (parks the scan on a delayed moderation backend, drops the body — mutation-verified against the take-before-await shape); existing block-mode oversized fail-closed tests still pass unchanged.
• E2E (tests/e2e): self-gating block→monitor flip on a >256 KiB /v1/responses stream — block mode 422s (pins the secure default), monitor mode releases the full SSE live.

Fixes api7/AISIX-Cloud#1010

[coderabbitai]

📝 Walkthrough

Walkthrough

/v1/responses streaming now distinguishes monitor-only output guardrails from hold-back enforcement, performs clean end-of-stream scans, propagates monitor hits to telemetry, and tests both verbatim and cross-provider oversized streams.

Changes

Responses streaming guardrails

Layer / File(s)	Summary
OpenAI streaming policy and telemetry crates/aisix-proxy/src/responses.rs	Shared guardrail chains and output policies control buffering; clean stream completion scans captured output and includes output monitor hits in usage events.
Cross-provider bridge streaming policy crates/aisix-proxy/src/responses.rs, crates/aisix-proxy/src/responses_bridge.rs	Bridge buffering, overflow handling, scanning, and redaction now differ between hold-back and live-forward modes.
Monitor-mode streaming regression coverage tests/e2e/src/cases/responses-streaming-monitor-guardrail-e2e.test.ts, crates/aisix-proxy/src/responses.rs	Tests verify oversized streams block in BLOCK mode but complete with forbidden content and monitor observations in MONITOR mode.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ResponsesHandler
  participant StreamingPath
  participant Guardrail
  participant UsageEvent
  Client->>ResponsesHandler: request streaming /v1/responses
  ResponsesHandler->>StreamingPath: apply output policy
  StreamingPath-->>Client: forward live SSE or hold frames
  StreamingPath->>Guardrail: scan output at stream end
  Guardrail-->>StreamingPath: monitor hits or redaction result
  StreamingPath->>UsageEvent: emit usage and monitor hits

Loading

Possibly related PRs

• api7/aisix#640: Defines the monitor-mode stream_output_policy behavior used to select end-of-stream checking.
• api7/aisix#731: Adds the monitor-hit model and telemetry APIs consumed by this streaming completion flow.
• api7/aisix#694: Overlaps with the bridged Responses stream redaction and held-frame handling.

🚥 Pre-merge checks | ✅ 4 | ❌ 2

❌ Failed checks (2 inconclusive)

Check name	Status	Explanation	Resolution
E2e Test Quality Review	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Security Check	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address #1010 by keeping monitor-only guardrails from blocking oversized streaming responses while still recording monitor hits.
Out of Scope Changes check	✅ Passed	The code and test changes stay focused on streaming output-guardrail behavior and related regression coverage.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: monitor-mode output guardrails should stream live without hold-back or fail-closed behavior.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/responses-monitor-stream-holdback

---

_{Comment @coderabbitai help to get the list of available commands.}