feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes

[devin-ai-integration]

TLDR

Problem this solves:

• Admins can't run a guardrail only on streaming or only on non-streaming traffic
• The first take in feat(guardrails): run each mode only on streaming, non-streaming, or both #43174 treated Bedrock converse-stream and invoke-with-response-stream as non-streaming
• It also dropped caller body fields named is_streaming_request, and broke reads of stored rows holding a bad value

How it solves it:

• New stream_scope guardrail setting: streaming, non_streaming or both (default), as one value or per mode
• Bedrock streaming actions count as streaming on router and direct pass-through routes
• The server's own streaming marker lives under a namespaced key and is checked by identity, so caller fields pass through untouched
• Invalid stored or YAML values are read as unset (both) with a warning, while new create, update and patch calls reject them with a 422

Intentional product change: POST /guardrails, PUT /guardrails/{id} and PATCH /guardrails/{id} now return 422 for an invalid stream_scope, where main stored the value and ignored it

User Flow

Before: a proxy admin can't limit a guardrail to streamed requests, so every rail runs on every request

1. The admin sets stream_scope: streaming on a pre-call rail and stream_scope: non_streaming on another
2. A developer sends POST https://litellm-domain/bedrock/model//converse-stream with a key that carries both rails
3. Both rails scan the request, because main ignores stream_scope

After: each rail only runs on the request shape it is scoped to

1. The admin sets stream_scope: streaming on a pre-call rail and stream_scope: non_streaming on another
2. A developer sends POST https://litellm-domain/bedrock/model//converse-stream with a key that carries both rails
3. Only the streaming rail scans it, and the same call to /converse runs only the non_streaming rail. Chat, Messages and Responses behave the same way based on "stream": true
4. A body field the caller names is_streaming_request still reaches the pass-through target unchanged

Relevant issues

Follows up on #43174, which is left open for now. This branch carries its commits plus the fixes above

Linear ticket

Resolves LIT-8991

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Docs: litellm-docs#2191 documents stream_scope (values, per-mode maps, validation) in the guardrails quick start.

What this PR looks like (screenshots)

What this shows: before the reviewer fix, opening the Custom Code editor for a guardrail whose mode is the tag-scoped dict {tags: {team-a: pre_call}}, default: post_call crashed the whole editor (React cannot render the raw object).

What this shows: after the fix, the same editor opens and shows the mode read-only as post_call, pre_call (tag-based) — the stored tag scoping is preserved and never rewritten on save.

What this shows: the guardrail detail view rendering the mode read-only (same formatGuardrailMode convention as the rest of the dashboard).

What this shows: an editor save succeeding during a 24x concurrent burst on a 4-worker rig (chaos cell from the audit — the read-only path stays stable under load).

What this shows: animated before-to-after flow of the editor crash turning into the working read-only view.

Both proxies ran the same config with 4 workers, real Postgres and Redis, real OpenAI, Anthropic and AWS Bedrock calls, and a local guardrail sink that records which rail was called. The key carries r_pre_stream (pre_call, stream_scope: streaming) and r_pre_nonstream (pre_call, stream_scope: non_streaming). r_post_unset is a post_call rail with no stream_scope. /pt-echo is a configured pass-through to a local echo target

UI configuration (real dashboard screenshots)

What this shows: the Add Guardrail form's Request shape control configuring stream_scope per mode; pre_call applies to is set to Streaming only via the dropdown (Streaming and non-streaming / Streaming only / Non-streaming only).

What this shows: the stored guardrail's detail page rendering the per-mode scope summary read-only: pre_call: Streaming only, post_call: Streaming and non-streaming.

Before (c168199, merge base with main)

Bedrock pass-through

1. curl localhost:17751/bedrock/model/live-bedrock/converse -H 'Authorization: Bearer <key>' -d '{"messages":[{"role":"user","content":[{"text":"Reply ok"}]}],"inferenceConfig":{"maxTokens":5}}' returns 200 application/json, and the sink shows {'R_PRE_STREAM': 1, 'R_PRE_NONSTREAM': 1}
2. The same body to /converse-stream returns 200 application/vnd.amazon.eventstream, and the sink shows {'R_PRE_STREAM': 1, 'R_PRE_NONSTREAM': 1}
3. /invoke-with-response-stream with an Anthropic body returns 200 application/vnd.amazon.eventstream, and the sink shows {'R_PRE_STREAM': 1, 'R_PRE_NONSTREAM': 1}

Chat, Messages and Responses

1. POST /v1/chat/completions, /v1/messages and /v1/responses without stream all return 200 and every rail runs, including R_POST_STREAM
2. The same three calls with "stream": true return 200 chunked and every rail runs, including R_PRE_NONSTREAM

Caller field on pass-through

1. curl localhost:17751/pt-echo -H 'Authorization: Bearer <master key>' -d '{"hello":"x","is_streaming_request":"caller-value"}' returns 200
2. The echo target received {'hello': 'x', 'is_streaming_request': 'caller-value'}

Bad stream_scope values

1. POST /guardrails with "stream_scope": "sometimes" returns 200 and stores it
2. GET /guardrails/{id}/info on a stored row holding "sometimes" returns 200, and chat through that rail returns 200 with the rail run once
3. GET /guardrails/list with a YAML rail holding stream_scope: sometimes returns 200 and echoes "sometimes"

After (f38aa05, rerun at e9495a8)

e9495a8 and the tip 4d0da16 have the same litellm/ tree, since 4d0da16 only changes a test. The rerun at e9495a8 gave the same rail split as below across 33 scenarios, with the same status, content type and upstream calls on base and head

Bedrock pass-through

1. curl localhost:17750/bedrock/model/live-bedrock/converse -H 'Authorization: Bearer <key>' -d '{"messages":[{"role":"user","content":[{"text":"Reply ok"}]}],"inferenceConfig":{"maxTokens":5}}' returns 200 application/json, and the sink shows {'R_PRE_NONSTREAM': 1}
2. The same body to /converse-stream returns 200 application/vnd.amazon.eventstream, and the sink shows {'R_PRE_STREAM': 1}
3. /invoke-with-response-stream with an Anthropic body returns 200 application/vnd.amazon.eventstream, and the sink shows {'R_PRE_STREAM': 1}. Router models (/bedrock/model/<router model>/...) and direct /bedrock/... routes gave the same split

Chat, Messages and Responses

1. POST /v1/chat/completions, /v1/messages and /v1/responses without stream all return 200 and the sink shows R_PRE_NONSTREAM and R_POST_UNSET, with no R_PRE_STREAM or R_POST_STREAM
2. The same three calls with "stream": true return 200 chunked and the sink shows R_PRE_STREAM, R_POST_STREAM and R_POST_UNSET, with no R_PRE_NONSTREAM

Caller field on pass-through

1. curl localhost:17750/pt-echo -H 'Authorization: Bearer <master key>' -d '{"hello":"x","is_streaming_request":"caller-value"}' returns 200
2. The echo target received {'hello': 'x', 'is_streaming_request': 'caller-value'}, same as main. Sending is_streaming_request: true or litellm_server_streaming_classification: true to a guarded pass-through still runs only R_PRE_NONSTREAM, so callers can't fake a stream

Bad stream_scope values

1. POST /guardrails with "stream_scope": "sometimes", 1 or a mapping with an unknown mode returns 422, and PATCH with "sometimes" returns 422. Valid, mixed-case and null values return 200
2. GET /guardrails/{id}/info on a stored row holding "sometimes" returns 200, and chat through that rail returns 200 with the rail run once, streaming and non-streaming
3. GET /guardrails/list with a YAML rail holding stream_scope: sometimes returns 200 with stream_scope: null, where the previous head of this branch returned 500

Vertex Live websocket pass-through

1. Both proxies get a default_on pre_call rail with stream_scope: streaming and another with stream_scope: non_streaming, plus real Vertex credentials
2. A client opens ws://localhost:<port>/vertex_ai/live and sends a setup frame for gemini-live-2.5-flash-native-audio
3. Both legs get setupComplete back from Vertex. Main ran both rails, while this branch ran only the streaming rail

Integration audit (4d0da16 vs merge base 7be2983)

Checked-in tests, owned Postgres, Redis, scripted upstream and guardrail sinks, 2 proxy workers, run.py extensions --seed 4106601 --order-seed 0 --workers 2:

• tests/integration/observability/test_guardrail_stream_scope.py
• tests/integration/observability/test_guardrail_stream_scope_matrix.py
• tests/integration/observability/test_guardrail_stream_scope_chaos.py

run	collected	passed	failed	skipped
base	208	80	127	1
head-1	208	207	0	1
head-2	208	207	0	1

• The 127 base failures are all head-only behavior: scope filtering (main runs every rail on every request shape), the Vertex Live websocket cell, and the v1 list returning null for a bad YAML value. Unset, explicit both, caller field forwarding, unauthenticated and provider-error cells pass on both
• head-1 and head-2 collected the same nodes in the same order with the same outcomes
• The one skip is the Postgres-restart spend-row test, which fails on main too (LIT-9050)
• The websocket cell runs the real /vertex_ai/live route against a scripted Vertex peer and checks the upstream path, auth header, forwarded frame and setupComplete reply
• Covers chat, messages and responses streaming and non-streaming in all four modes, OpenAI and Anthropic SDK sync and async, Bedrock router and direct actions, configured, Anthropic, Gemini and websocket pass-through, key, team and default-on rails, per-mode maps, management 422s, bad stored and YAML values, MCP, policy pipelines, realtime transcription, and sink, worker, proxy and Postgres outages mid-burst
• Not covered: enterprise-only hooks and real vendor guardrails

ran /live-pr-risk and found no regressions/backward incompatible risks (re-verified at 2dbea2d: full live matrix incl. stream-scope gating, constructor shapes, tolerant register/PATCH composition, tag-mode read-only rendering, 8x bursts)

Type

🆕 New Feature
🐛 Bug Fix
✅ Test

Caveats (if any)

Medium

• Invalid stream_scope on create, update or patch now returns 422 instead of being stored
  • Rows already stored with a bad value keep working as both

Low

• A pre-call rail whose vendor hangs still holds requests open on both main and this branch
  • A rail scoped out of the request is skipped, so that request is no longer held
• Requests served while Postgres restarts can lose spend rows, on main and on this branch (LIT-9050). Its test is checked in and skipped
• Realtime transcription, MCP tool calls and team plus key rails were also run live at de47ffe on real OpenAI gpt-realtime-2.1 and gpt-5-mini and the remote DeepWiki MCP server. Only the intended scope differences showed up: a non_streaming transcription rail and a streaming MCP rail no longer run, and team and key rails each follow their own scope. A client that sends its own response.create after a clean transcript gets conversation_already_has_active_response, on main too
• A streamed /v1/responses call runs post-call rails more than once, on both main and this branch
• The tip de47ffe merges main into the audited 4d0da16 and adds fixes from two gauntlet runs. The server streaming marker is a str enum, so hooks can json.dumps their data, and it is still identity-checked so JSON can't forge it. It is now stripped by value too, so a hook that rebuilds the body can't leak it upstream, and it is kept out of pass-through guardrail scan text and the proxy_server_request snapshot. _update_stream_param_based_on_request_body and BEDROCK_STREAMING_ACTIONS keep their main names, and BEDROCK_STREAMING_ACTIONS is now a frozenset. YAML guardrails that only subclass CustomLogger boot again, subclasses that skip super().__init__() default to both, and guardrails deep-copy and pickle. The mapped unit suites were rerun there, but the audit and live runs were not
• The tip 3d6015d only merges main again, resolving a test import conflict, and the mapped unit suites pass there
• The tip 2ef0bc0 fixes three findings from a third gauntlet run. Pass-through guardrail scan text and the proxy_server_request snapshot now drop the server marker by value instead of by key name, so caller content under litellm_server_streaming_classification is scanned and logged like on main. Per-hook scopes are stored as a plain tuple with no custom __getstate__/__setstate__, so subclasses that define their own state hooks or __slots__ copy and pickle again, and a Router with such a guardrail in deployment callbacks builds. The mapped unit suites pass there
• Accepted from that run: custom logger hooks on streaming pass-through and websocket routes see the server marker key in their request data, and a caller field whose value equals the marker string litellm-server-streaming is stripped. A custom guardrail class that read its own stream_scope init argument no longer receives it, since stream_scope is now a reserved setting
• The tip 6da52e2 only changes the dashboard: the custom code editor shows a tag-scoped mode read-only instead of crashing, and leaves mode and stream_scope out of its update payload for those guardrails
• The tip 930b568 merges main and keeps stream_scope and the new logging_only_scope side by side

Link to Devin session: https://app.devin.ai/sessions/40ff0e0ca23e4d7cb5b602c39bfb221c
Open in Devin Desktop: https://app.devin.ai/desktop/session/40ff0e0ca23e4d7cb5b602c39bfb221c?variant=devin
Requested by: @yucheng-berri

Merge resolution (6d15060)

Current main was merged into this branch (13 conflicting files between per-mode stream_scope and logging_only_scope); both features are kept side by side — stream-scope state/payload/captions plus the logging-only-scope choice and clear-stored-scope flow, with the two tolerant parsers composed and main's GuardrailModeCard/GuardrailModeRows refactor kept with the stream-scope detail reattached. Mapped Python suites: 1226 passed; guardrails UI suite: 298 passed, no type errors. The osv-scan failure is a newly published next 16.3.6 advisory failing on every open PR and is not introduced here.

---

Note

Medium Risk
Changes when security guardrails execute across chat, Bedrock, pass-through, and pipelines; invalid scope on write now 422s instead of being stored silently.

Overview
Adds stream_scope so guardrails can run only on streaming, non-streaming, or both (default), as a single value or per hook (pre_call, post_call, etc.). CustomGuardrail.should_run_guardrail and policy pipeline steps now skip out-of-scope rails (skip / next).

The proxy classifies server-side streaming via litellm_server_streaming_classification (identity-checked marker), not caller fields like is_streaming_request. That marker is added for Bedrock streaming actions, pass-through/WebSocket routes, and realtime paths, and is stripped from upstream bodies, guardrail scan payloads, and request snapshots so clients cannot spoof streaming scope.

Bedrock streaming detection uses the final path action (converse-stream, invoke-with-response-stream) instead of substring matches on model names. Guardrail registry/API paths tolerate invalid stored stream_scope as unset (both) while create/update/patch reject bad values with 422; OpenAPI documents the new field.

^{Reviewed by Cursor Bugbot for commit 5640dc9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[mateo-berri]

bugbot run

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds per-mode stream scope configuration to guardrails system.

The current head appears safe to merge; no new actionable issue or outstanding previous finding remains

Summary

The PR adds per-mode streaming scopes for guardrails across model, pass-through, Bedrock, realtime, and policy-pipeline paths. It also adds tolerant reads for invalid stored scopes, strict validation for new writes, dashboard controls, and integration coverage. Since the previous review, the head removes an unused Bedrock constant re-export

_{Reviews (21) · Last reviewed commit: "chore(pass-through): drop unused BEDROCK..." · Reviewed by Greptile}

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@greptileai please review the current head ab01cf0 — the parameter-mutation finding is addressed as a documented mutable-ok (rebinding to a fresh mapping breaks 75 pass-through tests; the in-place strip is the feature's caller-visible contract), plus the earlier merge fallout fixes.

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@greptileai please review the current head 5640dc9 — one-line fix moving stream_scope after timeout in CustomGuardrail init to restore the base positional contract (positional timeout callers were binding to stream_scope); keyword callers unaffected; 1458 mapped tests passing.

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@greptileai please review the current head ee7b659 — typing pass for the lint gates (stream_scope out of the declared constructor signature restoring the base positional surface, typed boundaries for config values); all three lint gates green vs current main; 6158 mapped tests passing.

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@greptileai please review the current head ff4d50f — import-order fix on top of the typing pass; all three lint gates green locally (type_check, ruff strict, type discipline), ruff format clean, 236 guardrail tests passing.

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

The rust-wheel/rust-test failures are pre-existing on main, not from this branch: main's #45171 (lens_feedback, 03ba79d261) added a fourth TTL statement (lens_feedback) that test_schema_setup_uses_configured_retention was never updated for (the test still expects three: otel_traces, agent_traces_by_key, spend_logs). This branch does not touch rust/ or that test (verified byte-identical to main), and the same failure hits every sibling PR whose diff triggers the path-filtered LiteLLM Rust workflow today — litellm_responses_drop_malformed_prompt_cache_breakpoint, litellm_gemini3_sampling_defaults, litellm_ci_heal_rust_test_disk, and litellm_databricks_ai_decide_decisions all fail the identical assertion (verified in their rust-wheel logs). It needs a one-line expectation update on main's side.

[yucheng-berri]

@greptileai please review the current head 2dbea2d — dropped the unused BEDROCK_STREAMING_ACTIONS re-export (CodeQL finding), all lint gates re-verified green, full live matrix on this head: stream-scope gating (nonstream 200 / stream fires), constructor positional+keyword+map scopes, tolerant register/list/PATCH composition (invalid stream_scope → 422), tag-mode guardrail renders read-only in list and detail with no crash, 8x streaming burst + nonstream burst clean.

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2dbea2d. Configure here.}

[yucheng-berri]

Real Bedrock Guardrail verification (no mocks)

Verified at head 2dbea2d with a real AWS Bedrock Guardrail resource (word-deny policy, published version 1, created via the Bedrock API), wired through the stock guardrail: bedrock integration (mode: post_call, real ApplyGuardrail/streaming-scan calls), invoking real Bedrock (us.anthropic.claude-haiku-4-5 inference profile, SigV4). The model was asked to say the denied word, so the verdict in every cell is decided by AWS's own guardrail evaluation:

stream_scope	Non-streaming	Streaming
streaming	scan skipped → denied word passed through (content delivered)	guardrail ran → Violated guardrail policy block
non_streaming	guardrail ran → blocked	scan skipped → stream passed through (frames delivered)
both	guardrail ran → blocked	guardrail ran → blocked

Each skip leg delivering the denied word proves the scan genuinely did not run; each run leg returns Bedrock's own policy-violation block. All six cells match the per-mode stream_scope contract exactly. Temporary guardrail resource deleted after the run.