fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException by yucheng-berri · Pull Request #32289 · BerriAI/litellm

yucheng-berri · 2026-07-06T22:20:48Z

Relevant issues

Linear ticket

Resolves LIT-4186

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on :4000 against a real AWS Bedrock guardrail. The guardrail has one topic-policy denial on the word "coffee" with blockedInputMessaging set to Sorry, the model cannot answer this question.

Launched with:

export PYTHONPATH="$(pwd)"
export DATABASE_URL="postgresql://.../litellm_lit4186"
export LITELLM_MASTER_KEY="sk-1234"
set -a; source .env; set +a
uv run --no-sync litellm --config /tmp/lit4186_config.yaml --port 4000 --detailed_debug

Config declares four guardrails against the same Bedrock identifier: bedrock-guard-pre-call (mode pre_call, disable_exception_on_block: true), bedrock-guard-during-call (during_call, disable_exception_on_block: true), bedrock-guard-post-call (post_call, disable_exception_on_block: true), and bedrock-guard-pre-call-strict (pre_call, disable_exception_on_block: false) for the negative regression check.

Before the fix: `pre_call` returns HTTP 500 with the block message

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"tell me about coffee"}],"guardrails":["bedrock-guard-pre-call"]}'

{"error":{"message":"Sorry, the model cannot answer this question.","type":"None","param":"None","code":"500"}}
---HTTP 500---

Before the fix: `during_call` silently returns the real model response

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"tell me about coffee"}],"guardrails":["bedrock-guard-during-call"]}'

{"id":"chatcmpl-1f65b52c-...","model":"bedrock-nova-micro","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Coffee is a popular beverage made from the roasted seeds ... Coffee continues to be a beloved and versatile beverage enjoyed by millions worldwide.","role":"assistant"}}],"usage":{"completion_tokens":688,"prompt_tokens":4,"total_tokens":692}}
---HTTP 200---

688 completion tokens billed for a request the guardrail intervened on

After the fix: `pre_call` returns HTTP 200 with the block message as content

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"tell me about coffee"}],"guardrails":["bedrock-guard-pre-call"]}'

{"id":"chatcmpl-ebb301a0-...","model":"bedrock-nova-micro","choices":[{"finish_reason":"content_filter","index":0,"message":{"content":"Sorry, the model cannot answer this question.","role":"assistant"}}],"usage":{"completion_tokens":0,"prompt_tokens":0,"total_tokens":0}}
---HTTP 200---

After the fix: `during_call` also returns HTTP 200; zero usage, no billable LLM call

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"tell me about coffee"}],"guardrails":["bedrock-guard-during-call"]}'

{"id":"chatcmpl-a2b0fe08-...","model":"bedrock-nova-micro","choices":[{"finish_reason":"content_filter","index":0,"message":{"content":"Sorry, the model cannot answer this question.","role":"assistant"}}],"usage":{"completion_tokens":0,"prompt_tokens":0,"total_tokens":0}}
---HTTP 200---

After the fix: `post_call` non-streaming block reports upstream call's real usage

The guardrail also blocks output containing "coffee". Ask a question whose answer will contain it so the LLM runs first and the guardrail then blocks the output; the synthetic block reply carries the upstream call's real token usage instead of zero.

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"say the word coffee three times"}],"guardrails":["bedrock-guard-post-call"]}'

{"id":"chatcmpl-6966c695-...","model":"bedrock-nova-micro","choices":[{"finish_reason":"content_filter","index":0,"message":{"content":"Sorry, the model cannot answer this question.","role":"assistant"}}],"usage":{"completion_tokens":31,"prompt_tokens":6,"total_tokens":37}}
---HTTP 200---

31 completion tokens from the upstream Bedrock call preserved on the block response.

After the fix: `post_call` streaming block emits a clean SSE synthetic stream with real usage

Same request with "stream": true and stream_options.include_usage: true. SSE headers are already flushed by the time the block fires, so the streaming iterator can't rely on the endpoint handler's 200 conversion; it synthesizes the block chunks locally and copies the upstream call's usage onto them.

curl -sS -N -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"say the word coffee three times"}],"stream":true,"stream_options":{"include_usage":true},"guardrails":["bedrock-guard-post-call"]}'

data: {"id":"chatcmpl-48cef74e-...","model":"bedrock-nova-micro","object":"chat.completion.chunk","choices":[{"finish_reason":"content_filter","index":0,"delta":{"content":"Sorry, the model cannot answer this question.","role":"assistant"}}],"usage":{"completion_tokens":13,"prompt_tokens":6,"total_tokens":19}}

data: [DONE]

---HTTP 200---

Real upstream usage on the streaming block, finish_reason=content_filter, clean [DONE] terminator; no error frame.

Regression check: flag=false blocked request still returns HTTP 400

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"tell me about coffee"}],"guardrails":["bedrock-guard-pre-call-strict"]}'

{"error":{"message":"Violated guardrail policy","code":"400","provider_specific_fields":{"error":"Violated guardrail policy","bedrock_guardrail_response":"Sorry, the model cannot answer this question.","assessments":[{"policy":"topicPolicy","matches":[{"category":"topics","name":"coffee","type":"DENY","action":"BLOCKED"}]}]}}}
---HTTP 400---

Regression check: allowed input still hits the real model

curl -sS -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"What is 2+2? Reply in one short sentence."}],"guardrails":["bedrock-guard-pre-call"]}'

{"id":"chatcmpl-01c32577-...","model":"bedrock-nova-micro","choices":[{"finish_reason":"stop","index":0,"message":{"content":"2 + 2 equals 4.","role":"assistant"}}],"usage":{"completion_tokens":10,"prompt_tokens":13,"total_tokens":23}}
---HTTP 200---

Regression check: allowed input, streaming, still hits the real model and reports usage

curl -sS -N -w "\n---HTTP %{http_code}---\n" http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-nova-micro","messages":[{"role":"user","content":"say hello briefly"}],"stream":true,"stream_options":{"include_usage":true},"guardrails":["bedrock-guard-post-call"]}'

data: {"id":"chatcmpl-8f3e9dfb-...","model":"bedrock-nova-micro","object":"chat.completion.chunk","choices":[{"finish_reason":"stop","index":0,"delta":{"content":"Hello! How can I assist you today?","role":"assistant"}}],"usage":{"completion_tokens":10,"prompt_tokens":3,"total_tokens":13}}

data: [DONE]

---HTTP 200---

Type

🐛 Bug Fix

Changes

BedrockGuardrail used to raise a Bedrock-specific GuardrailInterventionNormalStringError when a block coincided with disable_exception_on_block: true. That exception predates the unified guardrails refactor and no proxy code path handles it. In pre_call mode it escaped the unified path as an uncaught Exception, surfacing as HTTP 500 whose body carried the guardrail's blockedInputMessaging, which end users read as the model itself refusing. In during_call mode the native hook did catch the exception and set data["mock_response"], but route_request in common_request_processing.py unpacks kwargs before the moderation task in the parallel asyncio.gather gets a chance to run, so the mutation was dead code and the real model response won.

The fix converts a block with the flag set into ModifyResponseException at the raise site inside make_bedrock_api_request. That exception is already the standard proxy contract for guardrails that want a 200 response with a synthetic content-filter reply; it is caught in proxy_server.py, anthropic_endpoints/endpoints.py, response_api_endpoints/endpoints.py, and pass_through_endpoints.py. during_call blocks now cancel the parallel LLM task via the existing _cancel_pending_gather_tasks path, so no billable model call happens. Non-streaming post-call blocks attach the LLM response to ModifyResponseException.original_response so the synthetic reply reports the upstream call's real token usage rather than discarding it.

Streaming post_call needs a different shape. The SSE response headers are already flushed by the time the block fires, so a raise from the streaming iterator would be serialized by async_streaming_data_generator as a proxy error frame with code 500. Instead the streaming hook catches ModifyResponseException locally, snapshots the assembled response's .usage, rebinds the assembled response to a synthetic ModelResponse whose single choice carries the block message with finish_reason=content_filter, copies the snapshotted usage onto it, and lets the existing MockResponseIterator emit it as chunks. That produces the same shape a non-streaming block produces on the same guardrail.

The now-orphaned GuardrailInterventionNormalStringError class and the dead create_guardrail_blocked_response / data["mock_response"] plumbing in the Bedrock hooks are removed. Two pre-existing tests that had locked in the buggy contract are updated to assert the correct behavior. The mapped test file tests/test_litellm/proxy/guardrails/guardrail_hooks/test_bedrock_guardrails.py gains eight focused regression tests covering the raise-site conversion, all three hook types (pre_call, during_call, post_call), the unified apply_guardrail path, the streaming post_call synthetic-stream contract, and streaming usage preservation; each test was mutation-checked against the unfixed source and observed to fail before passing on the fix.

…ModifyResponseException The Bedrock-specific GuardrailInterventionNormalStringError predates the unified guardrails refactor and no proxy code path handles it, so a block with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call mode and was silently discarded in during_call mode (model call proceeded in the parallel asyncio.gather; the block hook's data["mock_response"] mutation happened after route_request had already unpacked kwargs). Convert the block to ModifyResponseException at the raise site inside make_bedrock_api_request. That exception is the industry-standard proxy contract already caught in proxy_server, anthropic_endpoints, response_api _endpoints, and pass_through_endpoints; it turns into a 200 response with finish_reason=content_filter and the block message as content, which is exactly what the flag was documented to yield. Post-call blocks attach the LLM response to original_response so the synthetic reply reports the upstream call's real token usage instead of zero. Deletes the now-orphaned GuardrailInterventionNormalStringError class and the dead create_guardrail_blocked_response / mock_response plumbing in the Bedrock hooks; updates the existing tests that had locked in the buggy contract. Resolves LIT-4186

greptile-apps · 2026-07-06T22:25:17Z

Greptile Summary

This PR fixes two bugs in the Bedrock guardrail when disable_exception_on_block: true is set: pre_call was returning HTTP 500 (block message escaped as an uncaught exception) and during_call was silently billing the model (guard result was stored in data["mock_response"] which never reached the LLM dispatch).

Replaces the now-orphaned GuardrailInterventionNormalStringError with ModifyResponseException at the raise site in make_bedrock_api_request, aligning Bedrock with the unified proxy guardrail contract already caught by proxy_server.py and the other endpoint handlers.
post_call attaches the real LLM response as original_response so the synthetic block reply reports actual token usage; streaming post_call catches the exception inside the generator (SSE headers already flushed) and emits a synthetic content_filter stream instead.
Removes dead code (create_guardrail_blocked_response, data["mock_response"] mutation paths) and narrows types accordingly; six focused regression tests added, two pre-existing tests corrected to assert the intended behavior.

Confidence Score: 5/5

Safe to merge; the changes are well-scoped to the Bedrock guardrail hooks and align with the existing ModifyResponseException contract used by all other endpoint handlers.

The fix is mechanically straightforward — a single raise-site conversion backed by thorough regression tests. Removed code has no surviving callers. The streaming post_call case is correctly handled in-band. Type narrowing for _update_messages_with_updated_bedrock_guardrail_response is safe because all callers now always receive a BedrockGuardrailResponse.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/exceptions.py	Removes `GuardrailInterventionNormalStringError`; no remaining imports of the class anywhere in the codebase.
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py	Core fix: `_get_http_exception_for_blocked_guardrail` now returns `ModifyResponseException` instead of the orphaned `GuardrailInterventionNormalStringError`; pre/during_call hooks let the exception propagate; post_call attaches original LLM response for usage preservation; streaming post_call catches and converts to synthetic stream; dead `create_guardrail_blocked_response` and `data["mock_response"]` paths removed.
tests/enterprise/litellm_enterprise/proxy/guardrails/test_bedrock_apply_guardrail.py	Regression test updated from `GuardrailInterventionNormalStringError` to `ModifyResponseException`; assertion now directly accesses `.message` attribute instead of `str(exc_info.value.message)`.
tests/guardrails_tests/test_bedrock_guardrails.py	Two tests that locked in the buggy contract (no exception raised, mock_response set) updated to assert correct behavior (ModifyResponseException raised; streaming delivers synthetic content_filter chunks).
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_bedrock_guardrails.py	Six new regression tests added covering all three hook types (pre_call, during_call, post_call), the unified apply_guardrail path, streaming synthetic chunk shape, and streaming usage preservation.

_{Reviews (4): Last reviewed commit: "fix(guardrails/bedrock): preserve upstre..." | Re-trigger Greptile}

codecov · 2026-07-06T22:28:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

yucheng-berri · 2026-07-07T02:00:57Z

bugbot run

…ith_updated_bedrock_guardrail_response Follow-up to the disable_exception_on_block fix. That method used to receive either a BedrockGuardrailResponse or a plain string (the block message, when the flag was set). Now that a block always raises ModifyResponseException before this method runs, the string branch is unreachable; tighten the type to BedrockGuardrailResponse and delete the guard.

yucheng-berri · 2026-07-07T02:28:03Z

@greptileai

yucheng-berri · 2026-07-07T02:28:41Z

Fixed in 483d34c: dropped the unreachable str branch and tightened the type to BedrockGuardrailResponse. On the Cursor Bugbot finding: streaming post_call blocks are out of scope for this PR (the ticket covers pre_call and during_call). Pre-fix, streaming post_call silently discarded blocks; post-fix, it surfaces as a 500 via the streaming error path. Both wrong; the fix is teaching the bedrock streaming hook to emit a synthetic block through build_block_sse_chunks the way unified_guardrail does. Filing a follow-up.

…tream instead of surfacing as SSE 500 Regression from the LIT-4186 refactor: pre-refactor, the streaming post_call iterator caught GuardrailInterventionNormalStringError locally and replaced the assembled response with a synthetic content-filter message, then re-emitted it as chunks via MockResponseIterator. After the refactor the exception was re-raised as ModifyResponseException, which async_streaming_data_generator serializes as a proxy 500 error frame because the SSE response headers are already flushed by the time the block fires. Non-streaming paths still let ModifyResponseException propagate to the endpoint handler (which converts it into a 200). Streaming can't do that, so keep the local synthesis: on the exception, rebind the assembled response to a ModelResponse whose single choice carries the block message as content and finish_reason=content_filter, and let the downstream MockResponseIterator emit it as chunks. Same shape a non-streaming block produces. Adds a mapped-file regression test that mutation-kills the raise behavior and locks in the synthetic-stream contract.

yucheng-berri · 2026-07-07T02:42:59Z

bugbot run

yucheng-berri · 2026-07-07T02:43:48Z

@greptileai

…ll block Non-streaming post_call blocks report the upstream LLM call's real token usage via ModifyResponseException.original_response, which the endpoint handler unwraps through _blocked_response_usage. Streaming post_call synthesizes its own ModelResponse locally (the exception can't escape the SSE generator), and previously left .usage unset, so the client saw accurate billing on non-streaming blocks and zero on streaming blocks -- silent revenue leak. Copy the assembled response's .usage onto the synthetic block response before yielding. Pre-refactor code had the same gap (create_guardrail_blocked_response never set usage); this is a net improvement, not a regression fix.

yucheng-berri · 2026-07-07T02:54:45Z

@greptileai

yucheng-berri · 2026-07-07T02:54:46Z

Valid finding, fixed in b9ce36f. Streaming block now copies .usage from the assembled response to the synthetic block ModelResponse before yielding, so streaming and non-streaming blocks report equivalent token usage. Note this is a net improvement over pre-refactor behavior (the old create_guardrail_blocked_response never set usage either), not a regression fix.

yucheng-berri · 2026-07-07T02:57:22Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b9ce36f. Configure here.}

cursor Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py Outdated

cursor Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py

cursor Bot reviewed Jul 7, 2026

View reviewed changes

yuneng-berri approved these changes Jul 7, 2026

View reviewed changes

mateo-berri mentioned this pull request Jul 9, 2026

fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path #32651

Closed

3 tasks

yucheng-berri mentioned this pull request Jul 9, 2026

fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path #32665

Merged

4 tasks

yucheng-berri merged commit 6eed38b into litellm_internal_staging Jul 9, 2026
128 checks passed

yucheng-berri deleted the litellm_bedrock_disable_exception_on_block branch July 9, 2026 20:18

yucheng-berri mentioned this pull request Jul 14, 2026

feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode #33299

Merged

5 tasks

devin-ai-integration Bot mentioned this pull request Jul 20, 2026

fix(anthropic-messages): handle ModelResponse mock_response from guardrail block #31992

Closed

Uh oh!

Uh oh!

Conversation

yucheng-berri commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before the fix: pre_call returns HTTP 500 with the block message

Before the fix: during_call silently returns the real model response

After the fix: pre_call returns HTTP 200 with the block message as content

After the fix: during_call also returns HTTP 200; zero usage, no billable LLM call

After the fix: post_call non-streaming block reports upstream call's real usage

After the fix: post_call streaming block emits a clean SSE synthetic stream with real usage

Regression check: flag=false blocked request still returns HTTP 400

Regression check: allowed input still hits the real model

Regression check: allowed input, streaming, still hits the real model and reports usage

Type

Changes

Uh oh!

greptile-apps Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

yucheng-berri commented Jul 7, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yucheng-berri commented Jul 6, 2026 •

edited

Loading

Before the fix: `pre_call` returns HTTP 500 with the block message

Before the fix: `during_call` silently returns the real model response

After the fix: `pre_call` returns HTTP 200 with the block message as content

After the fix: `during_call` also returns HTTP 200; zero usage, no billable LLM call

After the fix: `post_call` non-streaming block reports upstream call's real usage

After the fix: `post_call` streaming block emits a clean SSE synthetic stream with real usage

greptile-apps Bot commented Jul 6, 2026 •

edited

Loading

codecov Bot commented Jul 6, 2026 •

edited

Loading