Skip to content

fix(proxy): invoke post-call guardrails on pass-through endpoint responses (#20270) #26262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
krrish-berri-2 merged 2 commits into from
Apr 24, 2026
Merged

fix(proxy): invoke post-call guardrails on pass-through endpoint responses (#20270) #26262

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
059815f
fix(proxy): invoke post-call guardrails on pass-through endpoint resp…
tuhinspatra Apr 23, 2026
32cdace
test: add unit tests for pass-through post-call guardrails
tuhinspatra Apr 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion litellm/exceptions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

## LiteLLM versions of the OpenAI Exception Types

from typing import Optional
from typing import Any, Dict, Optional

import httpx
import openai
Expand Down Expand Up @@ -1016,6 +1016,35 @@ def __repr__(self):
return self.__str__()


class ModifyResponseException(Exception):
"""
Exception raised when a guardrail wants to modify the response.

This exception carries the synthetic response that should be returned
to the user instead of calling the LLM or instead of the LLM's response.
It should be caught by the proxy and returned with a 200 status code.

This is a base exception that all guardrails can use to replace responses,
allowing violation messages to be returned as successful responses
rather than errors.
"""

def __init__(
self,
message: str,
model: str,
request_data: Dict[str, Any],
guardrail_name: Optional[str] = None,
detection_info: Optional[Dict[str, Any]] = None,
):
self.message = message
self.model = model
self.request_data = request_data
self.guardrail_name = guardrail_name
self.detection_info = detection_info or {}
super().__init__(message)


class GuardrailInterventionNormalStringError(
Exception
): # custom exception to raise when a guardrail intervenes, but we want to return a normal string to the user
Expand Down
47 changes: 8 additions & 39 deletions litellm/integrations/custom_guardrail.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,43 +41,7 @@
dc = DualCache()


class ModifyResponseException(Exception):
"""
Exception raised when a guardrail wants to modify the response.

This exception carries the synthetic response that should be returned
to the user instead of calling the LLM or instead of the LLM's response.
It should be caught by the proxy and returned with a 200 status code.

This is a base exception that all guardrails can use to replace responses,
allowing violation messages to be returned as successful responses
rather than errors.
"""

def __init__(
self,
message: str,
model: str,
request_data: Dict[str, Any],
guardrail_name: Optional[str] = None,
detection_info: Optional[Dict[str, Any]] = None,
):
"""
Initialize the modify response exception.

Args:
message: The violation message to return to the user
model: The model that was being called
request_data: The original request data
guardrail_name: Name of the guardrail that raised this exception
detection_info: Additional detection metadata (scores, rules, etc.)
"""
self.message = message
self.model = model
self.request_data = request_data
self.guardrail_name = guardrail_name
self.detection_info = detection_info or {}
super().__init__(message)
from litellm.exceptions import ModifyResponseException as ModifyResponseException

@tuhinspatra tuhinspatra Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a pre-existing cyclic import through litellm/__init__.py — not introduced by this PR. Our change actually improved the situation: we moved the import target from custom_guardrail (which chains into custom_logger → proxy code) to exceptions (a leaf module with only typing, httpx, openai, litellm.types.utils imports). The cycle CodeQL detects is the top-level litellm package re-export chain that affects virtually every module in the codebase.



class CustomGuardrail(CustomLogger):
Expand Down Expand Up @@ -417,7 +381,9 @@ def should_run_guardrail(
"""
requested_guardrails = self.get_guardrail_from_metadata(data)
disable_global_guardrail = self.get_disable_global_guardrail(data)
opted_out_global_guardrails = self.get_opted_out_global_guardrails_from_metadata(data)
opted_out_global_guardrails = (
self.get_opted_out_global_guardrails_from_metadata(data)
)
verbose_logger.debug(
"inside should_run_guardrail for guardrail=%s event_type= %s guardrail_supported_event_hooks= %s requested_guardrails= %s self.default_on= %s",
self.guardrail_name,
Expand All @@ -426,7 +392,10 @@ def should_run_guardrail(
requested_guardrails,
self.default_on,
)
if self.default_on is True and self.guardrail_name in opted_out_global_guardrails:
if (
self.default_on is True
and self.guardrail_name in opted_out_global_guardrails
):
return False

if self.default_on is True and disable_global_guardrail is not True:
Expand Down
10 changes: 10 additions & 0 deletions litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,16 @@ async def async_post_call_success_hook(
if call_type is None:
call_type = _infer_call_type(call_type=None, completion_response=response) # type: ignore

# Fallback: resolve call_type from logging_obj for pass-through endpoints
if call_type is None:
litellm_logging_obj = data.get("litellm_logging_obj")
if (
litellm_logging_obj is not None
and getattr(litellm_logging_obj, "call_type", None)
== CallTypes.pass_through.value
):

@tuhinspatra tuhinspatra Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a pre-existing log line in unified_guardrail.py (line 237: verbose_proxy_logger.debug("async_post_call_success_hook response: %s", response)) — not introduced by this PR. Our changes start at line 248 (the call_type fallback block). The CodeQL finding is on existing code that happens to be in the diff context.

call_type = CallTypes.pass_through.value

if call_type is None:
return response

Expand Down
78 changes: 73 additions & 5 deletions litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -635,6 +635,7 @@ async def pass_through_request( # noqa: PLR0915
custom_llm_provider: Optional field - custom LLM provider for the endpoint
guardrails_config: Optional field - guardrails configuration for passthrough endpoint
"""
from litellm.exceptions import ModifyResponseException

@tuhinspatra tuhinspatra Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above — this is the same pre-existing litellm/__init__.py cycle, not introduced by this PR. The function-level import ensures no module-load-time issue; CodeQL is flagging the broader package-level cycle that exists for virtually every litellm.* module.

from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.proxy.pass_through_endpoints.passthrough_guardrails import (
PassthroughGuardrailHandler,
Expand Down Expand Up @@ -915,8 +916,41 @@ async def pass_through_request( # noqa: PLR0915

content = await response.aread()

## LOG SUCCESS
## POST-CALL GUARDRAILS ##
_content_modified = False
response_body: Optional[dict] = get_response_body(response)
if response_body is not None and guardrails_to_run:
# Build an enriched data dict: _parsed_body has been stripped of
# `metadata` by both pre_call_hook and _init_kwargs_for_pass_through_endpoint,
# so we re-attach the configured guardrails here so should_run_guardrail
# sees them.
hook_data = dict(_parsed_body or {})
existing_metadata = hook_data.get("metadata")
if not isinstance(existing_metadata, dict):
existing_metadata = {}
hook_data["metadata"] = {
**existing_metadata,
"guardrails": guardrails_to_run,
}
response_body = await proxy_logging_obj.post_call_success_hook(
data=hook_data,
user_api_key_dict=user_api_key_dict,
response=response_body, # type: ignore[arg-type]
)
Comment on lines +922 to +939

@greptile-apps greptile-apps Bot Apr 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 post_call_success_hook also fires non-guardrail CustomLogger callbacks

proxy_logging_obj.post_call_success_hook() iterates litellm.callbacks and calls async_post_call_success_hook on every non-guardrail CustomLogger instance unconditionally (see utils.py lines 2086-2091). The PR description and backwards-compatibility note state that "CustomLogger callbacks" won't activate on pass-through routes, but any CustomLogger in the global callback list will actually fire whenever guardrails_to_run is non-empty — this is a broader behavior change than the documentation implies. Users who have both custom logging callbacks and a guardrail configured on a pass-through route may observe unexpected side effects from those logging callbacks. Consider documenting this, or filtering to only the guardrail code path if that was the intent.

@tuhinspatra tuhinspatra Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — this is consistent with how post_call_success_hook works on all other endpoints (/chat/completions, /v1/messages, etc.) where both guardrail and non-guardrail CustomLogger callbacks fire in the same hook. Since pass-through guardrails are opt-in only (user must explicitly configure guardrails_config), enabling the hook is an intentional decision. Filtering to guardrail-only callbacks would diverge from the standard endpoint behavior and could surprise users who expect their logging callbacks to fire when guardrails are active. Happy to add a note in the PR description clarifying this.

if isinstance(response_body, dict):
content = json.dumps(response_body).encode("utf-8")
Comment thread
tuhinspatra marked this conversation as resolved.
_content_modified = True
else:
verbose_proxy_logger.debug(
"pass_through_endpoint: post_call_success_hook returned %s, expected dict — using original response",
type(response_body).__name__,
)
elif response_body is None:
verbose_proxy_logger.debug(
"pass_through_endpoint: response body not JSON-parseable, skipping post-call guardrails"
)

## LOG SUCCESS
passthrough_logging_payload["response_body"] = response_body
end_time = datetime.now()
asyncio.create_task(
Expand Down Expand Up @@ -944,13 +978,47 @@ async def pass_through_request( # noqa: PLR0915
api_base=str(url._uri_reference),
)

response_headers = HttpPassThroughEndpointHelpers.get_response_headers(
headers=response.headers,
custom_headers=custom_headers,
)
if _content_modified:
response_headers.pop("content-length", None)

return Response(
content=content,
status_code=response.status_code,
headers=HttpPassThroughEndpointHelpers.get_response_headers(
headers=response.headers,
custom_headers=custom_headers,
),
headers=response_headers,
)
except ModifyResponseException as e:
verbose_proxy_logger.info(
"pass_through_endpoint: Guardrail %s modified response: %s",
e.guardrail_name,
str(e.message or "")[:200],
)
try:
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict,
original_exception=e,
request_data=e.request_data,
)
except Exception:
verbose_proxy_logger.warning(
"pass_through_endpoint: post_call_failure_hook raised during guardrail block",
exc_info=True,
)
error_body = {
"error": {
"message": e.message or "Response blocked by guardrail",
"type": "content_filter",
"guardrail_name": e.guardrail_name,
"model": e.model,
}
}
return Response(
content=json.dumps(error_body),
status_code=200,
media_type="application/json",
)
except Exception as e:
Comment on lines +1010 to 1023

@greptile-apps greptile-apps Bot Apr 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 ModifyResponseException returns 400, but the contract requires 200

ModifyResponseException's own docstring says "It should be caught by the proxy and returned with a 200 status code", and both proxy_server.py (line 7168, comment: "return 200 with violation message") and anthropic_endpoints/endpoints.py (line 72, same comment) honour that. Returning 400 breaks clients that rely on the semantic guarantee that guardrail substitutions are surfaced as valid (200) responses, not as HTTP errors.

The PR description even names the test test_modify_response_exception_returns_200, but the committed test asserts result.status_code == 400 — confirming the intent was 200 and this is an unintended divergence.

        return Response(
            content=json.dumps(error_body),
            status_code=200,          # ← should be 200, not 400
            media_type="application/json",
        )

@tuhinspatra tuhinspatra Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a5c7abe — changed back to status_code=200 to match the ModifyResponseException contract (consistent with proxy_server.py and anthropic_endpoints.py). The response body uses a provider-agnostic error envelope instead of ModelResponse since pass-through clients may be any provider (Gemini, Bedrock, etc.).

custom_headers = ProxyBaseLLMRequestProcessing.get_custom_headers(
Expand Down
Loading
Loading