Repository navigation
Litellm OSS Staging #29161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mateo-berri
merged 50 commits into
litellm_internal_staging
from
litellm_oss_staging_28_05_26
Jun 2, 2026
Merged
Litellm OSS Staging #29161
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
aee740c
Cato Networks guardrail, based on Aim (#26597)
trexinc dbba9f1
Support Lemonade runtime context metadata (#28135)
neubig 4469d6d
fix(cato): address guardrail review feedback
Sameerlite 8747c17
fix(vertex_ai): route google/gemma-*-maas through partner-models Open…
icep87 a982643
fix(cato): guardrail all completion choices on output
Sameerlite 9b0a7c5
Fix review
Sameerlite f4d247d
fix(cato_networks): harden output anonymize handling and restructure …
Sameerlite 82d24e3
Fix mypy
Sameerlite bb73ed6
fix(cato): guard missing policy_drill_down and all_redacted_messages …
mateo-berri f885865
fix(cato): avoid KeyError bypassing block action on missing analysis_…
mateo-berri 21856c8
fix(cato): preserve non-text message fields during anonymize
mateo-berri 31f2fff
fix(cato): preserve trailing messages when fewer redacted messages re…
mateo-berri 6677812
Merge remote-tracking branch 'origin/litellm_internal_staging' into p…
mateo-berri 17675d1
fix(cato,model-info): preserve stream block signal on sender teardown…
mateo-berri 212d899
fix(cato): surface mid-stream forwarding errors instead of blocking o…
mateo-berri f9b3415
fix(cato): drop spoofable end_user_id from guardrail user identity
mateo-berri fad053d
fix(cato): harden output anonymize path against missing content key
mateo-berri 5fa9e8d
fix(cato): fall back to original message when redacted content key is…
mateo-berri 451672b
refactor(model-info): drop unused api_key from cached model-info helper
mateo-berri 3adc1df
fix(cato): preserve None content on tool-call-only choices in output …
mateo-berri e67add9
fix(ollama): respect static-model guard in OllamaConfig.get_model_info
mateo-berri 1b9e3d3
fix(lemonade,ollama): treat empty api_key as unset to avoid leaking s…
mateo-berri f1180b2
fix(cato): inspect and redact Responses-API input, not just messages
mateo-berri 0ab80b0
perf(utils): keep api_key out of get_model_info lru_cache key
mateo-berri 1273d35
fix(cato): propagate ssl_verify to streaming WebSocket connection
mateo-berri 1915626
refactor(utils): rename shadowing local in _get_model_info_helper
mateo-berri 9551ed5
fix(cato): flatten multimodal chat content before inspection
mateo-berri 8b45257
test(lemonade): clear get_model_info cache around api_base test
mateo-berri d9c5d4c
fix(cato): inspect and redact Responses-API input even when messages …
mateo-berri 34c758e
Merge branch 'litellm_internal_staging' into litellm_oss_staging_28_0…
mateo-berri 5f300d0
test(log_db_metrics): assert table_name event_metadata contract
mateo-berri 5caba25
fix(cato): inspect and redact completion prompt and Responses-API ins…
mateo-berri d0bee40
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri 7bd62af
fix(cato): serialize non-str/bytes websocket chunks before forwarding
mateo-berri f6b190d
fix(cato): inspect tool descriptions and tool-call arguments
mateo-berri 71d57ad
fix(cato): map redacted output by assistant index; restore get_model_…
mateo-berri 0879db6
fix(cato): block output even when detection_message is null/empty
mateo-berri 9c11f0c
fix(cato): inspect and redact nested tool param and legacy function d…
mateo-berri 53104d7
fix(cato): traverse schema descriptions iteratively to satisfy recurs…
mateo-berri 02a6458
fix(cato): inspect and redact response_format JSON schema descriptions
mateo-berri cfe4830
fix(cato): skip output rewrite when Cato returns no redaction
mateo-berri eeb1116
refactor(utils): resolve explicit api_key model info without the cache
mateo-berri 75d108c
fix(cato): inspect and redact non-description schema string values
mateo-berri 040d316
fix(model-info): surface swallowed dynamic model-info errors
mateo-berri 03970bd
fix(cato): inspect and redact Responses API output in post-call hook
mateo-berri 808bb92
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri 1ac9885
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri e65c15a
chore: reset _experimental/out folder
mateo-berri 9e8353b
chore(ui): remove orphaned prebuilt dashboard chunk files
mateo-berri 97c986b
fix(guardrails,ollama): forward ssl_verify to Cato init and raise_for…
mateo-berri File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
36 changes: 36 additions & 0 deletions
36
litellm/proxy/guardrails/guardrail_hooks/cato_networks/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from litellm.types.guardrails import SupportedGuardrailIntegrations | ||
|
|
||
| from .cato_networks import CatoNetworksGuardrail | ||
|
|
||
| if TYPE_CHECKING: | ||
| from litellm.types.guardrails import Guardrail, LitellmParams | ||
|
|
||
|
|
||
| def initialize_guardrail(litellm_params: "LitellmParams", guardrail: "Guardrail"): | ||
| import litellm | ||
| from litellm.proxy.guardrails.guardrail_hooks.cato_networks import ( | ||
| CatoNetworksGuardrail, | ||
| ) | ||
|
|
||
| _cato_callback = CatoNetworksGuardrail( | ||
| api_base=litellm_params.api_base, | ||
| api_key=litellm_params.api_key, | ||
| guardrail_name=guardrail.get("guardrail_name", ""), | ||
| event_hook=litellm_params.mode, | ||
| default_on=litellm_params.default_on, | ||
| ) | ||
| litellm.logging_callback_manager.add_litellm_callback(_cato_callback) | ||
|
|
||
| return _cato_callback | ||
|
|
||
|
|
||
| guardrail_initializer_registry = { | ||
| SupportedGuardrailIntegrations.CATO_NETWORKS.value: initialize_guardrail, | ||
| } | ||
|
|
||
|
|
||
| guardrail_class_registry = { | ||
| SupportedGuardrailIntegrations.CATO_NETWORKS.value: CatoNetworksGuardrail, | ||
| } |
323 changes: 323 additions & 0 deletions
323
litellm/proxy/guardrails/guardrail_hooks/cato_networks/cato_networks.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,323 @@ | ||
| # +-------------------------------------------------------------+ | ||
| # | ||
| # Use Cato Networks Guardrails for your LLM calls | ||
| # https://www.catonetworks.com/ | ||
| # | ||
| # +-------------------------------------------------------------+ | ||
| import asyncio | ||
| import json | ||
| import os | ||
| from typing import TYPE_CHECKING, Any, AsyncGenerator, Optional, Type, Union | ||
|
|
||
| from fastapi import HTTPException | ||
| from pydantic import BaseModel | ||
| from websockets.asyncio.client import ClientConnection, connect | ||
|
|
||
| from litellm import DualCache | ||
| from litellm._logging import verbose_proxy_logger | ||
| from litellm._version import version as litellm_version | ||
| from litellm.integrations.custom_guardrail import CustomGuardrail | ||
| from litellm.llms.custom_httpx.http_handler import ( | ||
| get_async_httpx_client, | ||
| httpxSpecialProvider, | ||
| ) | ||
| from litellm.proxy._types import UserAPIKeyAuth | ||
| from litellm.types.utils import ( | ||
| CallTypesLiteral, | ||
| Choices, | ||
| EmbeddingResponse, | ||
| ImageResponse, | ||
| ModelResponse, | ||
| ModelResponseStream, | ||
| ) | ||
|
|
||
| if TYPE_CHECKING: | ||
| from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel | ||
|
|
||
|
|
||
| class CatoNetworksGuardrailMissingSecrets(Exception): | ||
| pass | ||
|
|
||
|
|
||
| class CatoNetworksGuardrail(CustomGuardrail): | ||
| def __init__( | ||
| self, api_key: Optional[str] = None, api_base: Optional[str] = None, **kwargs | ||
| ): | ||
| ssl_verify = kwargs.pop("ssl_verify", None) | ||
| self.async_handler = get_async_httpx_client( | ||
| llm_provider=httpxSpecialProvider.GuardrailCallback, | ||
| params={"ssl_verify": ssl_verify} if ssl_verify is not None else None, | ||
| ) | ||
| self.api_key = api_key or os.environ.get("CATO_API_KEY") | ||
| if not self.api_key: | ||
| msg = ( | ||
| "Couldn't get Cato Networks api key, either set the `CATO_API_KEY` in the environment or " | ||
| "pass it as a parameter to the guardrail in the config file" | ||
| ) | ||
| raise CatoNetworksGuardrailMissingSecrets(msg) | ||
| self.api_base = ( | ||
| api_base | ||
| or os.environ.get("CATO_API_BASE") | ||
| or "https://api.aisec.catonetworks.com" | ||
| ) | ||
| self.api_base = self.api_base.rstrip("/") | ||
| self.ws_api_base = self.api_base.replace("http://", "ws://").replace( | ||
| "https://", "wss://" | ||
| ) | ||
| self.dlp_entities: list[dict] = [] | ||
| self._max_dlp_entities = 100 | ||
| super().__init__(**kwargs) | ||
|
|
||
| async def async_pre_call_hook( | ||
| self, | ||
| user_api_key_dict: UserAPIKeyAuth, | ||
| cache: DualCache, | ||
| data: dict, | ||
| call_type: CallTypesLiteral, | ||
| ) -> Union[Exception, str, dict, None]: | ||
| verbose_proxy_logger.debug("Inside Cato Pre-Call Hook") | ||
| return await self.call_cato_guardrail( | ||
| data, hook="pre_call", key_alias=user_api_key_dict.key_alias | ||
| ) | ||
|
|
||
| async def async_moderation_hook( | ||
| self, | ||
| data: dict, | ||
| user_api_key_dict: UserAPIKeyAuth, | ||
| call_type: CallTypesLiteral, | ||
| ) -> Union[Exception, str, dict, None]: | ||
| verbose_proxy_logger.debug("Inside Cato Moderation Hook") | ||
|
|
||
| await self.call_cato_guardrail( | ||
| data, hook="moderation", key_alias=user_api_key_dict.key_alias | ||
| ) | ||
| return data | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| async def call_cato_guardrail( | ||
| self, data: dict, hook: str, key_alias: Optional[str] | ||
| ) -> dict: | ||
| user_email = ( | ||
| data.get("metadata", {}).get("headers", {}).get("x-cato-user-email") | ||
|
veria-ai[bot] marked this conversation as resolved.
Resolved
|
||
| ) | ||
| call_id = data.get("litellm_call_id") | ||
| headers = self._build_cato_headers( | ||
| hook=hook, | ||
| key_alias=key_alias, | ||
| user_email=user_email, | ||
| litellm_call_id=call_id, | ||
| ) | ||
| response = await self.async_handler.post( | ||
| f"{self.api_base}/fw/v1/analyze", | ||
| headers=headers, | ||
| json={"messages": data.get("messages", [])}, | ||
|
veria-ai[bot] marked this conversation as resolved.
Resolved
|
||
| ) | ||
| response.raise_for_status() | ||
| res = response.json() | ||
| required_action = res.get("required_action") | ||
| action_type = required_action and required_action.get("action_type", None) | ||
| if action_type is None: | ||
| verbose_proxy_logger.debug("Cato: No required action specified") | ||
| return data | ||
| if action_type == "monitor_action": | ||
| verbose_proxy_logger.info("Cato: monitor action") | ||
| elif action_type == "block_action": | ||
| self._handle_block_action(res["analysis_result"], required_action) | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
| elif action_type == "anonymize_action": | ||
| return self._anonymize_request(res, data) | ||
| else: | ||
| verbose_proxy_logger.error(f"Cato: {action_type} action") | ||
| return data | ||
|
|
||
| def _handle_block_action(self, analysis_result: Any, required_action: Any) -> None: | ||
| detection_message = required_action.get("detection_message", None) | ||
| verbose_proxy_logger.info( | ||
| "Cato: Violation detected enabled policies: {policies}".format( | ||
| policies=list(analysis_result["policy_drill_down"].keys()), | ||
| ), | ||
| ) | ||
| raise HTTPException(status_code=400, detail=detection_message) | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| def _anonymize_request(self, res: Any, data: dict) -> dict: | ||
| verbose_proxy_logger.info("Cato: anonymize action") | ||
| redacted_chat = res.get("redacted_chat") | ||
| if not redacted_chat: | ||
| return data | ||
| data["messages"] = [ | ||
| { | ||
| "role": message["role"], | ||
| "content": message["content"], | ||
| } | ||
| for message in redacted_chat["all_redacted_messages"] | ||
| ] | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
| return data | ||
|
|
||
| async def call_cato_guardrail_on_output( | ||
| self, request_data: dict, output: str, hook: str, key_alias: Optional[str] | ||
| ) -> Optional[dict]: | ||
| user_email = ( | ||
| request_data.get("metadata", {}).get("headers", {}).get("x-cato-user-email") | ||
| ) | ||
| call_id = request_data.get("litellm_call_id") | ||
| response = await self.async_handler.post( | ||
| f"{self.api_base}/fw/v1/analyze", | ||
| headers=self._build_cato_headers( | ||
| hook=hook, | ||
| key_alias=key_alias, | ||
| user_email=user_email, | ||
| litellm_call_id=call_id, | ||
| ), | ||
| json={ | ||
| "messages": request_data.get("messages", []) | ||
| + [{"role": "assistant", "content": output}] | ||
| }, | ||
| ) | ||
| response.raise_for_status() | ||
| res = response.json() | ||
| required_action = res.get("required_action") | ||
| action_type = required_action and required_action.get("action_type", None) | ||
| if action_type and action_type == "block_action": | ||
| return self._handle_block_action_on_output( | ||
| res["analysis_result"], required_action | ||
| ) | ||
| redacted_chat = res.get("redacted_chat", None) | ||
|
|
||
| if action_type and action_type == "anonymize_action" and redacted_chat: | ||
| return { | ||
| "redacted_output": redacted_chat["all_redacted_messages"][-1]["content"] | ||
| } | ||
|
Sameerlite marked this conversation as resolved.
Resolved
|
||
| return {"redacted_output": output} | ||
|
|
||
| def _handle_block_action_on_output( | ||
| self, analysis_result: Any, required_action: Any | ||
| ) -> dict | None: | ||
| detection_message = required_action.get("detection_message", None) | ||
| verbose_proxy_logger.info( | ||
| "Cato: detected: {detected}, enabled policies: {policies}".format( | ||
| detected=True, | ||
| policies=list(analysis_result["policy_drill_down"].keys()), | ||
| ), | ||
| ) | ||
| return {"detection_message": detection_message} | ||
|
|
||
| def _build_cato_headers( | ||
| self, | ||
| *, | ||
| hook: str, | ||
| key_alias: Optional[str], | ||
| user_email: Optional[str], | ||
| litellm_call_id: Optional[str], | ||
| ): | ||
| """ | ||
| A helper function to build the http headers that are required by Cato guardrails. | ||
| """ | ||
| return ( | ||
| { | ||
| "Authorization": f"Bearer {self.api_key}", | ||
| # Used by Cato Networks to apply only the guardrails that should be applied in a specific request phase. | ||
| "x-cato-litellm-hook": hook, | ||
| # Used by Cato Networks to track LiteLLM version and provide backward compatibility. | ||
| "x-cato-litellm-version": litellm_version, | ||
| } | ||
| # Used by Cato Networks to track together single call input and output | ||
| | ({"x-cato-call-id": litellm_call_id} if litellm_call_id else {}) | ||
| # Used by Cato Networks to track guardrails violations by user. | ||
| | ({"x-cato-user-email": user_email} if user_email else {}) | ||
| | ( | ||
| { | ||
| # Used by Cato Networks apply only the guardrails that are associated with the key alias. | ||
| "x-cato-gateway-key-alias": key_alias, | ||
| } | ||
| if key_alias | ||
| else {} | ||
| ) | ||
| ) | ||
|
|
||
| async def async_post_call_success_hook( | ||
| self, | ||
| data: dict, | ||
| user_api_key_dict: UserAPIKeyAuth, | ||
| response: Union[Any, ModelResponse, EmbeddingResponse, ImageResponse], | ||
| ) -> Any: | ||
| if ( | ||
| isinstance(response, ModelResponse) | ||
| and response.choices | ||
| and isinstance(response.choices[0], Choices) | ||
| ): | ||
| content = response.choices[0].message.content or "" | ||
|
veria-ai[bot] marked this conversation as resolved.
Resolved
|
||
| cato_output_guardrail_result = await self.call_cato_guardrail_on_output( | ||
| data, content, hook="output", key_alias=user_api_key_dict.key_alias | ||
| ) | ||
| if cato_output_guardrail_result and cato_output_guardrail_result.get( | ||
| "detection_message" | ||
| ): | ||
| raise HTTPException( | ||
| status_code=400, | ||
| detail=cato_output_guardrail_result.get("detection_message"), | ||
| ) | ||
| if cato_output_guardrail_result and cato_output_guardrail_result.get( | ||
| "redacted_output" | ||
| ): | ||
| response.choices[0].message.content = cato_output_guardrail_result.get( | ||
| "redacted_output" | ||
| ) | ||
| return response | ||
|
|
||
| async def async_post_call_streaming_iterator_hook( | ||
| self, | ||
| user_api_key_dict: UserAPIKeyAuth, | ||
| response, | ||
| request_data: dict, | ||
| ) -> AsyncGenerator[ModelResponseStream, None]: | ||
| user_email = ( | ||
| request_data.get("metadata", {}).get("headers", {}).get("x-cato-user-email") | ||
| ) | ||
| call_id = request_data.get("litellm_call_id") | ||
| async with connect( | ||
| f"{self.ws_api_base}/fw/v1/analyze/stream", | ||
| additional_headers=self._build_cato_headers( | ||
| hook="output", | ||
| key_alias=user_api_key_dict.key_alias, | ||
| user_email=user_email, | ||
| litellm_call_id=call_id, | ||
| ), | ||
| ) as websocket: | ||
| sender = asyncio.create_task( | ||
| self.forward_the_stream_to_cato(websocket, response) | ||
| ) | ||
| while True: | ||
| result = json.loads(await websocket.recv()) | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
| if verified_chunk := result.get("verified_chunk"): | ||
| yield ModelResponseStream.model_validate(verified_chunk) | ||
| else: | ||
| sender.cancel() | ||
| if result.get("done"): | ||
| return | ||
| if blocking_message := result.get("blocking_message"): | ||
| from litellm.proxy.proxy_server import StreamingCallbackError | ||
|
|
||
| raise StreamingCallbackError(blocking_message) | ||
| verbose_proxy_logger.error( | ||
| f"Unknown message received from Cato: {result}" | ||
| ) | ||
| return | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| async def forward_the_stream_to_cato( | ||
| self, | ||
| websocket: ClientConnection, | ||
| response_iter, | ||
| ) -> None: | ||
| async for chunk in response_iter: | ||
| if isinstance(chunk, BaseModel): | ||
| chunk = chunk.model_dump_json() | ||
| if isinstance(chunk, dict): | ||
| chunk = json.dumps(chunk) | ||
| await websocket.send(chunk) | ||
| await websocket.send(json.dumps({"done": True})) | ||
|
|
||
| @staticmethod | ||
| def get_config_model() -> Optional[Type["GuardrailConfigModel"]]: | ||
| from litellm.types.proxy.guardrails.guardrail_hooks.cato_networks import ( | ||
| CatoNetworksGuardrailConfigModel, | ||
| ) | ||
|
|
||
| return CatoNetworksGuardrailConfigModel | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
20 changes: 20 additions & 0 deletions
20
litellm/types/proxy/guardrails/guardrail_hooks/cato_networks.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| from typing import Optional | ||
|
|
||
| from pydantic import Field | ||
|
|
||
| from .base import GuardrailConfigModel | ||
|
|
||
|
|
||
| class CatoNetworksGuardrailConfigModel(GuardrailConfigModel): | ||
| api_key: Optional[str] = Field( | ||
| default=None, | ||
| description="The API key for the Cato Networks guardrail. If not provided, the `CATO_API_KEY` environment variable is checked.", | ||
| ) | ||
| api_base: Optional[str] = Field( | ||
| default=None, | ||
| description="The API base for the Cato Networks guardrail. Default is https://api.aisec.catonetworks.com. Also checks if the `CATO_API_BASE` environment variable is set.", | ||
| ) | ||
|
|
||
| @staticmethod | ||
| def ui_friendly_name() -> str: | ||
| return "Cato Networks Guardrail" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.