diff --git a/docs/guides/security_settings.md b/docs/guides/security_settings.md index 3b6d44b00..e6b40d7b5 100644 --- a/docs/guides/security_settings.md +++ b/docs/guides/security_settings.md @@ -215,6 +215,17 @@ guardrails: ssl_verify: "/path/to/aim_cert.pem" # Use specific cert for AIM ``` +### Cato Networks Guardrail (Proxy) +You can configure `ssl_verify` per guardrail in your `config.yaml`. + +```yaml +guardrails: + - guardrail_name: cato-protected-app + litellm_params: + guardrail: cato_networks + ssl_verify: "/path/to/cato_cert.pem" # Use specific cert for AIM +``` + ### Priority Logic LiteLLM resolves `ssl_verify` using the following priority: 1. **Explicit Parameter**: Passed in `completion()` or guardrail config. diff --git a/docs/proxy/guardrails/cato_networks.md b/docs/proxy/guardrails/cato_networks.md new file mode 100644 index 000000000..b9a336e63 --- /dev/null +++ b/docs/proxy/guardrails/cato_networks.md @@ -0,0 +1,137 @@ +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Cato Networks + +## Quick Start +### 1. Create a new Cato Networks AI Security Guard + +Go to [Cato Networks CMA](https://cc.catonetworks.com/) and create a new AI Security guard. + +Name your guard and select the AI Gateway option. + +:::info +When using LiteLLM with virtual keys, use the virtual key alias as the name of the guard to be able to set key-specific policies. + +Only the aliases of your virtual keys (and not the actual key secrets) will be sent to Cato Networks. +::: + +### 2. Configure your Cato Networks AI Security Guard policies + +Create an Engine Profile, selecting which detections to enable. +Create a Guard Policy rule that references this Engine Profile and your Guard, then configure the action to apply. + +### 3. Add Cato Networks Guardrail on your LiteLLM config.yaml + +Define your guardrails under the `guardrails` section +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: openai/gpt-3.5-turbo + api_key: os.environ/OPENAI_API_KEY + +guardrails: + - guardrail_name: cato-protected-app + litellm_params: + guardrail: cato_networks + mode: [pre_call, post_call] # "During_call" is also available + api_key: os.environ/CATO_API_KEY + api_base: os.environ/CATO_API_BASE + ssl_verify: False # Optional, set to False to disable SSL verification or a string path to a custom CA bundle +``` + +Under the `api_key`, insert the API key you were issued. The key can be found in the guard's page. +You can also set `CATO_API_KEY` as an environment variable. + +By default, the `api_base` is set to `https://api.aisec.catonetworks.com`. Set the correct url for your region. +If you are using a self-hosted Outpost, you can set the `api_base` to your Outpost's URL. + +### 4. Start LiteLLM Gateway +```shell +litellm --config config.yaml +``` + +### 5. Make your first request + +:::note +The following example depends on enabling *PII* detection in your policy. +You can adjust the request content to match different guard's policies. +::: + + + + +:::note +When using LiteLLM with virtual keys, an `Authorization` header with the virtual key is required. +::: + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "gpt-3.5-turbo", + "messages": [ + {"role": "user", "content": "hi my email is ishaan@berri.ai"} + ], + "guardrails": ["cato-protected-app"] + }' +``` + +If configured correctly, since `ishaan@berri.ai` would be detected by the Cato Networks AI Security Guard as PII, you'll receive a response similar to the following with a `400 Bad Request` status code: + +```json +{ + "error": { + "message": "\"ishaan@berri.ai\" detected as email", + "type": "None", + "param": "None", + "code": "400" + } +} +``` + + + + + +:::note +When using LiteLLM with virtual keys, an `Authorization` header with the virtual key is required. +::: + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-3.5-turbo", + "messages": [ + {"role": "user", "content": "hi what is the weather"} + ], + "guardrails": ["cato-protected-app"] + }' +``` + +The above request should not be blocked, and you should receive a regular LLM response (simplified for brevity): + +```json +{ + "model": "gpt-3.5-turbo-0125", + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "I can’t provide live weather updates without the internet. Let me know if you’d like general weather trends for a location and season instead!", + "role": "assistant" + } + } + ] +} +``` + + + + + \ No newline at end of file diff --git a/docs/proxy/guardrails/quick_start.md b/docs/proxy/guardrails/quick_start.md index ed9d2ca12..6b76065f8 100644 --- a/docs/proxy/guardrails/quick_start.md +++ b/docs/proxy/guardrails/quick_start.md @@ -20,10 +20,10 @@ model_list: guardrails: - guardrail_name: general-guard litellm_params: - guardrail: aim + guardrail: cato_networks mode: [pre_call, post_call] - api_key: os.environ/AIM_API_KEY - api_base: os.environ/AIM_API_BASE + api_key: os.environ/CATO_API_KEY + api_base: os.environ/CATO_API_BASE default_on: true # Optional - guardrail_name: "aporia-pre-guard" @@ -114,7 +114,7 @@ litellm_settings: **Where this applies:** Only the **unified** guardrail path (providers that implement `apply_guardrail` and run through LiteLLM’s message translation layer) on **OpenAI Chat Completions** (`/v1/chat/completions`) and **Anthropic Messages** (`/v1/messages`). Examples include Presidio, Bedrock guardrails, `litellm_content_filter`, OpenAI Moderation, Generic Guardrail API, and custom code guardrails that define `apply_guardrail`. -**Where this does *not* apply:** Guardrails that run only via direct hooks on the raw request (e.g. Lakera v2, Aporia, DynamoAI, Javelin, Lasso, Pangea, Model Armor, Azure Content Safety hooks, Guardrails AI, AIM, tool permission, MCP security). It also does not apply to other routes until those endpoints use the same translation layer (e.g. Responses API, embeddings, speech). +**Where this does *not* apply:** Guardrails that run only via direct hooks on the raw request (e.g. Lakera v2, Aporia, DynamoAI, Javelin, Lasso, Pangea, Model Armor, Azure Content Safety hooks, Guardrails AI, AIM, Cato Networks, tool permission, MCP security). It also does not apply to other routes until those endpoints use the same translation layer (e.g. Responses API, embeddings, speech). ### Load Balancing Guardrails