Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/my-website/docs/proxy/config_settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ router_settings:
| key_generation_settings | object | Restricts who can generate keys. [Further docs](./virtual_keys.md#restricting-key-generation) |
| disable_add_transform_inline_image_block | boolean | For Fireworks AI models - if true, turns off the auto-add of `#transform=inline` to the url of the image_url, if the model is not a vision model. |
| use_chat_completions_url_for_anthropic_messages | boolean | If true, routes OpenAI `/v1/messages` requests through chat/completions instead of the Responses API. Can also be set via env var `LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true`. |
| skip_system_message_in_guardrail | boolean | If true, unified guardrails omit `role: system` from scanned input on **chat completions** and **Anthropic `/v1/messages`** only; the LLM still receives full messages. Per-guardrail override: `litellm_params.skip_system_message_in_guardrail` on each guardrail. [Guardrails quick start](./guardrails/quick_start#skip-system-messages-in-guardrail-evaluation) |
| disable_hf_tokenizer_download | boolean | If true, it defaults to using the openai tokenizer for all models (including huggingface models). |
| enable_json_schema_validation | boolean | If true, enables json schema validation for all requests. |
| enable_key_alias_format_validation | boolean | If true, validates `key_alias` format on `/key/generate` and `/key/update`. Must be 2-255 chars, start/end with alphanumeric, only allow `a-zA-Z0-9_-/.@`. Default `false`. |
Expand Down
117 changes: 67 additions & 50 deletions docs/my-website/docs/proxy/guardrails/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Setup Prompt Injection Detection, PII Masking on LiteLLM Proxy (AI Gateway)
## 1. Define guardrails on your LiteLLM config.yaml

Set your guardrails under the `guardrails` section

```yaml
model_list:
- model_name: gpt-3.5-turbo
Expand Down Expand Up @@ -82,27 +83,58 @@ For generic guardrail APIs you can also set **static headers** (`headers`: key/v
- `during_call` Run **during** LLM call, on **input** Same as `pre_call` but runs in parallel as LLM call. Response not returned until guardrail check completes
- A list of the above values to run multiple modes, e.g. `mode: [pre_call, post_call]`

### Skip system messages in guardrail evaluation

You can stop **unified** guardrails from scanning `role: system` content while still sending the full `messages` list to the model.

**Global** — in `litellm_settings`:

```yaml
litellm_settings:
skip_system_message_in_guardrail: true
```

**Per guardrail** — under that guardrail’s `litellm_params`: set `skip_system_message_in_guardrail: true` or `false`. If omitted, the global `litellm_settings` value is used; per-guardrail `false` forces system messages to be included even when the global flag is `true`.

**Via LiteLLM UI** — when **creating** or **editing** a guardrail in the LiteLLM Admin Dashboard, set **Skip system messages in guardrail** (under Basic Info on create, or in the edit / guardrail settings flows):


| UI option | Effect |
| ------------------------------------- | -------------------------------------------------------------------------------------- |
| **Use global default** | Uses `litellm_settings.skip_system_message_in_guardrail` from your proxy config |
| **Yes — exclude from guardrail scan** | Sets per-guardrail `skip_system_message_in_guardrail: true` |
| **No — always include in scan** | Sets per-guardrail `skip_system_message_in_guardrail: false` (overrides a global skip) |


<Image
img={require('../../../img/skip_system_message_guardrail_ui.png')}
alt="Create guardrail: Skip system messages in guardrail dropdown with Use global default, Yes exclude from guardrail scan, and No always include in scan"
style={{ width: '100%', maxWidth: '900px', height: 'auto' }}
/>

**Where this applies:** Only the **unified** guardrail path (providers that implement `apply_guardrail` and run through LiteLLM’s message translation layer) on **OpenAI Chat Completions** (`/v1/chat/completions`) and **Anthropic Messages** (`/v1/messages`). Examples include Presidio, Bedrock guardrails, `litellm_content_filter`, OpenAI Moderation, Generic Guardrail API, and custom code guardrails that define `apply_guardrail`.

**Where this does *not* apply:** Guardrails that run only via direct hooks on the raw request (e.g. Lakera v2, Aporia, DynamoAI, Javelin, Lasso, Pangea, Model Armor, Azure Content Safety hooks, Guardrails AI, AIM, tool permission, MCP security). It also does not apply to other routes until those endpoints use the same translation layer (e.g. Responses API, embeddings, speech).

### Load Balancing Guardrails

Need to distribute guardrail requests across multiple accounts or regions? See [Guardrail Load Balancing](./guardrail_load_balancing.md) for details on:

- Load balancing across multiple AWS Bedrock accounts (useful for rate limit management)
- Weighted distribution across guardrail instances
- Multi-region guardrail deployments


## 2. Start LiteLLM Gateway

## 2. Start LiteLLM Gateway

```shell
litellm --config config.yaml --detailed_debug
```

## 3. Test request
## 3. Test request

**[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)**

<Tabs>
<TabItem label="Unsuccessful call" value = "not-allowed">


Expect this to fail since since `ishaan@berri.ai` in the request is PII

Expand Down Expand Up @@ -141,9 +173,9 @@ Expected response on failure

```

</TabItem>

<TabItem label="Successful Call " value = "allowed">



```shell
curl -i http://localhost:4000/v1/chat/completions \
Expand All @@ -158,10 +190,8 @@ curl -i http://localhost:4000/v1/chat/completions \
}'
```

</TabItem>


</Tabs>


## **Default On Guardrails**
Expand All @@ -183,7 +213,6 @@ guardrails:

In this request, the guardrail `aporia-pre-guard` will run on every request because `default_on: true` is set.


```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
Expand All @@ -207,6 +236,7 @@ x-litellm-applied-guardrails: aporia-pre-guard
### Guardrail Policies

Need more control? Use [Guardrail Policies](./guardrail_policies.md) to:

- Group guardrails into reusable policies
- Enable/disable guardrails for specific teams, keys, or models
- Inherit from existing policies and override specific guardrails
Expand All @@ -217,7 +247,6 @@ Need more control? Use [Guardrail Policies](./guardrail_policies.md) to:

Pass `guardrails` to your request body to test it


```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
Expand All @@ -239,7 +268,6 @@ Follow this simple workflow to implement and tune guardrails:

First, check what guardrails are available and their parameters:


Call `/guardrails/list` to view available guardrails and the guardrail info (supported parameters, description, etc)

```shell
Expand Down Expand Up @@ -271,9 +299,12 @@ Expected response
}
```

>


This config will return the `/guardrails/list` response above. The `guardrail_info` field is optional and you can add any fields under info for consumers of your guardrail
>



```yaml
- guardrail_name: "aporia-post-guard"
litellm_params:
Expand All @@ -291,9 +322,10 @@ This config will return the `/guardrails/list` response above. The `guardrail_in
type: "boolean"
```


### 2. Apply Guardrails

Add selected guardrails to your chat completion request:

```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
Expand Down Expand Up @@ -322,7 +354,6 @@ curl -i http://localhost:4000/v1/chat/completions \
}'
```


### 4. ✨ Pass Dynamic Parameters to Guardrail

:::info
Expand All @@ -334,9 +365,8 @@ curl -i http://localhost:4000/v1/chat/completions \
Use this to pass additional parameters to the guardrail API call. e.g. things like success threshold. **[See `guardrails` spec for more details](#spec-guardrails-parameter)**


<Tabs>

<TabItem value="openai" label="OpenAI Python v1.0.0+">


Set `guardrails={"aporia-pre-guard": {"extra_body": {"success_threshold": 0.9}}}` to pass additional parameters to the guardrail

Expand Down Expand Up @@ -371,10 +401,10 @@ response = client.chat.completions.create(

print(response)
```
</TabItem>


<TabItem value="Curl" label="Curl Request">



```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
Expand All @@ -396,10 +426,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}
}'
```
</TabItem>


</Tabs>



Expand All @@ -426,9 +453,6 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g

<Image img={require('../../../img/gd_fail.png')} />




### ✨ Control Guardrails per API Key

:::info
Expand All @@ -438,12 +462,12 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g
:::

Use this to control what guardrails run per API Key. In this tutorial we only want the following guardrails to run for 1 API Key

- `guardrails`: ["aporia-pre-guard", "aporia-post-guard"]

**Step 1** Create Key with guardrail settings

<Tabs>
<TabItem value="/key/generate" label="/key/generate">


```shell
curl -X POST 'http://0.0.0.0:4000/key/generate' \
Expand All @@ -454,8 +478,7 @@ curl -X POST 'http://0.0.0.0:4000/key/generate' \
}'
```

</TabItem>
<TabItem value="/key/update" label="/key/update">


```shell
curl --location 'http://0.0.0.0:4000/key/update' \
Expand All @@ -467,8 +490,7 @@ curl --location 'http://0.0.0.0:4000/key/update' \
}'
```

</TabItem>
</Tabs>


**Step 2** Test it with new key

Expand Down Expand Up @@ -499,8 +521,7 @@ Run guardrails based on the user-agent header. This is useful for running pre-ca

Both `default` and tag values can be a single mode string or a list of modes.

<Tabs>
<TabItem value="single" label="Single Default Mode">


```yaml
model_list:
Expand All @@ -522,11 +543,10 @@ guardrails:
default_on: true # run on every request
```

</TabItem>
<TabItem value="multi" label="Multiple Default Modes">


```yaml
model_list:
Per guardrailmodel_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
Expand All @@ -545,8 +565,7 @@ guardrails:
default_on: true
```

</TabItem>
<TabItem value="tag-list" label="Multiple Tag Modes">


```yaml
model_list:
Expand All @@ -568,8 +587,6 @@ guardrails:
default_on: true
```

</TabItem>
</Tabs>


### ✨ Model-level Guardrails
Expand All @@ -580,10 +597,8 @@ guardrails:

:::


This is great for cases when you have an on-prem and hosted model, and just want to run prevent sending PII to the hosted model.


```yaml
model_list:
- model_name: claude-sonnet-4
Expand Down Expand Up @@ -620,8 +635,7 @@ guardrails:

:::


#### 1. Disable team from modifying guardrails
#### 1. Disable team from modifying guardrails

```bash
curl -X POST 'http://0.0.0.0:4000/team/update' \
Expand All @@ -633,7 +647,7 @@ curl -X POST 'http://0.0.0.0:4000/team/update' \
}'
```

#### 2. Try to disable guardrails for a call
#### 2. Try to disable guardrails for a call

```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
Expand Down Expand Up @@ -672,8 +686,7 @@ Expect to NOT see `+1 412-612-9992` in your server logs on your callback.
The `pii_masking` guardrail ran on this request because api key=sk-jNm1Zar7XfNdZXp49Z1kSQ has `"permissions": {"pii_masking": true}`
:::


## Specification
## Specification

### `guardrails` Configuration on YAML

Expand Down Expand Up @@ -723,16 +736,18 @@ The `guardrails` parameter can be passed to any LiteLLM Proxy endpoint (`/chat/c
#### Format Options

1. Simple List Format:

```python
"guardrails": [
"aporia-pre-guard",
"aporia-post-guard"
]
```

2. Advanced Dictionary Format:
1. Advanced Dictionary Format:

In this format the dictionary key is `guardrail_name` you want to run

```python
"guardrails": {
"aporia-pre-guard": {
Expand All @@ -745,6 +760,7 @@ In this format the dictionary key is `guardrail_name` you want to run
```

#### Type Definition

```python
guardrails: Union[
List[str], # Simple list of guardrail names
Expand All @@ -754,3 +770,4 @@ guardrails: Union[
class DynamicGuardrailParams:
extra_body: Dict[str, Any] # Additional parameters for the guardrail
```

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions litellm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@
langfuse_default_tags: Optional[List[str]] = None
langsmith_batch_size: Optional[int] = None
prometheus_initialize_budget_metrics: Optional[bool] = False
prometheus_latency_buckets: Optional[List[float]] = None
require_auth_for_metrics_endpoint: Optional[bool] = False
argilla_batch_size: Optional[int] = None
datadog_use_v1: Optional[bool] = False # if you want to use v1 datadog logged payload.
Expand Down Expand Up @@ -203,6 +204,7 @@
bool
] = None # adds user_id, team_id, token hash (params from StandardLoggingMetadata) to request headers
store_audit_logs = False # Enterprise feature, allow users to see audit logs
skip_system_message_in_guardrail: bool = False
### end of callbacks #############

email: Optional[
Expand Down
Loading
Loading