diff --git a/docs/proxy/tag_routing.md b/docs/proxy/tag_routing.md index 57d16a59b..5d9454c0a 100644 --- a/docs/proxy/tag_routing.md +++ b/docs/proxy/tag_routing.md @@ -1,24 +1,13 @@ -# Tag Based Routing - -Route requests based on tags. -This is useful for -- Implementing free / paid tiers for users -- Controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B (LLM Access Control For Teams ) +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; -:::info -## See here for spend tags -- [Track spend per tag](cost_tracking#-custom-tags) -- [Setup Budgets per Virtual Key, Team](users) -::: +# Tag Based Routing ## Quick Start -### 1. Define tags on config.yaml - -- A request with `tags=["free"]` will get routed to `openai/fake` -- A request with `tags=["paid"]` will get routed to `openai/gpt-4o` +### 1. Define tags on config.yaml -```yaml +```yaml showLineNumbers title="config.yaml" model_list: - model_name: gpt-4 litellm_params: @@ -37,19 +26,17 @@ model_list: api_key: os.environ/OPENAI_API_KEY api_base: https://exampleopenaiendpoint-production.up.railway.app/ tags: ["default"] # OPTIONAL - All untagged requests will get routed to this - router_settings: enable_tag_filtering: True # 👈 Key Change -general_settings: - master_key: sk-1234 + +general_settings: + master_key: sk-1234 ``` ### 2. Make Request with `tags=["free"]` -This request includes "tags": ["free"], which routes it to `openai/fake` - -```shell +```bash curl -i http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-1234" \ @@ -61,47 +48,31 @@ curl -i http://localhost:4000/v1/chat/completions \ "tags": ["free"] }' ``` -**Expected Response** -Expect to see the following response header when this works -```shell -x-litellm-model-api-base: https://exampleopenaiendpoint-production.up.railway.app/ -``` +**Response:** -Response -```shell +```json { - "id": "chatcmpl-33c534e3d70148218e2d62496b81270b", - "choices": [ - { - "finish_reason": "stop", - "index": 0, - "message": { - "content": "\n\nHello there, how may I assist you today?", - "role": "assistant", - "tool_calls": null, - "function_call": null - } - } - ], - "created": 1677652288, - "model": "gpt-3.5-turbo-0125", - "object": "chat.completion", - "system_fingerprint": "fp_44709d6fcb", - "usage": { - "completion_tokens": 12, - "prompt_tokens": 9, - "total_tokens": 21 - } + "id": "chatcmpl-33c534e3d70148218e2d62496b81270b", + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "\n\nHello there, how may I assist you today?", + "role": "assistant" + } + } + ], + "model": "gpt-3.5-turbo-0125", + "object": "chat.completion", + "usage": {"completion_tokens": 12, "prompt_tokens": 9, "total_tokens": 21} } ``` - ### 3. Make Request with `tags=["paid"]` -This request includes "tags": ["paid"], which routes it to `openai/gpt-4` - -```shell +```bash curl -i http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-1234" \ @@ -114,46 +85,30 @@ curl -i http://localhost:4000/v1/chat/completions \ }' ``` -**Expected Response** +**Response:** -Expect to see the following response header when this works -```shell -x-litellm-model-api-base: https://api.openai.com -``` - -Response -```shell +```json { - "id": "chatcmpl-9maCcqQYTqdJrtvfakIawMOIUbEZx", - "choices": [ - { - "finish_reason": "stop", - "index": 0, - "message": { - "content": "Good morning! How can I assist you today?", - "role": "assistant", - "tool_calls": null, - "function_call": null - } - } - ], - "created": 1721365934, - "model": "gpt-4o-2024-05-13", - "object": "chat.completion", - "system_fingerprint": "fp_c4e5b6fa31", - "usage": { - "completion_tokens": 10, - "prompt_tokens": 12, - "total_tokens": 22 - } + "id": "chatcmpl-9maCcqQYTqdJrtvfakIawMOIUbEZx", + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "Good morning! How can I assist you today?", + "role": "assistant" + } + } + ], + "model": "gpt-4o-2024-05-13", + "object": "chat.completion", + "usage": {"completion_tokens": 10, "prompt_tokens": 12, "total_tokens": 22} } ``` ## Calling via Request Header -You can also call via request header `x-litellm-tags` - -```shell +```bash curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' \ @@ -163,36 +118,37 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ "messages": [ { "role": "user", - "content": "Hey, how'\''s it going 123456?" + "content": "Hey, how'\''s it going?" } ] }' ``` -## Setting Default Tags +## Setting Default Tags -Use this if you want all untagged requests to be routed to specific deployments +### 1. Set default tag on yaml -1. Set default tag on your yaml -```yaml - model_list: - - model_name: fake-openai-endpoint - litellm_params: - model: openai/fake - api_key: fake-key - api_base: https://exampleopenaiendpoint-production.up.railway.app/ - tags: ["default"] # 👈 Key Change - All untagged requests will get routed to this - model_info: - id: "default-model" # used for identifying model in response headers +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["default"] # 👈 Key Change - All untagged requests will get routed to this + model_info: + id: "default-model" ``` -2. Start proxy -```shell +### 2. Start proxy + +```bash $ litellm --config /path/to/config.yaml ``` -3. Make request with no tags -```shell +### 3. Make request with no tags + +```bash curl -i http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-1234" \ @@ -204,22 +160,149 @@ curl -i http://localhost:4000/v1/chat/completions \ }' ``` -Expect to see the following response header when this works -```shell -x-litellm-model-id: default-model +## Negation Tags (Denylist) + +Prefix any tag with `!` to **exclude** deployments that carry that exact tag. This is useful when you want to avoid a specific provider or model family without listing every allowed alternative. + +### Quick example + +```bash +curl http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "gpt-4", + "messages": [{"role": "user", "content": "Hello"}], + "metadata": {"tags": ["!provider:anthropic"]} + }' +``` + +Any deployment tagged `provider:anthropic` is removed from the candidate pool before routing. All remaining deployments are eligible. + +### Config example + +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: chat + litellm_params: + model: anthropic/claude-haiku-4-5-20251001 + api_key: os.environ/ANTHROPIC_API_KEY + tags: ["provider:anthropic"] + + - model_name: chat + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + tags: ["provider:openai"] + + - model_name: chat + litellm_params: + model: vertex_ai/gemini-2.0-flash + api_key: os.environ/VERTEX_API_KEY + tags: ["provider:vertex"] + +router_settings: + enable_tag_filtering: true + +general_settings: + master_key: sk-1234 +``` + +### Combining positive and negation tags + +Use positive tags to select a tier and negation tags to exclude a provider within that tier: + +```bash +curl http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "chat", + "messages": [{"role": "user", "content": "Hello"}], + "metadata": {"tags": ["paid", "!provider:anthropic"]} + }' +``` + +### Excluding multiple providers + +Send multiple `!` tags to exclude more than one deployment group: + +```bash +curl http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "chat", + "messages": [{"role": "user", "content": "Hello"}], + "metadata": {"tags": ["!provider:anthropic", "!provider:openai"]} + }' +``` + +Only the vertex deployment remains eligible. + +### Negation with fallback chains + +When the primary model group is banned, the router falls through to the configured fallback automatically: + +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: primary + litellm_params: + model: anthropic/claude-haiku-4-5-20251001 + api_key: os.environ/ANTHROPIC_API_KEY + tags: ["provider:anthropic"] + + - model_name: fallback + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + tags: ["provider:openai"] + +router_settings: + enable_tag_filtering: true + fallbacks: + - {"primary": ["fallback"]} + +general_settings: + master_key: sk-1234 +``` + +```bash +curl http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "primary", + "messages": [{"role": "user", "content": "Hello"}], + "metadata": {"tags": ["!provider:anthropic"]} + }' +# primary is banned -> falls through to fallback (provider:openai) ``` +### Negation semantics + +| Behavior | Detail | +|----------|--------| +| Matching | Exact tag string match. `!provider:anthropic` removes only deployments tagged exactly `provider:anthropic` | +| No regex | Negation tags are plain strings, not regex patterns. `!provider:(anthropic\|openai)` only excludes a deployment tagged exactly `provider:(anthropic\|openai)`. To exclude multiple providers send separate tags: `["!provider:anthropic", "!provider:openai"]`. Note: `tag_regex` in deployment config is regex, but that is operator-configured and unrelated to client-supplied negation tags | +| Ban-only request | If the request carries only `!` tags and no positive tags, the base pool mirrors untagged-request behaviour: default-tagged deployments if any exist, otherwise all deployments. The exclusion set is then applied on top of that pool | +| All excluded | If negation tags remove every candidate, the request fails with `no_deployments_with_tag_routing` | +| Untagged deployments | Deployments with no `tags` field are never excluded by negation tags | +| Header | Negation tags work via `x-litellm-tags` header too: `-H 'x-litellm-tags: !provider:anthropic'` | + ## Regex-based tag routing (`tag_regex`) -Use `tag_regex` to route requests based on regex patterns matched against request headers, without requiring clients to pass a tag explicitly. This is useful when clients already send a recognisable header, such as `User-Agent`. +Use `tag_regex` on a deployment to match incoming requests by their headers (e.g. `User-Agent`) — without requiring the client to send explicit tags. Patterns are operator-configured and compiled server-side, not supplied by callers. -**Use case: route all Claude Code traffic to dedicated AWS accounts** +:::caution +User-Agent is a client-supplied header and can be set to any value by any caller. Use `tag_regex` for traffic classification, not access-control enforcement. -Claude Code always sends `User-Agent: claude-code/`. With `tag_regex` you can route that traffic to a dedicated deployment automatically — no per-developer configuration needed. +Header-based routing is not a security boundary on its own. It is only meaningful when requests pass through an upstream authentication layer (e.g., an API gateway or reverse proxy that validates credentials and rejects unauthenticated traffic before it reaches LiteLLM). Without such a layer, any client can spoof the User-Agent and be routed to a deployment it should not reach. +::: ### 1. Config -```yaml +```yaml showLineNumbers title="config.yaml" model_list: # Claude Code traffic → dedicated deployment, matched by User-Agent - model_name: claude-sonnet @@ -231,7 +314,6 @@ model_list: - "^User-Agent: claude-code\\/" # matches claude-code/1.x, 2.x, etc. model_info: id: claude-code-deployment - # All other traffic falls back to the default deployment - model_name: claude-sonnet litellm_params: @@ -253,37 +335,88 @@ general_settings: ### 2. Verify routing -Claude Code sets `User-Agent: claude-code/` automatically — no client config needed: - -```shell +```bash # Claude Code request (User-Agent set automatically by Claude Code) curl http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-1234" \ -H "User-Agent: claude-code/1.2.3" \ -d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}' -# → x-litellm-model-id: claude-code-deployment +# -> x-litellm-model-id: claude-code-deployment -# Any other client (no matching User-Agent) → default deployment +# Any other client (no matching User-Agent) -> default deployment curl http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-1234" \ -d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}' -# → x-litellm-model-id: regular-deployment +# -> x-litellm-model-id: regular-deployment ``` -### How matching works +### Matching semantics -| Priority | Condition | Result | -|----------|-----------|--------| -| 1 | Request has `tags` AND deployment has `tags` | Exact tag match (respects `match_any` setting) | -| 2 | Deployment has `tag_regex` AND request has a `User-Agent` | Regex match (always OR logic — any pattern match suffices) | -| 3 | Deployment has `tags: [default]` | Default fallback | -| 4 | No default set | All healthy deployments returned | +| Behavior | Detail | +|----------|--------| +| Engine | Python `re.search` — patterns do not need to be anchored unless you want to pin to the start (`^`) or end (`$`) of the string | +| Input format | Patterns are matched against `"Header-Name: value"` strings. Currently only `User-Agent` is exposed: `User-Agent: claude-code/1.2.3` | +| Logic | Always OR — any single pattern matching is enough to select the deployment. `tag_filtering_match_any=False` applies only to plain `tags`, not to `tag_regex` | +| Invalid patterns | A pattern that fails `re.compile` is logged and skipped; it never causes a hard error | +| Interaction with plain tags | When a deployment has both `tags` and `tag_regex`, and `tag_filtering_match_any=False`, the regex path is blocked if the strict tag check already failed. Regex cannot override a strict-tag policy | +| Trusted input | Patterns are set by the operator in config, never supplied by the caller. This is the key difference from negation tags (`!foo` in request metadata), which are always treated as plain literals | -`tag_regex` always uses OR semantics — `tag_filtering_match_any=False` applies only to exact tag matching, not to regex patterns. +### Interaction with negation tags -### Observability +Negation exclusion runs before `tag_regex` matching. The order matters when a deployment carries both a plain `tags` list and `tag_regex`: + +1. The router removes any deployment whose `tags` intersect the request's excluded set. +2. `tag_regex` matching runs only on the surviving candidates. + +**Case 1: negation removes a plain-tagged deployment; the `tag_regex` deployment is unaffected** -When a regex matches, `tag_routing` is written into request metadata and flows to SpendLogs: +```yaml +model_list: + - model_name: chat + litellm_params: + tag_regex: ["^User-Agent: claude-code\\/"] # no plain tags + model_info: {id: claude-code-deployment} + + - model_name: chat + litellm_params: + tags: ["provider:anthropic"] + model_info: {id: anthropic-deployment} +``` + +```bash +curl ... -H "User-Agent: claude-code/1.2.3" \ + -d '{"model":"chat","metadata":{"tags":["!provider:anthropic"]}}' +# anthropic-deployment is excluded; claude-code-deployment is matched by User-Agent +# -> x-litellm-model-id: claude-code-deployment +``` + +**Case 2: negation removes the deployment that holds `tag_regex`; ban-only path fires** + +If the negated tag is on the same deployment as `tag_regex`, that deployment is excluded first. With no `tag_regex` deployments left in the candidate pool, `has_tag_filter` becomes `False`, the ban-only path fires, and the remaining deployments are returned directly. + +```yaml +model_list: + - model_name: chat + litellm_params: + tag_regex: ["^User-Agent: claude-code\\/"] + tags: ["group:claude"] # negation target is on the tag_regex deployment + model_info: {id: claude-code-deployment} + + - model_name: chat + litellm_params: + tags: ["provider:openai"] + model_info: {id: openai-deployment} +``` + +```bash +curl ... -H "User-Agent: claude-code/1.2.3" \ + -d '{"model":"chat","metadata":{"tags":["!group:claude"]}}' +# claude-code-deployment excluded; no tag_regex deployments remain +# ban-only path returns openai-deployment regardless of User-Agent +# -> x-litellm-model-id: openai-deployment +``` + +### Observability ```json { @@ -296,138 +429,84 @@ When a regex matches, `tag_routing` is written into request metadata and flows t } ``` -### Security note +## Team based tag routing (Enterprise) -:::caution - -**`User-Agent` is a client-supplied header and can be set to any value.** Any API consumer can send `User-Agent: claude-code/1.0` regardless of whether they are actually using Claude Code. - -Do not rely on `tag_regex` routing to enforce access controls or spend limits — use [team/key-based routing](./users) for that. `tag_regex` is a **traffic classification hint** (useful for billing visibility, capacity planning, and routing convenience), not a security boundary. +### Configuration -::: +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["teamA"] # 👈 Key Change + model_info: + id: "team-a-model" + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["teamB"] # 👈 Key Change + model_info: + id: "team-b-model" + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["default"] # OPTIONAL - All untagged requests will get routed to this +router_settings: + enable_tag_filtering: True # 👈 Key Change ---- +general_settings: + master_key: sk-1234 +``` -## ✨ Team based tag routing (Enterprise) +### Create teams with tags -LiteLLM Proxy supports team-based tag routing, allowing you to associate specific tags with teams and route requests accordingly. Example **Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B** (LLM Access Control For Teams) +```bash +# Create Team A +curl -X POST http://0.0.0.0:4000/team/new \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"tags": ["teamA"]}' -:::info +# Create Team B +curl -X POST http://0.0.0.0:4000/team/new \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"tags": ["teamB"]}' +``` -This is an enterprise feature, [Contact us here to get a free trial](https://enterprise.litellm.ai/demo) +### Generate keys for team members -::: +```bash +# Generate key for Team A +curl -X POST http://0.0.0.0:4000/key/generate \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"team_id": "team_a_id_here"}' -Here's how to set up and use team-based tag routing using curl commands: - -1. **Enable tag filtering in your proxy configuration:** - - In your `proxy_config.yaml`, ensure you have the following setting: - - ```yaml - model_list: - - model_name: fake-openai-endpoint - litellm_params: - model: openai/fake - api_key: fake-key - api_base: https://exampleopenaiendpoint-production.up.railway.app/ - tags: ["teamA"] # 👈 Key Change - model_info: - id: "team-a-model" # used for identifying model in response headers - - model_name: fake-openai-endpoint - litellm_params: - model: openai/fake - api_key: fake-key - api_base: https://exampleopenaiendpoint-production.up.railway.app/ - tags: ["teamB"] # 👈 Key Change - model_info: - id: "team-b-model" # used for identifying model in response headers - - model_name: fake-openai-endpoint - litellm_params: - model: openai/fake - api_key: fake-key - api_base: https://exampleopenaiendpoint-production.up.railway.app/ - tags: ["default"] # OPTIONAL - All untagged requests will get routed to this - - router_settings: - enable_tag_filtering: True # 👈 Key Change - - general_settings: - master_key: sk-1234 - ``` - -2. **Create teams with tags:** - - Use the `/team/new` endpoint to create teams with specific tags: - - ```shell - # Create Team A - curl -X POST http://0.0.0.0:4000/team/new \ - -H "Authorization: Bearer sk-1234" \ - -H "Content-Type: application/json" \ - -d '{"tags": ["teamA"]}' - ``` - - ```shell - # Create Team B - curl -X POST http://0.0.0.0:4000/team/new \ - -H "Authorization: Bearer sk-1234" \ - -H "Content-Type: application/json" \ - -d '{"tags": ["teamB"]}' - ``` - - These commands will return JSON responses containing the `team_id` for each team. - -3. **Generate keys for team members:** - - Use the `/key/generate` endpoint to create keys associated with specific teams: - - ```shell - # Generate key for Team A - curl -X POST http://0.0.0.0:4000/key/generate \ - -H "Authorization: Bearer sk-1234" \ - -H "Content-Type: application/json" \ - -d '{"team_id": "team_a_id_here"}' - ``` - - ```shell - # Generate key for Team B - curl -X POST http://0.0.0.0:4000/key/generate \ - -H "Authorization: Bearer sk-1234" \ - -H "Content-Type: application/json" \ - -d '{"team_id": "team_b_id_here"}' - ``` - - Replace `team_a_id_here` and `team_b_id_here` with the actual team IDs received from step 2. - -4. **Verify routing:** - - Check the `x-litellm-model-id` header in the response to confirm that the request was routed to the correct model based on the team's tags. You can use the `-i` flag with curl to include the response headers: - - Request with Team A's key (including headers) - ```shell - curl -i -X POST http://0.0.0.0:4000/chat/completions \ - -H "Authorization: Bearer team_a_key_here" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "fake-openai-endpoint", - "messages": [ - {"role": "user", "content": "Hello!"} - ] - }' - ``` - - In the response headers, you should see: - ``` - x-litellm-model-id: team-a-model - ``` - - Similarly, when using Team B's key, you should see: - ``` - x-litellm-model-id: team-b-model - ``` - -By following these steps and using these curl commands, you can implement and test team-based tag routing in your LiteLLM Proxy setup, ensuring that different teams are routed to the appropriate models or deployments based on their assigned tags. +# Generate key for Team B +curl -X POST http://0.0.0.0:4000/key/generate \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"team_id": "team_b_id_here"}' +``` +### Verify routing +```bash +curl -i -X POST http://0.0.0.0:4000/chat/completions \ + -H "Authorization: Bearer team_a_key_here" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "fake-openai-endpoint", + "messages": [ + {"role": "user", "content": "Hello!"} + ] + }' +```