feat(router): tag routing denylist support via ! prefix - #31680
feat(router): tag routing denylist support via ! prefix#31680deepanshululla wants to merge 17 commits into
Conversation
Requests can now exclude deployments by prefixing any tag with `!`. `!provider:anthropic` removes all Anthropic deployments before routing; negated tags are treated as regex so `!provider:(anthropic|openai)` works in one tag. Positive tags and negation compose: `paid,!provider:anthropic` first strips Anthropic, then applies the `paid` inclusion filter. Invalid regex patterns are logged and skipped rather than erroring the request.
Verifies that negated tags propagate correctly across fallback hops: primary group fully banned -> fallback fires and succeeds; all groups banned -> error raised after the chain is exhausted.
|
Deepanshu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Curl verification (proxy running on localhost:4000 with mock_response)Config 1: three providers tagged by provider namemodel_list:
- model_name: chat
litellm_params:
model: openai/gpt-4o-mini
api_key: fake-key
mock_response: "hello from openai"
tags: ["provider:openai", "family:gpt-4o"]
- model_name: chat
litellm_params:
model: anthropic/claude-haiku-4-5-20251001
api_key: fake-key
mock_response: "hello from anthropic"
tags: ["provider:anthropic", "family:claude-haiku"]
- model_name: chat
litellm_params:
model: openai/gpt-4o
api_key: fake-key
mock_response: "hello from vertex"
tags: ["provider:vertex", "inference:vertex"]
router_settings:
enable_tag_filtering: trueTest 1: exclude anthropic, remainder (openai + vertex) are both valid Test 2: regex alternation excludes both anthropic and openai, only vertex remains Test 3: backward compat - no tags, all deployments available Test 4: exclude all providers -> error Config 2: fallback chainsmodel_list:
- model_name: primary-chat
litellm_params: {model: anthropic/..., tags: ["provider:anthropic"], mock_response: "hello from anthropic (primary)"}
- model_name: fallback-chat
litellm_params: {model: openai/..., tags: ["provider:openai"], mock_response: "hello from openai (fallback)"}
- model_name: all-banned-primary
litellm_params: {model: anthropic/..., tags: ["provider:anthropic"], mock_response: "hello from anthropic (all-banned primary)"}
- model_name: all-banned-fallback
litellm_params: {model: anthropic/..., tags: ["provider:anthropic"], mock_response: "hello from anthropic (all-banned fallback)"}
router_settings:
enable_tag_filtering: true
fallbacks:
- {"primary-chat": ["fallback-chat"]}
- {"all-banned-primary": ["all-banned-fallback"]}Test 5: ban anthropic on primary-chat -> fallback fires to openai Test 6: ban anthropic on all-banned-primary (both primary and fallback are anthropic) -> exhausted |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR adds denylist-style tag routing with
Confidence Score: 4/5The change is mostly isolated to tag-based routing, but literal bang-prefixed tags need attention before merge. Focused coverage exists for the new denylist behavior, and the remaining concern is a concrete compatibility regression in existing exact-match tag routing semantics. litellm/router_strategy/tag_based_routing.py
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "style: ruff format and remove unused imp..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| positive = [t for t in tags if not t.startswith("!")] | ||
| excluded_patterns = [ | ||
| p for tag in tags if tag.startswith("!") if (p := _compile_negation_pattern(tag[1:])) is not None |
There was a problem hiding this comment.
_split_tags now treats every request tag starting with ! as a negation, so an existing deployment/request pair using a literal tag like !paid stops matching under the existing enable_tag_filtering setting. This changes current routing behavior without a user-controlled migration path; reserve negation behind an explicit opt-in or escaping scheme so current !-prefixed tags keep their old exact-match behavior.
Rule Used: What: avoid backwards-incompatible changes without... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Artifacts
Repro: focused helper-level script for literal bang tag routing
- Contains supporting evidence from the run (text/x-python; charset=utf-8).
- Keeps the command output available without making the summary code-heavy.
There was a problem hiding this comment.
Fixed in 7d2eb24. Negation patterns are now treated as exact literal strings; the implementation uses frozenset intersection instead of regex. A bare ! with nothing after it is skipped. This also means a legacy deployment tagged !paid is unaffected — the exclusion only fires when a request tag exactly matches a deployment tag after stripping the ! prefix.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewThis PR adds tag-based routing denylist support to the router using a There is still one open security issue in the tag-routing behavior: ban-only tag requests can expand routing beyond the default/untagged pool and reach deployments the caller did not positively select or appear entitled to use. One previously reported issue has already been addressed, but this remaining authorization/routing bypass keeps the PR at a high current risk until deny-only requests are constrained appropriately. Open issues (1)
Fixed/addressed: 1 · PR risk: 7/10 |
Greptile SummaryThis PR adds negated tag routing for router deployment selection. The main changes are:
Confidence Score: 4/5Merge should wait for the request-controlled regex behavior in tag negation to be bounded or made literal. The change is well scoped and has meaningful tests, but runtime evidence confirms that crafted negation tags can stall routing work. litellm/router_strategy/tag_based_routing.py
T-Rex LogsWhat T-Rex did
Reviews (2): Last reviewed commit: "style: ruff format and remove unused imp..." | Re-trigger Greptile |
| def _compile_negation_pattern(pattern_str: str) -> Optional[re.Pattern[str]]: | ||
| try: | ||
| return re.compile(pattern_str) |
There was a problem hiding this comment.
pattern_str comes from request metadata tags, so re.compile(pattern_str) lets callers send expensive patterns that run against every deployment tag in the router path. A tag with nested quantifiers can monopolize CPU during routing; keep negation literal, use a safe pattern subset, or use matching with a timeout.
Artifacts
Repro: focused runtime harness for tag negation regex slowdown
- Contains supporting evidence from the run (text/x-python; charset=utf-8).
Stack trace captured during the T-Rex run
- Keeps the raw stack trace available without making the summary code-heavy.
There was a problem hiding this comment.
Fixed in 7d2eb24 — same fix as the veria-ai comment above. Exact literal matching via frozenset, no regex.
… of regex Negation tags come from client-supplied request metadata, so compiling them as regex patterns opens a ReDoS vector. Exact tag membership testing (frozenset intersection) is O(1), safe, and sufficient for the primary use case: !provider:anthropic excludes any deployment whose tag list contains that exact string. Replacing the regex alternation test with two separate literal negation tags covers the same multi-provider exclusion scenario without regex. Adding test_negation_literal_only_no_partial_match to document that !provider:anthropic does NOT match provider:anthropic-haiku.
Break two long f-strings in tag_based_routing.py to stay under the 120-char limit. Remove redundant local import and unused response assignment in the test file. Update tag_routing.md to include full existing live-page content plus the new negation section.
…ython files" This reverts commit a2ad5f6.
| ban_only = bool(excluded_set) and not has_tag_filter | ||
|
|
||
| if ban_only: | ||
| return _require_candidates(candidates, model, request_tags) |
There was a problem hiding this comment.
High: Tag routing bypass
An authenticated caller can send a ban-only tag such as metadata.tags: ["!default"] or x-litellm-tags: !default and this branch returns all non-banned candidates instead of preserving the untagged/default routing behavior. In a config with a default deployment plus tagged team/paid/provider deployments, that lets a caller with no positive entitlement tag route traffic to deployments outside the untagged pool; apply the denylist to the default pool for ban-only requests, or require at least one positive tag before expanding beyond defaults.
There was a problem hiding this comment.
Fixed in 36ffa2a. Added _ban_only_base_pool helper that mirrors untagged-request semantics: if any deployment carries the default tag, ban-only requests are restricted to that pool before the exclusion set is applied; otherwise all healthy deployments are the base. This means !default on a config with a default + paid deployment raises no_deployments_with_tag_routing instead of routing to the paid deployment. Two regression tests added.
Without an upstream authentication layer, any client can spoof User-Agent and be routed to unintended deployments.
A caller sending only !-prefixed tags (e.g. !default) was routed to all non-banned candidates, allowing them to reach tagged deployments outside the untagged/default pool. Ban-only requests now use the same base pool as untagged requests: default-tagged deployments if any exist, otherwise all healthy deployments. Adds _ban_only_base_pool helper and two regression tests.
…erals !provider:(anthropic|openai) must not exclude deployments tagged provider:anthropic or provider:openai; the frozenset intersection only matches exact strings, so both models must remain reachable.
|
Correction to Test 2 in the verification above: To exclude multiple providers, send separate negation tags: Added a regression test |
Two cases: (1) negation removes a plain-tagged deployment and the surviving tag_regex deployment is still matched by User-Agent; (2) negation removes the only tag_regex deployment, leaving no regex candidates, so has_tag_filter=False, ban_only fires, and the remaining plain-tagged deployment is returned.
Adds `!` prefix negation to tag-based routing so callers can exclude deployments by exact tag value without enumerating every allowed alternative. `!provider:anthropic` removes all deployments tagged exactly `provider:anthropic` before routing, and positive and negation tags compose. Matching is exact literal membership (frozenset intersection), so there is no regex or ReDoS surface for client-supplied tags. Ban-only requests that carry only negation tags stay within the default pool, mirroring untagged-request semantics so callers can't use negation to escape it. Fallback chains keep working because get_deployments_for_tag runs on each routing hop Copy of #31680; implementation credit to @deepanshululla Co-authored-by: deepanshululla <15312873+deepanshululla@users.noreply.github.com>
|
this has been merged via #31728. Thank you for the contribution! |
Adds `!` prefix negation to tag-based routing so callers can exclude deployments by exact tag value without enumerating every allowed alternative. `!provider:anthropic` removes all deployments tagged exactly `provider:anthropic` before routing, and positive and negation tags compose. Matching is exact literal membership (frozenset intersection), so there is no regex or ReDoS surface for client-supplied tags. Ban-only requests that carry only negation tags stay within the default pool, mirroring untagged-request semantics so callers can't use negation to escape it. Fallback chains keep working because get_deployments_for_tag runs on each routing hop Copy of BerriAI#31680; implementation credit to @deepanshululla Co-authored-by: deepanshululla <15312873+deepanshululla@users.noreply.github.com>
3 participants
Relevant issues
Fixes #31676
Docs: BerriAI/litellm-docs#437
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Run the proxy with a config that tags deployments by provider, then send requests with negation tags:
Exclude Anthropic, remaining pool (openai + vertex) is used:
Exclude both anthropic and openai with two negation tags, only vertex remains:
Exclude all providers -> error:
Fallback chain fires when primary group is banned (config with fallbacks defined):
Type
New Feature
Changes
Adds
!prefix negation tox-litellm-tagsfor tag-based routing. Callers can now exclude deployments by exact tag value without needing to enumerate every allowed alternative.Changes in
litellm/router_strategy/tag_based_routing.py:_split_tags(tags): partitions the request tag list into positive tags (unchanged, passed to existing inclusion logic) and excluded literals (strings after!prefix). A bare!with no suffix is ignored._exclude_deployments(deployments, excluded_set): new helper; filters deployments whosetagsintersect the excluded set. Called before tag or regex matching so negation always runs first._ban_only_base_pool(deployments): new helper; returns the default-tagged pool when one exists, otherwise all deployments. Used by the ban-only path to mirror untagged-request semantics and prevent callers from using negation tags to escape the default pool.get_deployments_for_tag: builds afrozensetof excluded literals and calls_exclude_deploymentsbefore any inclusion logic. When only negation tags are present (ban-only), routes within the default pool if one exists. Raisesno_deployments_with_tag_routingwhen the exclusion filter empties the pool.Matching is exact and literal (frozenset intersection). No regex engine is involved for client-supplied tags, so there is no ReDoS risk. A tag like
!provider:(anthropic|openai)only excludes a deployment tagged exactlyprovider:(anthropic|openai)— it does not act as a regex alternation. To exclude multiple providers, send separate tags:["!provider:anthropic", "!provider:openai"]. Operator-configuredtag_regexin deployment config is unaffected.Interaction with
tag_regex: negation exclusion runs before regex matching. If a deployment carries both a plaintagslist andtag_regex, and its plain tag is negated, it is excluded before regex matching runs. If that deployment was the only one withtag_regex,has_tag_filterbecomesFalseand the ban-only path fires instead.Fallback chains work automatically:
get_deployments_for_tagis called on each routing hop, so a banned primary group triggers the existing fallback mechanism without any extra wiring.18 new unit and integration tests cover:
_split_tagsedge cases (including bare!skip), single and multiple literal negation tags, mixed positive+negation, literal-only semantics (no partial match, no regex interpretation), ban-only exhausting all candidates, ban-only confined to the default pool (routing bypass regression), untagged deployments being kept, fallback chain firing when primary is banned, the full fallback chain exhausting with an error, and twotag_regex+ negation interaction cases.