feat(router): resolve auto-router routing plugins from proxy YAML config

[krrish-berri-2]

Relevant issues

Discussion: #32168

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

make pre-commit could not complete in the environment this PR was built in: uv sync fails building grpcio==1.78.0 (no wheel for macOS x86_64 at that pin) before it reaches basedpyright, and the dashboard node_modules aren't provisioned either, both unrelated to this diff. ruff check . passes clean against the changed files. CI runs in a correctly provisioned environment so its lint job is the authoritative check here.

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Commit under test: 8d5e033929

Router(plugins=[...]) (merged separately, discussion #32168) only accepted pre-instantiated plugin objects through the Python constructor, and narrowed candidates from the outer model alias rather than the auto-router's real tier pool, so it was unusable through proxy config and a no-op for auto_router/complexity_router deployments even from Python. This PR adds complexity_router_config.plugins, a list of dotted-path strings resolved the same way litellm_settings.callbacks already is, and wires the resolved plugins into the tier pool that get_model_for_tier actually picks from.

Config used below (complexity_router_config.tiers.COMPLEX: [gpt-4o, gpt-4o-mini], default_model: gpt-4o-mini), with a CostCeilingPlugin that drops any candidate whose real per-token input cost exceeds 0.000001 (gpt-4o-mini = 1.5e-07, gpt-4o = 2.5e-06, from litellm/model_prices_and_context_window_backup.json). All requests below are real openai/gpt-4o and openai/gpt-4o-mini calls against the live proxy, billed against a real OpenAI key; the routed model is read off x-litellm-response-cost since gpt-4o-mini's cost (9.45e-06) and gpt-4o's cost (0.0001575) are only reachable by one deployment each here.

Before, config.yaml (no plugins key in complexity_router_config):

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: ["gpt-4o-mini"]
          MEDIUM: ["gpt-4o-mini"]
          COMPLEX: ["gpt-4o", "gpt-4o-mini"]
          REASONING: ["gpt-4o", "gpt-4o-mini"]
        default_model: gpt-4o-mini
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      max_tokens: 5
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      max_tokens: 5

general_settings:
  master_key: sk-1234

8 identical COMPLEX-tier requests against that config:

$ for i in $(seq 1 8); do
  curl -s -i http://localhost:4114/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Design a distributed microservice architecture with Kubernetes orchestration, implementing proper authentication, encryption, and database optimization for high throughput. Think step by step about the performance implications and scalability requirements."}]}' \
  | grep -i "^x-litellm-response-cost:"
done

x-litellm-response-cost: 0.0001575
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 0.0001575
x-litellm-response-cost: 0.0001575
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 0.0001575
x-litellm-response-cost: 0.0001575

gpt-4o (over the would-be ceiling) gets picked 5 of 8 times, as expected from an unfiltered pool.

After, config.yaml (same config plus complexity_router_config.plugins):

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: ["gpt-4o-mini"]
          MEDIUM: ["gpt-4o-mini"]
          COMPLEX: ["gpt-4o", "gpt-4o-mini"]
          REASONING: ["gpt-4o", "gpt-4o-mini"]
        default_model: gpt-4o-mini
        plugins:
          - plugins.cost_ceiling_plugin.cost_ceiling_plugin
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      max_tokens: 5
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      max_tokens: 5

general_settings:
  master_key: sk-1234

plugins.cost_ceiling_plugin.cost_ceiling_plugin is a dotted path resolved relative to config.yaml's own directory, i.e. a sibling plugins/cost_ceiling_plugin.py file.

5 identical COMPLEX-tier requests against that config:

$ for i in $(seq 1 5); do
  curl -s -i http://localhost:4113/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Design a distributed microservice architecture with Kubernetes orchestration, implementing proper authentication, encryption, and database optimization for high throughput. Think step by step about the performance implications and scalability requirements."}]}' \
  | grep -i "^x-litellm-response-cost:"
done

x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 9.45e-06
x-litellm-response-cost: 9.45e-06

gpt-4o never gets picked once the plugin is configured; every request lands on gpt-4o-mini, the only candidate the plugin leaves in the tier pool.

plugins/cost_ceiling_plugin.py used for both runs:

class CostCeilingPlugin:
    def __init__(self, max_cost_per_token: float, cost_by_model: dict):
        self.max_cost_per_token = max_cost_per_token
        self.cost_by_model = cost_by_model

    async def run(self, context):
        context.candidate_models = [
            m for m in context.candidate_models if self.cost_by_model.get(m, 0.0) <= self.max_cost_per_token
        ]
        context.signals["cost-ceiling-plugin"] = {"max_cost_per_token": self.max_cost_per_token}
        return context


cost_ceiling_plugin = CostCeilingPlugin(
    max_cost_per_token=0.000001,
    cost_by_model={"gpt-4o-mini": 1.5e-07, "gpt-4o": 2.5e-06},
)

Type

🆕 New Feature
🐛 Bug Fix

Changes

complexity_router_config gains a plugins: list[str] | None field (dotted-path refs, e.g. mypackage.mymodule.my_plugin_instance). The proxy config loader resolves each path through the existing get_instance_fn (same helper litellm_settings.callbacks and custom guardrails already use) before Router.__init__ runs, so ComplexityRouterConfig.plugins ends up holding live RoutingPlugin instances.

ComplexityRouter._pick_model_for_tier builds a RoutingContext scoped to the classified tier's actual candidate pool (not the outer auto_router/complexity_router alias), runs the configured plugins in order, and picks from whatever candidates survive. It's wired into all three model-pick sites inside _classify_and_route (the weighted-scoring path, the keyword_tier_rules override path, and the no-user-message default-tier path) so a policy plugin can't be bypassed by any of those paths. A tier narrowed to zero candidates raises rather than falling back to default_model, since default_model was never checked against the plugins and would otherwise function as an unconditional escape hatch around whatever policy a plugin enforces. adaptive=True combined with plugins raises at config-validation time, since the bandit selector in _soft_floor_pick doesn't consume plugin-narrowed pools yet.

Also fixes Router._generate_model_id, which builds a deployment id by json.dumps-ing every litellm_params value; it crashed with TypeError: Object ... is not JSON serializable the moment a resolved plugin instance landed inside complexity_router_config. Uses json.dumps(v, default=...) with a fallback that returns the value's fully-qualified class name. (An earlier revision of this PR used default=str, which Greptile correctly flagged: str() on an arbitrary object falls back to object.__repr__'s <module.Class object at 0x...>, embedding the memory address and making the deployment id change on every hot-reload for any deployment with plugins configured. Fixed in a follow-up commit.)

Three policy-bypass gaps flagged by Veria AI's security review, all fixed in follow-up commits: the session_affinity cache-pin shortcut returned a session's first-turn model on every later turn without ever re-running it through the plugin pipeline, so a plugin's decision (e.g. a budget cap crossed mid-session) was only enforced on turn one -- the pin shortcut is now disabled whenever plugins are configured. The no-user-message path used self.config.default_model or await self._pick_model_for_tier(...), and Python's or short-circuits on a truthy default_model -- the plugin pipeline never ran at all whenever default_model was configured; that short-circuit is removed, falling through to the same tier-then-default_model resolution every other call site already uses. Separately, get_instance_fn accepts any dotted path and returns whatever object it finds there, so a misconfigured plugins entry passed proxy startup silently and only surfaced as a confusing AttributeError on the first request that reached the plugin pipeline -- extracted the resolution logic into resolve_complexity_router_plugins() with an isinstance(..., RoutingPlugin) check that now fails proxy startup immediately with a clear error instead.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR wires complexity_router_config.plugins — a new list of dotted-path strings — into the ComplexityRouter pipeline so that operator-defined routing plugins can be configured via proxy YAML rather than requiring pre-instantiated objects in Python code. Plugin paths are resolved at proxy startup through the existing get_instance_fn helper, with an isinstance + iscoroutinefunction guard that catches misconfigured entries before the first request.

• Plugin pipeline: _pick_model_for_tier builds a RoutingContext scoped to each tier's actual candidate pool and runs configured plugins in order; it is wired into all three routing decision sites (complexity scorer, keyword-tier override, no-user-message default). Zero candidates after plugin filtering raise rather than fall back to default_model, intentionally closing a policy-bypass escape hatch.
• Safety guards: session-affinity pin shortcut is disabled when plugins are configured (avoids stale pins bypassing per-turn policy decisions); adaptive=True + plugins raises at validation time since the bandit selector doesn't yet consume plugin-narrowed pools.
• Deployment-ID stability: _generate_model_id gains a _json_default_stable_id fallback that serializes plugin instances by fully-qualified class name instead of repr, preventing deployment-ID churn on hot-reload.

Confidence Score: 5/5

Safe to merge; all routing paths are covered by tests and the plugin pipeline is fail-closed by design.

The change is well-scoped: new behaviour only activates when plugins is explicitly configured (a new field), so no existing deployments are affected. The three routing paths are each covered by dedicated tests, including regression guards for the no-plugin population and the no-user-message path. The one outstanding issue is a code comment in _classify_and_route that incorrectly describes the default_model fallback behaviour when plugins are active, which could mislead future developers but does not affect runtime correctness.

litellm/router_strategy/complexity_router/complexity_router.py — the misleading comment in the no-user-message branch (lines 993-996) should be corrected before it becomes a maintenance trap.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Adds resolve_complexity_router_plugins() to resolve dotted-path plugin strings at proxy startup, with an isinstance + iscoroutinefunction guard that correctly catches both missing run and synchronous run implementations before the first request.
litellm/router.py	Adds _json_default_stable_id to produce a stable, memory-address-free serialization fallback for plugin instances in _generate_model_id, preventing deployment-ID churn on hot-reload.
litellm/router_strategy/complexity_router/complexity_router.py	Adds _pick_model_for_tier to run the plugin pipeline on the tier's candidate pool; wires it into all three routing paths. A comment in the no-user-message branch incorrectly states that get_model_for_tier still provides a default_model fallback when plugins are active, but _pick_model_for_tier bypasses get_model_for_tier entirely when plugins are configured.
litellm/router_strategy/complexity_router/config.py	Adds plugins field to ComplexityRouterConfig and a model validator that hard-rejects plugins + adaptive=True combination since the bandit selector doesn't yet consume narrowed pools.
litellm/types/router.py	Adds @runtime_checkable decorator to RoutingPlugin Protocol so that isinstance checks work at config-load time for the startup validation guard.
tests/test_litellm/router_strategy/test_complexity_router.py	Adds TestRoutingPlugins with nine scenario tests covering plugin narrowing, zero-candidate raise, metadata propagation, all three routing paths (complexity scorer, keyword override, no-user-message), session-affinity bypass disable, and a backwards-compat regression guard for the no-plugin path.
tests/test_litellm/proxy/proxy_server/test_proxy_config.py	Adds four unit tests for resolve_complexity_router_plugins: no-op on missing key, successful dotted-path resolution, rejection of non-RoutingPlugin objects, and rejection of a synchronous run method.
tests/test_litellm/router_strategy/test_router_routing_plugins.py	Adds two tests for _json_default_stable_id and _generate_model_id stability when litellm_params contain a plugin instance, ensuring the deployment ID doesn't change between process restarts.

_{Reviews (8): Last reviewed commit: "fix(router): address Greptile findings o..." | Re-trigger Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d5e033929

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fail closed when plugins deny every candidate

When a routing plugin removes every candidate, returning default_model bypasses the plugin's policy; for example, a tenant or budget plugin that intentionally blocks the whole tier will still send the request to whatever default is configured, even if that model was not allowed. The existing Router plugin path treats an empty filtered set as a policy denial and raises, so the complexity-router path should only route to a model that survived the plugin pipeline, or fail closed.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Run plugins before choosing the no-user default

When default_model is set, this branch returns it without invoking _pick_model_for_tier, so complexity_router_config.plugins are skipped for requests whose normalized messages contain no role="user" content. In deployments where plugins enforce tenant, provider, or budget policy, a system/assistant-only request can bypass those filters and still hit the default model; run the plugin pipeline against the fallback tier/default before returning.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Seed plugins with deployment models instead of tier aliases

This populates RoutingContext.candidate_models from the tier config entries, which are usually router model-group aliases, while the shared RoutingPlugin contract and existing policies operate on deployment litellm_params.model values such as openai/.... After the hook returns an alias, the downstream router can still select any deployment in that group because these plugin candidates are not carried into the deployment filter, so provider/budget policies cannot reliably remove a specific backend deployment.

Useful? React with 👍 / 👎.

[codecov]

Codecov Report

❌ Patch coverage is 95.65217% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/proxy_server.py	91.66%	1 Missing ⚠️
litellm/router.py	80.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_router_plugins_auto_router_yaml (bae865c) with litellm_internal_staging (9cca6c3)}

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review

[krrish-berri-2]

@greptileai review