fix(router): resolve routing group names to member deployment ids

[pjdurden]

Issue #36373: addressing a whole group of differently-named models by one name

1. Root cause

The capability the issue asks for, one virtual model name in front of several deployments that
carry different model_names, already exists in the current tree as router_settings.routing_groups:

router_settings:
  routing_groups:
    - group_name: my-group
      models: ["srv-a", "srv-b"]
      routing_strategy: least-busy

my-group is callable, shows up in /v1/models and /model_group/info, unions the members'
deployments, applies the group's own routing strategy, and cools members down on 429. I verified all
of that against a live proxy before changing anything.

What is still wrong is that a routing group name is not resolvable to the deployment IDs behind it.
Router.get_model_ids(model_name=...) only consults model_name_to_deployment_indices, which is
keyed by concrete deployment model_names, so a group name (and an alias pointing at one) returns
[]. The plain model_group_alias case happens to work only by accident: the request path rewrites
model to the alias target before the error path runs, whereas a group call deliberately keeps the
group name so strategy selectors key their state by it.

The user-visible consequence is on the 429 path. When every member of a group is cooling down,
async_raise_no_deployment_exception (and the three sibling RouterRateLimitError sites in
router.py) calls cooldown_cache.get_min_cooldown(model_ids=[]), which falls back to the router's
default cooldown instead of the members' actual cooldown. The proxy turns that number into the
retry-after header, so a client backs off for the wrong amount of time, 5s where the deployments
asked for 120s in my repro. The same lookup is used by the pass-through cost-tracking handlers,
which fall back to the raw name for a group.

2. The fix

Router._resolve_to_deployment_model_names(model) maps any name the router serves to the concrete
deployment model_names behind it: a model_group_alias resolves to its target, a callable routing
group to its members, anything else to itself. get_model_ids now iterates those names instead of
looking the requested name up directly.

That mirrors what get_model_list, get_model_group_info, is_recognized_model and
routing_group_has_alternatives already do, so the group name behaves like a model group name at
this lookup too. It is one resolution helper rather than a fix at each of the six call sites, and it
is a no-op for ordinary model names (_get_model_from_alias and get_routing_group both return
None for them, and get_routing_group deliberately lets a real deployment name shadow a
same-named group).

I did not add a second way to declare a group (for example, letting model_group_alias take a list
of models). routing_groups already covers the issue's use case, and a parallel mechanism would be
two sources of truth for the same thing.

3. Files changed

• litellm/router.py: added _resolve_to_deployment_model_names, and get_model_ids now resolves
  through it.
• tests/test_litellm/router_strategy/test_router_routing_groups.py: two regression tests, one on
  the resolution itself and one end-to-end on the cooldown time reported for a fully cooled-down
  group.

4. Risk / uncertainty

• Low blast radius: the only behaviour change is for names that previously resolved to no IDs at
  all. Callers that pass a real deployment model_name are byte-for-byte unchanged.
• get_model_ids() with no model_name is untouched, so the /model/info-style callers that
  enumerate every deployment are unaffected.
• A hidden alias ({"alias": {"model": "x", "hidden": true}}) now resolves here too. hidden only
  governs listing, so resolving it for an ID lookup is consistent with _get_model_from_alias.
• The issue itself is broad and the reporter never posted a config. I read it as "how do I address a
  group of differently-named models by one name", which routing_groups answers. If they meant
  something narrower, this fix still stands on its own as a correctness bug on the 429 path.

5. How I verified it

Live proxy on localhost:4121, no mocks in the routing path (the deployments mock the upstream 429
so the cooldown is deterministic). Config: srv-a + srv-b grouped as my-group, plus two
deployments sharing the name both as the control, all four with cooldown_time: 120.

$ curl -s -i -m 10 http://localhost:4121/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model":"my-group","messages":[{"role":"user","content":"hi"}]}' | grep -i retry-after

Before, third call (both members cooling down):

retry-after: 5
{"error":{"message":"No deployments available for selected model, Try again in 5 seconds. Passed model=my-group. ..."}}

Control, same call against the same-named group both:

retry-after: 120
{"error":{"message":"No deployments available for selected model, Try again in 120 seconds. Passed model=both. ..."}}

After the fix, the group matches the control:

retry-after: 120
{"error":{"message":"No deployments available for selected model, Try again in 120 seconds. Passed model=my-group. ..."}}

Both new tests fail on the unpatched tree (assert 5 == 120, and [] != ['deploy-1', 'deploy-2', 'deploy-3']) and pass with it; the full test_router_routing_groups.py file is green at 64 tests.

For regressions across the get_model_ids consumers: tests/test_litellm/router_strategy is 929
passed / 4 skipped, and tests/test_litellm/test_router.py plus tests/test_litellm/router_utils
is 566 passed. ruff check is clean on both changed files (ruff format wants to reflow lines that
were already unformatted before this change, so I left them alone).

[Copilot]

Pull request overview

This PR fixes router model ID resolution so routing group names and model group aliases correctly map to member deployment IDs, improving cooldown and retry-after behavior on 429 paths

Changes:

• Add Router._resolve_to_deployment_model_names() to map served names (alias, routing group, direct model) to underlying deployment model_names
• Update Router.get_model_ids() to resolve through that helper, enabling correct per-deployment cooldown lookups for groups
• Add regression tests covering group and alias ID resolution plus correct cooldown propagation for fully cooled-down groups

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
litellm/router.py	Resolve routing-group and alias names to underlying deployment model names during get_model_ids()
tests/test_litellm/router_strategy/test_router_routing_groups.py	Add regression tests for group/alias get_model_ids() resolution and correct cooldown time reporting

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[greptile-apps]

Greptile Summary

Resolves routing-group names and aliases to their member deployment IDs so cooldown consumers can use deployment-specific values

• Adds a shared requested-name-to-deployment-model-name resolver
• Updates get_model_ids to collect IDs across routing-group members
• Adds direct resolution and fully-cooled-group regression coverage

Confidence Score: 4/5

The team-scoped cooldown calculation should be corrected before merging because it can report a retry interval from an ineligible deployment

Routing applies team eligibility while selecting group members, but the changed ID expansion discards that context before calculating the minimum cooldown

Files Needing Attention: litellm/router.py

Important Files Changed

Filename	Overview
litellm/router.py	Adds group-aware ID resolution, but the expansion includes deployments excluded from a team-scoped request and can report another team's cooldown
tests/test_litellm/router_strategy/test_router_routing_groups.py	Adds focused group and alias resolution tests plus cooldown regression coverage, but does not cover team-filtered routing groups

_{Reviews (1): Last reviewed commit: "fix(router): resolve routing group names..." | Re-trigger Greptile}

[greptile-apps]

Cooldown lookup ignores team scope

When a team-scoped group is exhausted, this includes other teams' deployment cooldowns, causing an incorrect retry-after value.

Knowledge Base Used: Router: deployment selection, retries, fallbacks, and cooldowns

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_completion_streaming

✅ 31 untouched benchmarks

---

_{Comparing pjdurden:fix/36373 (026bb1c) with main (19327a6)}

[pjdurden]

checked the P1 before changing anything. a team deployment is indexed under its own internal model_name and its public team name lives in a separate index, so resolving a group through the model_name index cant reach another teams ids.

ran the same probe on base and on this branch. the plain model name gives ['shared-1'] on both. the group name gives [] on base and ['shared-1'] here. so team scope was already not consulted at that call site and this doesnt change it, it only stops the group name resolving to nothing.

pushed a test pinning that, plus the docstring reword.

[pjdurden]

Pushed eff6ed31 to fix the code-quality failure, which was the one red check actually caused by this PR.

tests/code_coverage_tests/router_code_coverage.py requires every function defined in router.py to be called from a test file with "router" in its name. _resolve_to_deployment_model_names is added by this PR and was only reached indirectly through get_model_ids, so the gate failed:

Exception: 0.27% of functions in router.py are not tested:
['_resolve_to_deployment_model_names']

The commit calls it directly and pins each of the three branches its docstring documents: an alias resolves to its target, a routing group resolves to its member model_names in declaration order, and any other name resolves to itself, including an unknown name that is passed through so the caller's own lookup is what misses. A second test covers the team-scoped case, where a deployment indexed under its internal model_name is never yielded by group resolution and is reachable only under that name.

After the change:

$ python3 tests/code_coverage_tests/router_code_coverage.py
untested_perc:  0.0                                    # exit 0

$ pytest tests/test_litellm/router_strategy/test_router_routing_groups.py
77 passed

The other two red checks are not from this PR, which touches only litellm/router.py and tests/test_litellm/router_strategy/test_router_routing_groups.py:

• proxy-behavior: tests/proxy_behavior/auth/test_auth_object_prefetch.py::test_join_binds_the_membership_to_the_requested_team fails with TypeError: object MagicMock can't be used in 'await' expression. That file came in with perf(auth): read user, team, membership, org, project and spend counters in one MGET, one query and one pipeline #40834 on litellm_internal_staging on Sep 12, which this branch picked up in the Sep 15 rebase. Nothing here touches proxy auth.
• Vertex AI: four failures in gemini_embeddings/test_batch_embed_content_transformation.py, all embedding-cost assertions (assert 0.0 == 0.00012 and similar). Same four fail on my other open PR fix(guardrails): register sensitive_data_routing as a built-in guardrail #37480, which shares no code with this one, so it looks like base-branch pricing drift rather than anything either branch did.

Happy to rebase again if you would rather see those cleared first. @krrishdholakia this has been waiting on a first human look since Aug 19 if you get a chance.

[pjdurden]

Following up on my last comment: the coverage fix worked, but code-quality is still red, and the reason is worth recording because it is not this PR.

The job runs its steps in order. router_code_coverage.py used to fail first and abort the run, so the later steps never executed. With that step now passing, the job reaches documentation_test_router_settings and fails there instead:

Run uv run --no-sync python ./tests/code_coverage_tests/router_code_coverage.py
untested_perc:  0.0                                    # the step I fixed, now green

...
File "./tests/documentation_tests/test_router_settings.py", line 73
Exception: Keys not documented in 'router settings - Reference':
{'timeout', 'redis_host', 'redis_url', 'redis_port', 'set_verbose', 'debug_level',
 'model_group_alias', 'retry_policy', 'alerting_config', 'guardrail_list', ... }   # 28 keys

Two things say this is base-branch debt rather than mine:

1. This PR adds no Router.__init__ keyword. The net diff is 27 lines in router.py and the test file, and it contains no def __init__ change and no new setting. routing_groups is not in the undocumented set. The 28 names are long-standing settings like timeout, redis_host and set_verbose.

2. litellm_internal_staging is failing this same step on its own. Run 35410132561 on the base branch, commit c01f17d7, Sep 19, with no PR involved, fails at documentation_test_router_settings. The last green Code Quality run on that branch was 252c71c0 on Sep 15.

That also explains why my other PR #37480 shows code-quality passing: its run is from Sep 15, against the base before this broke.

So the one red check this PR was responsible for is fixed, and what is left needs either the docs page updated for those 28 settings or the gate adjusted, both of which look like base-branch work rather than something to smuggle into a routing change. Happy to send a separate PR documenting them if that would help, just say the word.

[CLAassistant]

All committers have signed the CLA.

[pjdurden]

merged current main in (d8704a608b). one real conflict in get_model_group_info: main now builds the list with get_model_list_of_routed_group, kept that and computed the model group description off it. the routing group tests and main's new priority group tests were both appended at the same spot, kept both. tests/unit/router_strategy/test_router_routing_groups.py passes (139).