test(e2e): cover router strategy selection and context-window fallback

[yassin-berriai]

TLDR

Problem this solves:

• Router strategy selection had no e2e coverage
• Context-window fallback rerouting was untested end to end
• Five reliability registry cells sat uncovered

How it solves it:

• One model group per strategy, one deciding difference
• Asserts which deployment served, from response headers
• Real oversized prompt proves context-window reroute

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Captured at commit 53a2138d0523a9060e8a2b71d619c70fabdc66e5, against a live proxy on port 4057 backed by postgres and redis with store_model_in_db on, driving real openai/gpt-5.5 and openai/gpt-4 traffic that costs real money (roughly five cents a run, almost all of it the ten thousand token prompt the context-window test sends)

$ cd tests/e2e
$ set -a; source .env; set +a
$ PYTHONPATH=. LITELLM_PROXY_URL=http://localhost:4057 python -m pytest \
    router/test_reliability_routing_e2e.py \
    router/test_reliability_context_window_fallback_e2e.py -v

============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /.../tests/e2e
configfile: pytest.ini
plugins: anyio-4.14.2
collecting ... collected 5 items

router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_cost_based_picks_lowest_cost PASSED [ 20%]
router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_usage_based_picks_deployment_under_tpm PASSED [ 40%]
router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_simple_shuffle_picks_healthy_deployment PASSED [ 60%]
router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_picks_lowest_latency PASSED [ 80%]
router/test_reliability_context_window_fallback_e2e.py::TestReliabilityContextWindowFallback::test_context_window_overflow_routes_to_fallback PASSED [100%]

============================== 5 passed in 20.22s ==============================

Every assertion these tests make was also checked in reverse. For each test the deciding setup was flipped, the flip was verified to have actually reached the file, and the test was confirmed to go red before the file was restored from a saved copy and byte-compared. Swapping the two price constants made cost-based routing serve the expensive deployment and failed on the served model id; moving the tpm=1 ceiling onto the other deployment failed the same way; moving the shuffle weight onto the unreachable deployment failed immediately with a 500 connection error from the dead base, which is the outcome the weighted pick exists to prevent; widening the seeding deadline from one millisecond to sixty seconds meant no latency penalty was ever recorded and failed on the 408 seeding assertion; and pointing the context-window primary at openai/gpt-5.5 failed on the rejection assertion because a million token window cannot be overflowed by the prompt

Type

✅ Test

Changes

Two new files under tests/e2e/router/, no shared harness module touched. test_reliability_routing_e2e.py covers four of the router's routing strategies, and test_reliability_context_window_fallback_e2e.py covers context-window fallback rerouting. Together they add five rows to tests/e2e/coverage_registry/reliability.yaml, taking Reliability and Performance from 8 of 26 to 13 of 26

Each routing test builds its own model group of real openai/gpt-5.5 deployments that differ in exactly one dimension the strategy under test reads, so only one deployment can win and the pick is a statement about the strategy rather than about luck. The strategy is chosen per request through router_settings_override.routing_strategy on the /chat/completions body, so one long-lived proxy serves every strategy with no config change and no restart. The deployment that actually served each call is read back from the x-litellm-model-id response header and compared against the model_id that /model/new returned, so a 200 on its own never passes anything

How each test makes the correct choice unambiguous:

• Cost: two deployments of the same model priced a million-fold apart, 1e-3 against 1e-9 per input and output token, and the cheap model_id has to serve
• Usage: one deployment's tpm ceiling is a single token, so no real prompt fits under it, and the deployment that still has token budget has to serve
• Simple shuffle: a zero-weight deployment pointed at an unreachable base sits beside a weight-one healthy one, so four consecutive calls all have to come back from the healthy model_id; picking the other one fails the connection rather than merely picking differently
• Latency: the first deployment answers a request held to a one millisecond deadline, which is the worst latency observation the router can record, then a second deployment joins and answers normally; a simple-shuffle control call in the middle forces a pick onto the timed-out deployment by weight and gets a real answer, which proves it is still healthy and selectable and rules out cooldown as the reason it is later skipped

The context-window test registers a real openai/gpt-4 deployment, chosen because 8k is the smallest context window OpenAI still serves and therefore the only one a prompt can overflow for a few cents. The same oversized prompt is sent twice. The first call carries no fallback and has to come back as a real context-length rejection from OpenAI, which is what proves the prompt genuinely overflows the primary instead of the deployment being broken some other way. The second names gpt-5.5 as the context-window fallback and has to come back 200, served by a different model_id than the primary, with x-litellm-attempted-fallbacks at one or more

Every prompt carries a unique marker because the proxy under test has its response cache on; a repeated prompt would be answered from cache before the router ever routed it

Three of the eight cells this work was scoped to are deliberately not covered here, so their absence does not read as an oversight. reliability.routing.least_busy.picks_lowest_traffic would have to prove that the router had counted an in-flight request against a specific deployment at the moment of the pick, and nothing the proxy exposes says that: the least-busy tally never appears in the router's in-memory cache dump, /metrics is not served, and a spend row only lands once a call finishes, which is exactly when the request stops being in flight. Barriers built on what is observable, a fixed wait or a virtual-key concurrency cap, both sit at an earlier layer than routing, so the pick could still be made while both deployments looked idle; that is a probability bet dressed as a barrier and it does not belong in the suite. reliability.fallback.content_policy.routes_to_fallback needs a ContentPolicyViolationError, which litellm raises only for an OpenAI-shaped invalid_request_error carrying content_policy_violation, an "Invalid prompt ... violating our usage policy" message, or "request was rejected as a result of the safety system"; OpenAI chat completions returns none of those, it answers 200 with a refusal, so content_policy_fallbacks never fires. That error surface belongs to the Azure content filter, Bedrock guardrails, and the image endpoints, and this environment has credentials for none of them. reliability.cache.prompt_caching_model_select.returns_cached needs the router constructed with optional_pre_call_checks: ["prompt_caching"]; the default list is empty and Router.update_settings does not accept that key, so covering it requires a router_settings addition to the e2e gateway config and a proxy restart rather than a test-only change

On timing: none of these tests waits on the clock for a state it then asserts against. The latency test's one millisecond deadline cannot be met by any real provider round trip, and every other test is a sequence of completed calls whose outcome is read from response headers. The one behaviour that would need a wall-clock or concurrency setup, least-busy, is left uncovered for the reason given above rather than shipped as a timing bet

QA runbook

Prerequisites for every item below: a live proxy on http://localhost:4000 with store_model_in_db on and OPENAI_API_KEY in its environment, the master key sk-1234, and curl -i so the x-litellm-* response headers are visible. Every /model/new call returns a model_id; note it, because each check compares it against the x-litellm-model-id of the served response. Delete the deployments with POST /model/delete when done

• tests/e2e/router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_cost_based_picks_lowest_cost - cost-based routing spends the request on the cheaper of two identically capable deployments

  • Register the expensive deployment: curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -H 'Content-Type: application/json' -d '{"model_name":"routing-cost-demo","litellm_params":{"model":"openai/gpt-5.5","api_key":"os.environ/OPENAI_API_KEY","input_cost_per_token":0.001,"output_cost_per_token":0.001},"model_info":{}}'
  • Register the cheap one under the same model_name with input_cost_per_token and output_cost_per_token of 1e-9
  • Send one completion pinned to the strategy: curl -i -X POST http://localhost:4000/chat/completions -H "Authorization: Bearer sk-1234" -H 'Content-Type: application/json' -d '{"model":"routing-cost-demo","messages":[{"role":"user","content":"answer in one word: what colour is a clear midday sky?"}],"max_tokens":256,"router_settings_override":{"routing_strategy":"cost-based-routing"}}'
  • Expect 200 and x-litellm-model-id equal to the cheap deployment's model_id
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_usage_based_picks_deployment_under_tpm - usage-based routing skips a deployment whose tpm ceiling cannot fit the request

  • Register a capped deployment: same /model/new body as above under model_name routing-tpm-demo, with "tpm": 1 in litellm_params and no cost overrides
  • Register a second deployment under the same name with no tpm at all
  • Send one completion with "router_settings_override":{"routing_strategy":"usage-based-routing-v2"}
  • Expect 200 and x-litellm-model-id equal to the uncapped deployment's model_id, since a single token of budget fits no real prompt
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_simple_shuffle_picks_healthy_deployment - simple shuffle's weighted pick keeps every call off a zero-weight deployment that cannot answer

  • Register the unreachable deployment under model_name routing-shuffle-demo with "weight": 0 and "api_base": "http://127.0.0.1:9/v1"
  • Register the healthy one under the same name with "weight": 1 and no api_base
  • Send four completions with "router_settings_override":{"routing_strategy":"simple-shuffle"}
  • Expect all four to return 200 with x-litellm-model-id equal to the healthy deployment's model_id; a pick of the zero-weight deployment surfaces as a 500 naming a connection error
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_picks_lowest_latency - latency-based routing avoids the deployment it measured as slow, and does so on latency rather than on cooldown

  • Register one deployment under model_name routing-latency-demo with "weight": 1
  • Send a completion with "router_settings_override":{"routing_strategy":"latency-based-routing","timeout":0.001} and expect a 408 naming the timeout; this is the observation that gives that deployment the router's worst recorded latency
  • Register a second deployment under the same name with "weight": 0
  • Send a completion with "router_settings_override":{"routing_strategy":"latency-based-routing"} and expect the second deployment's model_id, since a deployment with no history reads as the lowest known latency
  • Send a control completion with "router_settings_override":{"routing_strategy":"simple-shuffle"} and expect 200 from the first deployment, which proves it is still healthy and selectable rather than in cooldown
  • Send one more latency-based completion and expect the second deployment's model_id again, now that both have history
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_context_window_fallback_e2e.py::TestReliabilityContextWindowFallback::test_context_window_overflow_routes_to_fallback - a prompt that overflows a deployment's context window is rerouted to the configured context-window fallback and answered

  • Register a small-context deployment: curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -H 'Content-Type: application/json' -d '{"model_name":"reliability-ctx-demo","litellm_params":{"model":"openai/gpt-4","api_key":"os.environ/OPENAI_API_KEY"},"model_info":{}}'
  • Build a prompt of roughly ten thousand tokens, for example the sentence "The quick brown fox jumps over the lazy dog. " repeated nine hundred times followed by "summarize the text above"
  • Send it to reliability-ctx-demo with "max_tokens":256 and no fallback, and expect a 400 whose body names the model's maximum context length
  • Send the same prompt again with "router_settings_override":{"context_window_fallbacks":[{"reliability-ctx-demo":["gpt-5.5"]}]} and expect 200, x-litellm-model-name of openai/gpt-5.5, an x-litellm-model-id different from the primary's, and x-litellm-attempted-fallbacks of at least 1
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Adds live end-to-end coverage for four router selection strategies and context-window fallback rerouting.

• Verifies selected deployments through response headers.
• Uses isolated model groups and real provider calls to make routing decisions observable.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/router/test_reliability_context_window_fallback_e2e.py	Adds an end-to-end test proving that a genuine context-window rejection reroutes to a larger fallback model.
tests/e2e/router/test_reliability_routing_e2e.py	Adds end-to-end coverage for cost-based, usage-based, weighted shuffle, and latency-based routing; the previously reported least-busy readiness race is no longer present.

_{Reviews (3): Last reviewed commit: "test(e2e): cover router strategy selecti..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please re-review the current head 68da7af. The findings from your previous review have been addressed in an amended commit.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai please re-review the current head 53a2138. The least-busy cell has been removed rather than covered with a barrier that cannot observe deployment-level in-flight accounting; the remaining five cells are unaffected.

[mateo-berri]

Closing this since #39857 merged with live routing-strategy and context-window fallback cells, so this one is superseded. Thanks for the groundwork