test(e2e): cover the reliability retry, cooldown, fallback, and routing-strategy cells

[mateo-berri]

TLDR

Problem this solves:

• The reliability registry sits at 20 of 38 cells covered on today's staging, so a router regression ships unnoticed
• 14 rows had no live test: 3 retry, 4 cooldown, 5 routing strategy, content-policy fallback, prompt-cache affinity
• test(e2e): cover router retry behavior across failure classes #34836 and test(e2e): cover router strategy selection and context-window fallback #34840 tried this with mocked deployments, so they proved nothing about the real router

How it solves it:

• Every failure is real: a 1ms deadline, a bogus key, an Azure content filter, this proxy fronting a dead or rate-limited group for the 500 and 429
• Every group is a two-deployment pair whose shuffle weights make the first pick deterministic; the shuffle cell itself takes ten picks, since three even picks land on the weighted deployment one time in eight
• Strategies come in per request through router_settings_override, so one proxy serves all of them
• Harness gains open_stream (a held in-flight request) and GET /router/settings
• Registry moves to 32 of 38 on the CI stack, where project-releaser (User Request) Expose log probs for hf calls  #223 now sets E2E_PROMPT_CACHING_STACK, and to 31 of 38 on a laptop without that marker (the prompt-cache test stays behind prompt_caching_stack, mirroring managed_files, so a stack without the pre-call check skips it instead of failing). The context-window retry row stays uncovered on purpose, since the router never retries a 400
• Opt-in marker deselection (weekly, managed_files, prompt_caching_stack) now lives once in tests/e2e/conftest.py as OPT_IN_MARKERS instead of a copy per directory
• Cooldown and least-busy tests allow for the CI stack's two replicas: a 15s bench propagation window over the cooldown cache's 1s Redis re-read from fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025, and three idle deployments beside the busy one so a stale per-process count can never tie it
• Datadog's payload truncation rewrote the standard logging payload every callback of a request shares, so with Datadog on the prompt caching router check never pinned a cached conversation to the deployment holding its cache; the truncation now works on a copy (LIT-7684)
• The latency-based and least-busy cells are skipped: since fix(router): keep per-request routing_strategy override selectors out of global callbacks #40229 a per-request routing_strategy override builds its selector with register_callbacks=False, so those strategies route without their signal (LIT-7682)

User Flow

Before: an engineer breaks cooldown recovery in a router change and the e2e stage stays green, so the regression ships

1. They push a branch off main that changes how a benched deployment comes back once its cooldown_time lapses, and open a PR at https://github.com/BerriAI/litellm/pulls
2. The Buildkite e2e stage brings up a proxy on the branch and runs tests/e2e/router; nothing in it registers a failing deployment, sends it traffic, watches it get benched, and waits for it to return, and the reliability registry counts 20 of 38 cells covered
3. The stage finishes green, the PR's checks list shows buildkite/e2e-tests passed, and it merges
4. The first person to notice is an operator whose deployment stays benched after a transient outage, reading x-litellm-model-id on every response and seeing the backup forever

After: the same change fails the e2e stage at the cooldown test, naming the deployment that never came back

1. They push a branch off main that changes how a benched deployment comes back once its cooldown_time lapses, and open a PR at https://github.com/BerriAI/litellm/pulls
2. The Buildkite e2e stage brings up a proxy on the branch and runs tests/e2e/router; test_reliability_cooldowns_e2e.py registers a pair through POST http://e2e-pr--router.litellm.svc.cluster.local/model/new (a deployment that always answers 500, with cooldown_time: 30 and InternalServerErrorAllowedFails: 0, plus a healthy backup at weight: 0), sends POST http://e2e-pr--router.litellm.svc.cluster.local/v1/chat/completions with router_settings_override: {"num_retries": 0} and gets the 500, keeps sending it and within 15 seconds gets a 200 whose x-litellm-model-id is the backup from every replica, sees only the backup until 26 seconds after the trip, then repeats that request once a second
3. Under the broken change the failing deployment never gets traffic again, so 40 seconds after its last failure the test fails with never sent traffic back to its benched deployment within 40s of its last failure, so the cooldown never lapsed, the stage goes red, and the PR's checks list shows buildkite/e2e-tests failed
4. Retries, fallbacks, each routing strategy, and prompt-cache affinity guard their own cells the same way (registry at 32 of 38 on the CI stack, 31 of 38 on a laptop without E2E_PROMPT_CACHING_STACK), so a regression in any of them stops at the same gate before an operator sees it

Before: an operator turns on Datadog logging and a cached conversation drifts off the deployment holding its prompt cache

1. They run the proxy with litellm_settings.callbacks: ["datadog"] and router_settings.optional_pre_call_checks: ["prompt_caching"], and POST http://localhost:47925/model/new twice into one group: vertex_ai/claude-haiku-4-5 with weight: 1, then openai/gpt-5.5 with weight: 20
2. They POST http://localhost:47925/v1/chat/completions with a system turn of about 600 filler clauses carrying "cache_control": {"type": "ephemeral"} and a short user turn, and get a 200 whose x-litellm-model-id is the Claude deployment with usage.cache_creation_input_tokens: 12601
3. They send the same conversation three more times and two of the three answers come from the gpt-5.5 deployment with prompt_tokens: 12017 and no Anthropic cache read (the third landed on the Claude deployment by the weighted shuffle's luck), so the cache they paid to write is only read when the shuffle happens to pick that deployment

After: every follow-up stays on the Claude deployment holding the cache while the Datadog logger keeps sending its payloads

1. They run the proxy with litellm_settings.callbacks: ["datadog"] and router_settings.optional_pre_call_checks: ["prompt_caching"], and POST http://localhost:47925/model/new twice into one group: vertex_ai/claude-haiku-4-5 with weight: 1, then openai/gpt-5.5 with weight: 20
2. They POST http://localhost:47925/v1/chat/completions with a system turn of about 600 filler clauses carrying "cache_control": {"type": "ephemeral"} and a short user turn, and get a 200 whose x-litellm-model-id is the Claude deployment with usage.cache_creation_input_tokens: 12601
3. They send the same conversation three more times, through either replica or the load balancer, and every answer comes from the Claude deployment with usage.prompt_tokens_details.cached_tokens: 12601; the Datadog logger still sends every request's log with messages and response cut at 10,000 characters

Relevant issues

Linear ticket

Resolves LIT-5731, LIT-7684

Scope note: the ticket's rows-to-close list names 17 rows and 16 of them get a cell here (plus the 5xx and timeout fallback rows): 14 pass and two, latency-based and least-busy, are skipped under LIT-7682 because #40229 builds a per-request strategy override with register_callbacks=False, so those strategies route without their signal. The one row without a cell is the context-window retry, filed as LIT-7036: the router hands a 400 context error to context_window_fallbacks and never retries it. The Redis circuit breaker, throughput SLO, and LLM complexity classifier rows predate the ticket and are not in its list. The prompt-cache cell runs in CI now that project-releaser #223 sets E2E_PROMPT_CACHING_STACK; it surfaced LIT-7684 (the Datadog fix in this PR) and LIT-7683 (OpenAI-family deployments never pin, left alone). A follow-up tracks a tighter sibling-propagation cell now that #40025 gave cooldowns a 1s Redis re-read

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs ran on rigs shaped like the CI e2e stack: two proxy processes started with --num_workers 2 on random ports behind an nginx round-robin front (127.0.0.1:59076 for the Before leg, 21496 for the regression run at 7c85be2, 56721 for the After leg at 308f66f, then 38543, 47915, 40937, 58586, and 40051 for rigs F, M1, M2, M3, and F2 on the working tree that became 6de9087), sharing one Postgres 17 and one Redis, so consecutive requests from the tests alternate replicas (nginx's access log counted 258 and 259 upstream hits over the last run). The proxy config is the CI stack's router_settings (routing_strategy: simple-shuffle, num_retries: 3, allowed_fails: 5, cooldown_time: 30) plus optional_pre_call_checks: ["prompt_caching"], store_model_in_db: true, a Redis cache, and gpt-5.5, claude-haiku-4-5, and gemini-2.5-flash with real OpenAI, Anthropic, Gemini, and Azure credentials. Tests ran from tests/e2e with LITELLM_PROXY_URL at the nginx front, which the 500 and 429 deployments call back into. The regression injected on both legs makes a benched deployment stay benched 60x its cooldown_time: sed -i '' -e 's/cooldown_time=_cooldown_time,$/cooldown_time=_cooldown_time * 60,/' -e 's/ttl=_cooldown_time,$/ttl=_cooldown_time * 60,/' litellm/router_utils/cooldown_cache.py, with the proxies rebooted from the patched tree

Before (2151dcb)

The new cells do not exist

1. cd tests/e2e && python -m pytest --collect-only -q router/test_reliability_cooldowns_e2e.py router/test_reliability_routing_strategies_e2e.py

ERROR: file or directory not found: router/test_reliability_cooldowns_e2e.py
no tests collected in 0.07s
COLLECT_EXIT=4

The injected cooldown regression slips through the base suite

1. Regression applied on the base worktree, both replicas rebooted from it
2. cd tests/e2e && python -m pytest router/test_reliability_cache_e2e.py router/test_reliability_fallbacks_e2e.py router/test_reliability_retries_e2e.py router/test_reliability_timeouts_e2e.py -v -p no:cacheprovider --tb=short

router/test_reliability_cache_e2e.py::TestReliabilityCache::test_exact_cache_returns_cached PASSED
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_5xx_routes_to_fallback PASSED
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_timeout_routes_to_fallback PASSED
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_context_window_routes_to_fallback PASSED
router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_timeout_on_first_deployment_succeeds_on_retry PASSED
router/test_reliability_timeouts_e2e.py::TestReliabilityTimeouts::test_request_timeout_exceeds_deadline PASSED
router/test_reliability_timeouts_e2e.py::TestReliabilityTimeouts::test_stream_timeout_exceeds_deadline PASSED
======================== 7 passed in 152.22s (0:02:32) =========================
PYTEST_EXIT=0

3. Registry collector on the base branch: cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector

MODULE                             COVERED    COVERAGE
Reliability & Performance            18/36       50.0%

After (308f66f, 6de9087, 038a7c6, cae009c, and 6db93a9)

Every cell runs at the round-2 tip 038a7c6 (rig G)

1. Two proxy processes from this branch at 038a7c6 (proxy_cli.py --num_workers 2 each, on 45716 and 30396) share one Postgres 17 database and one Redis behind nginx round robin on 49472, LITELLM_PROXY_URL points at nginx so consecutive requests alternate replicas (the nginx access log ends with 212 hits on 45716 and 211 on 30396 across both runs below), and E2E_PROMPT_CACHING_STACK=1 selects the prompt-cache cell
2. cd tests/e2e && E2E_PROMPT_CACHING_STACK=1 python -m pytest router/test_reliability_cooldowns_e2e.py router/test_reliability_fallbacks_e2e.py router/test_reliability_prompt_caching_e2e.py router/test_reliability_retries_e2e.py router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short --durations=0
3. Output:

(17 PASSED lines trimmed: every cell not listed here passed on its first attempt)
test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_5xx_trips_cooldown_then_recovers RERUN [  5%]
test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_auth_failure_trips_cooldown_then_recovers RERUN [ 16%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out RERUN [ 94%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out FAILED [ 94%]
AssertionError: latency-based call failed with 500: {"error":{"message":"float division by zero","type":"internal_server_error","param":null,"code":"500"}}
============== 1 failed, 17 passed, 3 rerun in 1079.74s (0:17:59) ==============

4. The three reruns and the one failure are the two tracked router races, not the cells: the replica logs hold dictionary changed size during iteration from router.py:7325 (LIT-7042: 3 hits on 45716, 6 on 30396) behind the two cooldown reruns, and float division by zero from lowest_latency.py:450 (LIT-7053: 0 hits on 45716, 3 on 30396, every one inside the latency-based cell) behind that cell's rerun and its final failure. Both bugs are filed with fix shapes (fix(logging): finish response metadata before the sync logging thread reads it #39869 is open for LIT-7042) and neither is touched by this PR; the cell fails on them by design because the caller sees a 500
5. Re-run of the strategies file on the same rig, same replicas: cd tests/e2e && python -m pytest router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short --durations=0
6. Output:

(5 progress lines trimmed)
======================== 5 passed in 286.44s (0:04:46) =========================

Cooldown and routing-strategy cells pass across both replicas

1. Two proxy processes from this branch at 308f66f (proxy_cli.py --num_workers 2 each, on 20086 and 54430) share one Postgres database and one Redis behind nginx round robin on 56721, and LITELLM_PROXY_URL points at nginx so consecutive requests alternate replicas
2. cd tests/e2e && python -m pytest router/test_reliability_cooldowns_e2e.py router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short --durations=0

(9 PASSED lines trimmed: the two RERUN cells passed on their second attempt, the rest on their first)
router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_429_trips_cooldown_then_recovers RERUN [ 22%]
router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_timeout_trips_cooldown_then_recovers RERUN [ 44%]
router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out RERUN [ 88%]
==================== 9 passed, 3 rerun in 667.71s (0:11:07) ====================

3. Durations, which are the 30s cooldown plus the recovery window for each cooldown cell and the drained 3000-token stream for least-busy

(5 shorter durations trimmed)
105.29s call     router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_least_busy_avoids_deployment_with_request_in_flight
85.28s call     router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_5xx_trips_cooldown_then_recovers
67.52s call     router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_429_trips_cooldown_then_recovers
62.95s call     router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_timeout_trips_cooldown_then_recovers
61.70s call     router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_auth_failure_trips_cooldown_then_recovers

4. The three reruns were proxy 500s that the harness retries once, and every retried cell passed: two on replica A where a successful backup call died in merge_litellm_metadata with dictionary changed size during iteration (23:58:52 in the 429 cell, 00:01:39 in the timeout cell, LIT-7042), and one on replica B where latency-based selection raised ZeroDivisionError at lowest_latency.py:450 (00:05:16, LIT-7053)
5. nginx's access log for the run shows nine 500s: six are the deliberate faults of the 5xx cooldown cell (each fault is a pair, the outer call and its nested upstream), the other three are the reruns above

Retry, fallback, and prompt-cache cells pass on the same rig

1. Same rig, cd tests/e2e && python -m pytest router/test_reliability_fallbacks_e2e.py router/test_reliability_retries_e2e.py router/test_reliability_prompt_caching_e2e.py -v -p no:cacheprovider --tb=short --durations=0

router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_5xx_routes_to_fallback PASSED [ 11%]
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_timeout_routes_to_fallback PASSED [ 22%]
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_context_window_routes_to_fallback PASSED [ 33%]
router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_content_policy_routes_to_fallback PASSED [ 44%]
router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_timeout_on_first_deployment_succeeds_on_retry PASSED [ 55%]
router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_5xx_on_first_deployment_succeeds_on_retry PASSED [ 66%]
router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_429_on_first_deployment_succeeds_on_retry PASSED [ 77%]
router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_auth_failure_on_first_deployment_succeeds_on_retry PASSED [ 88%]
router/test_reliability_prompt_caching_e2e.py::TestReliabilityPromptCachingAffinity::test_cached_conversation_stays_on_deployment_holding_its_cache PASSED [100%]
======================== 9 passed in 290.15s (0:04:50) =========================

The injected cooldown regression fails every cooldown cell

1. Run at 7c85be2 on the same two-replica topology (nginx front on 21496); the two commits since, 1bd70a6 and 308f66f, touch only test_reliability_routing_strategies_e2e.py, so the cooldown cells are the ones this run exercised
2. Regression applied to litellm/router_utils/cooldown_cache.py in this branch's worktree (the bench and its TTL multiplied by 60), both replicas rebooted from it
3. cd tests/e2e && python -m pytest router/test_reliability_cooldowns_e2e.py -v -p no:cacheprovider --tb=short

(5 FAILED/RERUN progress lines trimmed; the four E lines below are the four cells)
E   Failed: reliability-cooldown-5xx-1544c7db7a58 never sent traffic back to its benched deployment within 40s of its last failure, so the cooldown never lapsed
E   Failed: reliability-cooldown-429-cc8f5a963506 never sent traffic back to its benched deployment within 40s of its last failure, so the cooldown never lapsed
E   Failed: reliability-cooldown-auth-2a4029c79e48 never sent traffic back to its benched deployment within 40s of its last failure, so the cooldown never lapsed
E   Failed: reliability-cooldown-timeout-0eb3c5310ae3 never sent traffic back to its benched deployment within 40s of its last failure, so the cooldown never lapsed
==================== 4 failed, 1 rerun in 363.62s (0:06:03) ====================
PYTEST_EXIT=1

4. Regression reverted (git checkout -- litellm/router_utils/cooldown_cache.py), git status shows only the test files

Every cell runs again on the working tree behind 6de9087 (rig F)

1. Two proxy processes from this branch's working tree (308f66f plus the changes that became 6de9087: the opt-in marker, the shared deselection hook, the tuple default) with proxy_cli.py --num_workers 2 each, on 28957 and 46548, sharing one Postgres 17 database and one Redis behind nginx round robin on 38543 (149 and 148 upstream hits), and E2E_PROMPT_CACHING_STACK=1 exported so the prompt-cache cell is selected
2. cd tests/e2e && python -m pytest router/test_reliability_cooldowns_e2e.py router/test_reliability_fallbacks_e2e.py router/test_reliability_prompt_caching_e2e.py router/test_reliability_retries_e2e.py router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short --durations=0

(per-cell lines trimmed; this run is superseded by rig G above)
============== 1 failed, 17 passed, 1 rerun in 896.37s (0:14:56) ===============

3. The one failure is the latency cell meeting LIT-7042 on both of its attempts ({"error":{"message":"dictionary changed size during iteration","type":"internal_server_error","param":null,"code":"500"}}), the proxy's logging thread iterating the request metadata while the response path is still writing it, on a box running at load 19 to 28 on 18 cores; fix(logging): finish response metadata before the sync logging thread reads it #39869 fixes the race. The same cell passed first try in CI build 245 and in rig F2 below

Router mutations fail exactly the cells that cover them (rigs M1, M2, M3)

1. Three mutations applied to the worktree's litellm/ (diff below): the retry path raises instead of retrying, the fallback lookup returns no chain, and the prompt-cache check stops pinning the deployment holding the cache; both replicas rebooted from it (22358 and 33157 behind nginx on 47915, 56 upstream hits each)

(context lines trimmed to the hunks)
diff --git a/litellm/router.py b/litellm/router.py
@@ -7544,7 +7544,7 @@ class Router:
-            if num_retries > 0:
+            if (num_retries := 0) > 0:
@@ -7762,7 +7762,7 @@ class Router:
-        if model_group is None:
+        if model_group is None or True:
diff --git a/litellm/router_utils/fallback_event_handlers.py b/litellm/router_utils/fallback_event_handlers.py
@@ -326,7 +326,7 @@ def get_fallback_model_group_for_lookup_groups(
-    results: Final = tuple(get_fallback_model_group(fallbacks=fallbacks, model_group=group) for group in lookup_groups)
+    results: Final = ()
diff --git a/litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py b/litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py
@@ -100,7 +100,7 @@ class PromptCachingDeploymentCheck(CustomLogger):
-                        return [deployment]
+                        return healthy_deployments

2. cd tests/e2e && E2E_PROMPT_CACHING_STACK=1 python -m pytest router/test_reliability_retries_e2e.py router/test_reliability_fallbacks_e2e.py router/test_reliability_prompt_caching_e2e.py -v -p no:cacheprovider --tb=short

(9 FAILED lines trimmed: the nine failures are the four retry cells, the four fallback cells, and the prompt-cache cell)
AssertionError: expected 200 after fallback, got 400: 
AssertionError: expected 200 after fallback, got 408: 
AssertionError: expected 200 after fallback, got 500: 
AssertionError: follow-up 0 landed on 'c955f485-2545-454b-b47b-b504e9cf0910' instead of the deployment holding the cache (880dcb84-e9d8-43d5-a28a-d206ba1b64a2)
AssertionError: the retry should have landed on the healthy backup, got 401: 
AssertionError: the retry should have landed on the healthy backup, got 408: 
AssertionError: the retry should have landed on the healthy backup, got 429: 
AssertionError: the retry should have landed on the healthy backup, got 500: 
==================== 9 failed, 2 rerun in 337.07s (0:05:37) ====================
PYTEST_EXIT=1

3. Mutations reverted (git checkout -- litellm/router.py litellm/router_utils/fallback_event_handlers.py litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py), git status --porcelain -- litellm empty
4. Second pair of mutations (diff below): the per-request strategy override keeps its name but loses its selector, and simple-shuffle weighs every deployment the same; replicas rebooted from it (54025 and 31600 behind nginx on 40937)

(context lines trimmed to the hunks)
diff --git a/litellm/router.py b/litellm/router.py
@@ -1514,7 +1514,7 @@ class Router:
-            return override, self._get_override_strategy_selector(override)
+            return override, None
diff --git a/litellm/router_strategy/simple_shuffle.py b/litellm/router_strategy/simple_shuffle.py
@@ -43,7 +43,7 @@ def simple_shuffle(
-            weights = [m["litellm_params"].get(weight_by, 0) for m in healthy_deployments]
+            weights = [1 for m in healthy_deployments]

5. Same rig, cd tests/e2e && python -m pytest router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short

test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_simple_shuffle_honors_weights PASSED [ 20%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_cost_based_picks_cheapest_deployment FAILED [ 40%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_usage_based_picks_deployment_with_tpm_headroom FAILED [ 60%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out FAILED [ 80%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_least_busy_avoids_deployment_with_request_in_flight FAILED [100%]
AssertionError: cost-based-routing call failed with 429: 
AssertionError: latency-based call failed with 429: 
AssertionError: the long stream should have opened with a 200, got 429
AssertionError: usage-based-routing-v2 call failed with 429: 
=================== 4 failed, 1 passed in 215.14s (0:03:35) ====================

6. The shuffle cell survived that run with its three picks: a shuffle that ignores the weights spreads calls evenly, and three even picks all land on the weighted deployment one time in eight. 6de9087 raises that cell to ten picks (one time in a thousand) and rig M3 re-applies the same two mutations against it (replicas 56522 and 55300 behind nginx on 58586)

(5 progress lines trimmed)
AssertionError: simple-shuffle picked ['e660b2b1-307b-47a5-806e-8507cf9c5b9d', 'a1f8d7e3-4267-4900-92e1-4ebb94974507', 'a1f8d7e3-4267-4900-92e1-4ebb94974507', 'e660b2b1-307b-47a5-806e-8507cf9c5b9d', 'a1f8d7e3-4267-4900-92e1-4ebb94974507', 'a1f8d7e3-4267-4900-92e1-4ebb94974507', 'e660b2b1-307b-47a5-806e-8507cf9c5b9d', 'a1f8d7e3-4267-4900-92e1-4ebb94974507', 'e660b2b1-307b-47a5-806e-8507cf9c5b9d', 'e660b2b1-307b-47a5-806e-8507cf9c5b9d'], expected every call on a1f8d7e3-4267-4900-92e1-4ebb94974507 (it holds all of the group's shuffle weight)
AssertionError: cost-based-routing call failed with 429:
AssertionError: usage-based-routing-v2 call failed with 429:
AssertionError: latency-based call failed with 429:
AssertionError: the long stream should have opened with a 200, got 429
======================== 5 failed in 213.27s (0:03:33) =========================

7. Mutations reverted, git status --porcelain -- litellm empty

The strategy cells pass again unmutated at the ten-pick shuffle (rig F2)

1. Two fresh proxy processes from the same working tree (54249 and 33458 behind nginx on 40051), cd tests/e2e && python -m pytest router/test_reliability_routing_strategies_e2e.py -v -p no:cacheprovider --tb=short --durations=0

test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_simple_shuffle_honors_weights PASSED [ 20%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_cost_based_picks_cheapest_deployment PASSED [ 40%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_usage_based_picks_deployment_with_tpm_headroom PASSED [ 60%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out PASSED [ 80%]
test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_least_busy_avoids_deployment_with_request_in_flight PASSED [100%]
======================== 5 passed in 285.64s (0:04:45) =========================

Round 3: the full CI run at 036a380 finds the two bugs behind the last three red cells

The e2e-tests gate on 036a380 (staging merged in, so #40025 and #40229 included) triggered litellm-e2e-pr build 464 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/464): 3 failed, 963 passed, 58 skipped in 31:51. The stack carried E2E_PROMPT_CACHING_STACK from project-releaser #223, so the prompt-cache cell ran in CI for the first time, and the timeout cooldown cell passed with #40025 in. All three failures are in this PR's files and each is a product bug rather than a test bug:

• test_latency_based_routes_around_deployment_that_times_out: the simple-shuffle control should have timed out on the weighted deployment ..., got 200: it was benched. With no latency samples the picks kept landing on the 1ms deployment until its timeouts benched it, so the control call met a cooldown instead of the timeout the cell expects
• test_least_busy_avoids_deployment_with_request_in_flight: least-busy picked [<the in-flight deployment> x3]. Same cause: since fix(router): keep per-request routing_strategy override selectors out of global callbacks #40229 a per-request routing_strategy override builds its selector with register_callbacks=False, so least-busy never learns which deployment is busy and latency-based never learns which one is slow. Both cells are @pytest.mark.skip under LIT-7682 rather than deleted, since their registry rows and runbook steps still describe the behavior the router should have
• test_cached_conversation_stays_on_deployment_holding_its_cache: follow-up 0 landed on '53d918b3...' instead of the deployment holding the cache. The CI stack logs to Datadog, and the Datadog logger truncated messages and response on the standard logging payload in place, so the prompt caching check that reads that same payload after it never wrote its pin. That is LIT-7684, fixed here in cae009c and shown below

Datadog A/B on rig H: old tree 036a380 against cae009c

Rig H: two proxies (:23980 and :58063, --num_workers 1 each) behind nginx on :47925, one Postgres and one Redis, litellm_settings.callbacks: ["datadog"] with a dummy DD_API_KEY (Datadog answers 403, which the logger logs and drops, so the truncation path runs on every call), router_settings.optional_pre_call_checks: ["prompt_caching"]. The driver registers vertex_ai/claude-haiku-4-5 at weight 1 (Vertex-hosted because the direct Anthropic key had hit its credit floor that evening), writes the cache through replica A, reads the Redis deployment:*:prompt_caching pins, registers openai/gpt-5.5 at weight 20 into the same group, then sends the same conversation to replica B, replica A, and nginx. Usage dicts below are trimmed to the token counts that matter

Old tree (036a380), Redis never gets a pin and the follow-ups drift to the heavier deployment:

cached deployment 1d499b5b-0611-49d1-961c-36bb6517fd10 in us-east5
first (replica A): status=200 model_id=1d499b5b-... prompt_tokens 12612, cache_creation_input_tokens 12601
redis prompt_caching pins: 0 -> []
heavyweight deployment 040052fd-c608-48a5-b492-8a93358a38df
follow-up 0 (replica B): status=200 model_id=1d499b5b-... prompt_tokens 12615, cached_tokens 12601 -> PINNED
follow-up 1 (replica A): status=200 model_id=040052fd-... prompt_tokens 12017, cached_tokens 0 -> NOT pinned
follow-up 2 (nginx): status=200 model_id=040052fd-... prompt_tokens 12017, cached_tokens 11008 (OpenAI's own cache) -> NOT pinned
redis prompt_caching pins: 0 -> []

cae009c on the same rig, the pin lands and every follow-up reads the Anthropic cache:

cached deployment 4784b32d-f479-41da-a80e-090f099148cb in us-east5
first (replica A): status=200 model_id=4784b32d-... prompt_tokens 12612, cache_creation_input_tokens 12601
redis prompt_caching pins: 1 -> ['{"model_id": "4784b32d-f479-41da-a80e-090f099148cb"}']
heavyweight deployment a60fc0dd-c06d-4628-9dda-d1beaa0d766a
follow-up 0 (replica B): status=200 model_id=4784b32d-... prompt_tokens 12615, cached_tokens 12601 -> PINNED
follow-up 1 (replica A): status=200 model_id=4784b32d-... prompt_tokens 12615, cached_tokens 12601 -> PINNED
follow-up 2 (nginx): status=200 model_id=4784b32d-... prompt_tokens 12615, cached_tokens 12601 -> PINNED
redis prompt_caching pins: 1 -> ['{"model_id": "4784b32d-f479-41da-a80e-090f099148cb"}']

The fix makes truncate_standard_logging_payload_content return a copy that carries exactly the caller's keys, with only the fields over 10,000 characters among error_str, messages, and response replaced; Datadog logs the copy and every callback after it reads the payload the request built. Two alternatives were considered and set aside: running Datadog last in the callback order, which only moves the bug to whichever callback an operator adds after it, and deep-copying the whole payload inside the Datadog logger, which copies every payload on every call to protect three fields. The 10,000-character threshold and the three fields are unchanged; the helper's return value is the one contract change (LIT-7684)

The 18 router cells again at the cae009c and 6db93a9 tree

Same rig H, E2E_PROMPT_CACHING_STACK=1, both replicas plus nginx, run from the working tree that became those two commits (~/.claude/detached/LIT-5731-after.log, started 2026-09-13T00:28Z):

collected 18 items
16 passed, 2 skipped in 649.53s (0:10:49)
slowest: 5xx cooldown 77.89s, auth cooldown 62.80s, 429 cooldown 62.00s, timeout cooldown 61.90s, 5xx retry 46.81s, shuffle 45.15s

The two skips are the latency-based and least-busy cells (LIT-7682). The prompt-cache cell is not among the 18: its local run at 22:22Z on the old tree failed at the cache-writing call with Anthropic's Your credit balance is too low to access the Anthropic API 400 before reaching the assertion it exists for, and build 483 at the tip ran it on the stack, where E2E_PROMPT_CACHING_STACK selects it, and it passed on the first attempt (test_cached_conversation_stays_on_deployment_holding_its_cache, gw6). 3d22ee8, the tip, changes only how the helper treats a payload missing one of the three keys (a KeyError the required integrations / Run tests job caught at cae009c) and is covered by the 222 tests in tests/logging_callback_tests/test_standard_logging_payload.py and tests/test_litellm/integrations/datadog at the tip, so the cells' behavior at the tip is this tree's

Blast radius of the Datadog change (/live-pr-risk at 3d22ee8)

Changed surface: CustomLogger.truncate_standard_logging_payload_content now returns a new payload instead of mutating and returning its argument, and the private _truncate_field takes the field value instead of the payload and key. Dependents: one in-repo caller, litellm/integrations/datadog/datadog.py:566, which now logs the returned copy; no subclass overrides either method and nothing reaches them by name (grep over litellm/, enterprise/, and tests/). Downstream of the change, every callback that runs after Datadog on the same request now sees the payload the request built: verified live on rig H for the prompt caching pre-call check (the A/B above) and covered on the Datadog side by test_datadog_payload_truncation_leaves_shared_payload_intact plus the 222 tests named above. Not verified: Datadog ingestion itself (rig H used a dummy key, so Datadog answered 403 and nothing was read back from a Datadog account), the stage-mirror Run changed e2e tests job (waiting on an environment approval), and a laptop run of the prompt-cache pytest cell at the tip (build 483 below is its run)

Registry

1. Same collector on this branch without E2E_PROMPT_CACHING_STACK (a laptop run, where the prompt-cache test is deselected and its cell does not count): cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format text | grep -i reliability

Reliability & Performance            31/38       81.6%

2. Same command with E2E_PROMPT_CACHING_STACK=1, which project-releaser (User Request) Expose log probs for hf calls  #223 now sets on both pipelines' stacks

Reliability & Performance            32/38       84.2%

The six rows still open: the latency-based and least-busy cells (skipped, LIT-7682), the LLM complexity classifier (its own ticket), the context-window retry (the router never retries a 400, tracked in LIT-7036), the Redis circuit breaker, and the throughput SLO. Staging without this PR (a969319, the same collector in a throwaway worktree) reads 20/38 either way

CI

1. Buildkite e2e-tests build 9396 on 308f66f triggered litellm-e2e-pr build 245: all 13 new reliability cells passed on the first attempt with no reruns and COVERAGE_MODULE module=reliability_performance read 32/36 (88.9%). Build 245 still ended red (9 failed, 767 passed, 6 errors): the prompt-cache cell waited on project-releaser (User Request) Expose log probs for hf calls  #223 (optional_pre_call_checks=[] on the stack at the time), and every other failure (six test_batch_lifecycle cases, the semantic auto-router /v1/responses case, the one-delta messages stream case, and the six test_model_access_group_e2e.py setup errors) reproduced on the staging scheduled build litellm-e2e Standardize usage of Token / key + accept both  #125 without this PR and ran before the first reliability test started. Build 240 on cb291b4 had four of this PR's own cells failing, fixed by 7c85be2 and 308f66f
2. Buildkite e2e-tests build 9448 on 6de9087 triggered litellm-e2e-pr build 247 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/247): all 13 new reliability cells passed on the first attempt, the prompt-cache test was deselected (no E2E_PROMPT_CACHING_STACK on the stack yet), and the coverage line read 31/36. Build 247 ended red (9 failed, 766 passed, 55 skipped, 6 errors) on the same shared set as 245 and staging Standardize usage of Token / key + accept both  #125 plus one together_ai response-format case, none in this PR's files. buildkite/e2e-tests is not among the required checks on main, the base since the 2026-09-13 default-branch swap, so a red suite does not block a merge on its own; this PR should still land only with its own cells green on its tip
3. Buildkite e2e-tests build 9493 on 038a7c6 triggered litellm-e2e-pr build 248 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/248): 10 failed, 765 passed, 55 skipped, 6 errors in 23:07. Twelve of the 13 reliability cells passed on the first attempt and the coverage line again read 31/36 with the prompt-cache cell deselected. The one failure in this PR's files was test_timeout_trips_cooldown_then_recovers (gw4): 18.5s into the 30s cooldown a replica answered a 408 from the benched deployment, the in-memory eviction plus 10s Redis re-read throttle reproduced on LIT-7041 and since fixed by fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025, which gave cooldowns their own cache with a 1s re-read. The other nine failures and six errors were the shared set from 247 and staging Standardize usage of Token / key + accept both  #125 plus an OTel export deadline on the shared stack, none in this PR's files
4. The e2e-tests gate on 036a380 triggered litellm-e2e-pr build 464 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/464): 3 failed, 963 passed, 58 skipped in 31:51, the three failures being the latency-based, least-busy, and prompt-cache cells covered in round 3 above; the timeout cooldown cell passed with fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025 in. COVERAGE_MODULE module=reliability_performance percent=89.5 covered=34 total=38 counts the three red cells, since the collector counts a collected cell whether or not it passed
5. The e2e-tests gate on 3d22ee8 (build 12489, retried once after a transient project-releaser clone failure) triggered litellm-e2e-pr build 483 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/483): 1 failed, 960 passed, 60 skipped, 3 errors, 1 rerun in 39:54, and COVERAGE_MODULE module=reliability_performance percent=84.2 covered=32 total=38, the two skipped cells no longer counted. Every reliability cell passed on the first attempt except those two skips, the prompt-cache affinity cell included (it ran with E2E_PROMPT_CACHING_STACK set). The one failure is router/test_reliability_memory_e2e.py::TestReliabilityMemory::test_failing_requests_do_not_grow_rss_or_stored_request, a file this PR does not touch (identical on staging, from test(e2e): memory regression test for failing requests on the release gate #40773): its second batch of 300 failing requests grew the single worker's RSS by 59.8 MB against a 48 MB budget while its deterministic stored-request check passed. The same cell failed at 98.6 MB on build 458 for PR feat(memory): add automatic gateway memory with admin policies #40844, which touches neither custom_logger.py nor the Datadog logger, and passed on build 464 for this branch, so the reading is the shared stack's variance rather than this PR's Datadog change (a follow-up tracks the flake). The retried gate step ran the suite again as build 488 (https://buildkite.com/berriai-1/litellm-e2e-pr/builds/488): 2 failed, 962 passed, 60 skipped in 39:14 with the same 32 of 38 line; the memory cell passed and the 3 errors did not recur, and the two failures are claude_code/tool_search/test_bedrock_invoke.py's two tests on a Bedrock 503 for claude-haiku-4-5-bedrock-invoke, outside this PR's files and green on build 483 at the same tip. The 3 errors are router/test_auto_router_regressions_e2e.py cases outside this PR's files: one nginx 504 and two 403s from the auto-router license cap

GitHub Actions at 3d22ee8: every required check passes, and still does on main after the landing watch retargeted the PR there at 04:12Z on 2026-09-13. integrations / Run tests failed once at cae009c on a KeyError the first version of the helper raised for a payload missing one of the three keys, fixed in 3d22ee8 with a regression test; proxy-behavior failed once at 6db93a9 and passes at the tip, on staging, and locally. CircleCI pipeline 89540 at the tip: two jobs are red and neither is required. ocr_testing dies with the same 50-line coverage: failed workers / exit status 123 crash on staging pipelines 89523 and 89533 and on every other branch's pipeline that day (#40926 closed unmerged). logging_testing fails test_generic_api_callback.py::test_generic_api_callback_sumologic_uses_ndjson with Expected 2 lines of NDJSON, got 4 on both PR pipelines (89538, 89540) and passes on staging 89533 and on the other branches' pipelines that day; the failed test's captured stdout shows test_custom_callback_router.py's log_pre_api_call handler still running inside it, so the xdist worker carried another file's callback registration into the test (a local replay of that worker's 41-test order at the tip fails the same way with got 6, fails identically with the two source files checked out from staging, and the bisect pins the extra lines on test_bedrock_knowledgebase_hook.py's seven tests, the only preceding file that reproduces it ahead of the sumologic test, which passes alone and after every other file). The generic API logger never calls the helper this PR changed (its only caller is datadog.py:566) and the PR's two new logging tests register no global callbacks. A rerun from failed at 03:04Z (workflow dc284472) left both red the same way, ocr_testing with 0 tests reported and logging_testing on the same test with got 4. The stage-mirror job waits on the e2e-changed environment review, and Validate PR title shows one run cancelled by concurrency with five passing runs at the same sha after it

Closing observations:

• Cooldowns now reach sibling replicas within about 1s (fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025); the cells keep the 15s window and a follow-up adds the tight cell
• Least-busy counts are per process; PR leaves alone (LIT-7039)
• Metadata dict race 500s a fresh success; PR leaves alone (LIT-7042)
• Latency selection divides by an empty sample list; PR leaves alone (LIT-7053)
• Per-request strategy overrides route without their signal since fix(router): keep per-request routing_strategy override selectors out of global callbacks #40229; the two cells are skipped (LIT-7682)
• Datadog truncation rewrote the shared logging payload; fixed here (LIT-7684)
• OpenAI-family deployments never pin a cached conversation because cache_control is stripped in place; PR leaves alone (LIT-7683)

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Medium

• Every cooldown cell benches through an explicit per-deployment allowed_fails_policy with zero tolerance, so the router-level allowed_fails counter path is not driven: that counter is per worker (router.py:921), and the 5xx cell's error text relays a nested APIConnectionError that the default gate skips (cooldown_handlers.py:222) (LIT-7070)
• The cooldown tests keep a 15s propagation window although fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025 brought sibling propagation down to about 1s, so a regression back toward the old 10s lag passes them; a follow-up adds the tight cell
• The latency-based and least-busy cells are skipped, not passing: since fix(router): keep per-request routing_strategy override selectors out of global callbacks #40229 a per-request routing_strategy override builds its selector with register_callbacks=False, so those strategies route without their signal and their registry rows count as open (LIT-7682); the empty-sample race (LIT-7053) and the per-process counts (LIT-7039) those cells tolerated are moot while they are skipped
• A metadata dict race 500s a successful call right after a bench, twice in 104 calls here; reruns on 5xx absorb it (LIT-7042)
• CustomLogger.truncate_standard_logging_payload_content now returns a new payload instead of the one passed in, so a subclass outside this repo that called it for the side effect and kept reading its own argument sees untruncated fields from now on; the in-repo caller is Datadog alone
• The prompt-cache pytest cell has not passed on a laptop at the tip: its 22:22Z run on the old tree hit Anthropic's credit floor before the assertion, and the fixed tree's evidence is the Vertex-hosted Claude A/B on rig H plus CI build 483
• A cached conversation on an OpenAI-family deployment never pins at all, because cache_control is stripped from the messages in place before the check reads them; the cell uses an Anthropic deployment and LIT-7683 tracks the rest

Low

• Each cooldown test runs about 40s, so the router suite takes ~3 minutes longer
• buildkite/e2e-tests is not a required check on main, so a red e2e suite does not block a merge by itself
• GET /router/settings reports the config the proxy loaded, not live router state, so the prompt-cache precondition trusts the config
• Every per-deployment cooldown_time: 30 equals the router-level default, so the per-deployment knob itself is not what these tests prove; a different value would need recovery-timing asserts that the 15s window makes coarse
• The 5xx cooldown cell's nested upstream counts failures per process against allowed_fails: 5, so a stack with more workers than replicas could bench the upstream later than the test's inner window expects
• The 15s propagation window predates fix(router): give cooldowns their own cache so siblings see a bench in ~1s #40025's 1s cooldown re-read and is now generous; a follow-up tightens it
• Error responses carry no model id header, so the cooldown cells' recovery check takes any answer with the benched deployment's failure status as traffic reaching it; an unrelated 500 inside the 10s recovery window would pass that check early
• The content-policy fallback cell depends on Azure's content filter refusing the fixed prompt with a 400 on every run
• The mutation legs ran at 6de9087 and 7c85be2; the tip's diff over them (registry exercised_on trim, two dead test-model fields, the 12s to 15s window, the two LIT-7682 skips, the cooldown cache docstring, and the Datadog fix with its regression tests) touches nothing those legs assert
• _truncate_field on CustomLogger now takes the field value and returns the truncated text or None, a private signature change with no caller outside the class
• The 17 registry rows the new cells cover now claim chat_completions only; the messages claims they carried had no cell behind them, and the collector reads exercised_on for nothing
• The reliability memory cell (from test(e2e): memory regression test for failing requests on the release gate #40773, not in this PR's diff) failed build 483 at the tip at 59.8 MB against its 48 MB budget and passed build 488; build 458 for an unrelated PR failed it at 98.6 MB and seven other builds passed it, so it reads as shared-stack variance and a follow-up tracks it

QA runbook

Prerequisites: a proxy with OPENAI_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, and AZURE_API_BASE (an Azure OpenAI resource serving gpt-5.4-nano) in its environment, gpt-5.5 in its model list, router_settings.optional_pre_call_checks: ["prompt_caching"], litellm_settings.callbacks: ["datadog"] with any DD_API_KEY (the Datadog leg needs the truncation path to run), and LITELLM_PROXY_URL pointing at it (the 500 and 429 deployments call back into that same URL). Every step below uses the master key for /model/new, /model/delete, /key/generate, and /router/settings, and a fresh all-models key for /v1/chat/completions. x-litellm-model-id on a response names the deployment that served it

• tests/e2e/router/test_reliability_fallbacks_e2e.py::TestReliabilityFallbacks::test_content_policy_routes_to_fallback - a policy refusal reaches the customer as a 400 without a reroute, and content_policy_fallbacks reroutes it to a completion

  • POST /model/new for a group with one deployment: azure/gpt-5.4-nano, api_base: os.environ/AZURE_API_BASE, api_key: os.environ/AZURE_API_KEY, api_version: 2024-10-21, max_retries: 0
  • POST /v1/chat/completions on that group with the DAN jailbreak prompt from CONTENT_POLICY_PROMPT and expect a 400 from the Azure content filter
  • Send it again with "router_settings_override": {"content_policy_fallbacks": [{"<group>": ["gpt-5.5"]}]} and expect a 200 with assistant content (or finish_reason: length with completion tokens spent on reasoning)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_timeout_on_first_deployment_succeeds_on_retry - a deployment that always times out is retried onto its healthy sibling and the customer still gets a completion

  • POST /model/new twice for one group: openai/gpt-5.5 with timeout: 0.001, weight: 1, and model_info.allowed_fails_policy: {"TimeoutErrorAllowedFails": 0}; then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 2} and expect a 200 whose x-litellm-attempted-retries header is at least 1 and whose body has assistant content
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_5xx_on_first_deployment_succeeds_on_retry - a deployment that always answers 500 is retried onto its healthy sibling

  • POST /model/new for an upstream group with one deployment: openai/gpt-5.5 at api_base: http://127.0.0.1:9/v1
  • POST /model/new twice for the group under test: openai/<upstream group> at api_base: $LITELLM_PROXY_URL/v1 with the fresh key as api_key, max_retries: 0, extra_body: {"router_settings_override": {"num_retries": 0}}, weight: 1, allowed_fails_policy: {"InternalServerErrorAllowedFails": 0}; then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 2} and expect a 200 with x-litellm-attempted-retries at least 1 and assistant content
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_429_on_first_deployment_succeeds_on_retry - a deployment whose upstream is out of rpm answers 429 and is retried onto its healthy sibling

  • POST /key/generate with {"models": ["gpt-5.5"], "rpm_limit": 1}
  • POST /model/new twice for the group: openai/gpt-5.5 (the real gpt-5.5 group) at api_base: $LITELLM_PROXY_URL/v1 with that key as api_key, max_retries: 0, extra_body: {"router_settings_override": {"num_retries": 0}}, weight: 1, allowed_fails_policy: {"RateLimitErrorAllowedFails": 0}; then openai/gpt-5.5 with weight: 0
  • Spend the key's one request with a POST /v1/chat/completions on gpt-5.5, expecting a 200 (this opens its 60 second rpm window, so it comes after the registrations)
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 2} inside the same minute and expect a 200 with x-litellm-attempted-retries at least 1 and assistant content
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_retries_e2e.py::TestReliabilityRetries::test_auth_failure_on_first_deployment_succeeds_on_retry - a deployment with a bogus key answers 401 and is retried onto its healthy sibling

  • POST /model/new twice for one group: openai/gpt-5.5 with api_key: sk-not-a-real-key, max_retries: 0, weight: 1, allowed_fails_policy: {"AuthenticationErrorAllowedFails": 0}; then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 2} and expect a 200 with x-litellm-attempted-retries at least 1 and assistant content
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_5xx_trips_cooldown_then_recovers - a 500 benches its deployment on every replica within 15 seconds, keeps it benched through its 30 second cooldown, and the router tries it again afterwards

  • Register the same upstream group and pair as the 5xx retry test, adding cooldown_time: 30 to the failing deployment
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 0} and expect a 500 (the trip)
  • For the next 15 seconds send that request every 250ms: each answer must be a 500 or a 200 whose x-litellm-model-id is the weight-0 backup's id, and at least one must be that 200 (a replica that had just re-read Redis keeps sending the deployment traffic until it looks again, at most 10 seconds later)
  • From 15 to 26 seconds after the trip expect every answer to be a 200 from the backup's id, whichever replica takes it
  • Then send it once a second and expect a 500 again by 40 seconds after the last 500 seen during propagation (each one re-arms the 30 second cooldown, and the failing deployment is back in rotation once it lapses)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_429_trips_cooldown_then_recovers - a 429 benches its deployment on every replica within 15 seconds, keeps it benched through its 30 second cooldown, and the router tries it again afterwards

  • Generate the rpm-1 key and register the same pair as the 429 retry test, adding cooldown_time: 30 to the failing deployment
  • Spend the key's one request with a POST /v1/chat/completions on gpt-5.5, expecting a 200, right before the trip so its 60 second window outlasts the bench
  • POST /v1/chat/completions on the group with "router_settings_override": {"num_retries": 0} and expect a 429 (the trip)
  • For the next 15 seconds send that request every 250ms: each answer must be a 429 or a 200 whose x-litellm-model-id is the weight-0 backup's id, and at least one must be that 200 (a replica that had just re-read Redis keeps sending the deployment traffic until it looks again, at most 10 seconds later)
  • From 15 to 26 seconds after the trip expect every answer to be a 200 from the backup's id, whichever replica takes it
  • Then send it once a second and expect, by 40 seconds after the last 429 seen during propagation, a 429 again or a 200 whose x-litellm-model-id is the failing deployment's id (it is back in rotation, and its key's minute may be up)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_auth_failure_trips_cooldown_then_recovers - a 401 benches its deployment on every replica within 15 seconds, keeps it benched through its 30 second cooldown, and the router tries it again afterwards

  • Run the 5xx cooldown cell's five steps with the auth retry test's pair, cooldown_time: 30 on the bogus-key deployment, and 401 in place of 500 throughout
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_cooldowns_e2e.py::TestReliabilityCooldowns::test_timeout_trips_cooldown_then_recovers - a timeout benches its deployment on every replica within 15 seconds, keeps it benched through its 30 second cooldown, and the router tries it again afterwards

  • Run the 5xx cooldown cell's five steps with the timeout retry test's pair, cooldown_time: 30 on the 1ms deployment, and 408 in place of 500 throughout
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_simple_shuffle_honors_weights - simple-shuffle always opens on the deployment holding all the weight

  • POST /model/new twice for one group: openai/gpt-5.5 with weight: 1, then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions three times with "router_settings_override": {"routing_strategy": "simple-shuffle"} and expect every x-litellm-model-id to be the weight-1 deployment
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_cost_based_picks_cheapest_deployment - cost-based routing sends every call to the cheaper deployment even though the shuffle weight points at the pricier one

  • POST /model/new twice: openai/gpt-5.5 with weight: 1, input_cost_per_token: 0.001, output_cost_per_token: 0.001; then openai/gpt-5.5 with weight: 0, input_cost_per_token: 1e-9, output_cost_per_token: 1e-9
  • POST /v1/chat/completions three times with "router_settings_override": {"routing_strategy": "cost-based-routing"} and expect every x-litellm-model-id to be the cheap deployment
  • Send one more with "routing_strategy": "simple-shuffle" and expect the pricey deployment (proves it was healthy, so the cheap picks were the strategy's doing)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_usage_based_picks_deployment_with_tpm_headroom - usage-based routing skips the deployment whose tpm cap no prompt fits

  • POST /model/new twice: openai/gpt-5.5 with weight: 1, tpm: 1; then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions three times with "router_settings_override": {"routing_strategy": "usage-based-routing-v2"} and expect every x-litellm-model-id to be the uncapped deployment
  • Send one more with "routing_strategy": "simple-shuffle" and expect the capped deployment
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_latency_based_routes_around_deployment_that_times_out - latency-based routing routes around the deployment that times out on every call it gets (skipped under LIT-7682 until per-request overrides carry their callbacks; the steps describe the behavior the router should have)

  • POST /model/new twice: openai/gpt-5.5 with weight: 1 and timeout: 0.001, then openai/gpt-5.5 with weight: 0
  • POST /v1/chat/completions with "router_settings_override": {"routing_strategy": "latency-based-routing", "num_retries": 0} up to 12 times: every answer is either a 408 (the 1ms deployment) or a 200 whose x-litellm-model-id is the weight-0 deployment, and you stop once you have seen a 408 followed by three of those 200s in a row (each proxy process meets the slow deployment at most once, so the CI stack's two replicas give at most two 408s)
  • Send one more with "routing_strategy": "simple-shuffle", "num_retries": 0 and expect a 408 (the slow deployment was still routable, so the fast picks were latency's doing and not a cooldown's)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_routing_strategies_e2e.py::TestReliabilityRoutingStrategies::test_least_busy_avoids_deployment_with_request_in_flight - least-busy routing avoids the deployment with a long stream in flight (skipped under LIT-7682 until per-request overrides carry their callbacks; the steps describe the behavior the router should have)

  • POST /model/new four times into a fresh group, each openai/gpt-5.5 with weight: 1
  • Open POST /v1/chat/completions with "stream": true, "max_tokens": 3000, a 1500-word essay prompt, and "router_settings_override": {"routing_strategy": "least-busy"}; read only the response head (200) and its x-litellm-model-id, and leave the body unread (the stream goes through least-busy because a proxy process only starts counting in-flight requests once it has routed a least-busy request)
  • While it streams, send three short calls with "routing_strategy": "least-busy" and expect every x-litellm-model-id to be one of the other three deployments (a process can still count the previous call or two against whichever deployment took them, but with three idle deployments one is always at zero, strictly below the one holding the stream); then drain the stream and expect it to end with data: [DONE] and no truncation, which proves the deployment holding it stayed healthy
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/router/test_reliability_prompt_caching_e2e.py::TestReliabilityPromptCachingAffinity::test_cached_conversation_stays_on_deployment_holding_its_cache - a conversation that wrote a prompt cache keeps landing on the deployment holding it, over a much heavier newcomer

  • GET /router/settings and expect current_values.optional_pre_call_checks to contain prompt_caching (the test fails naming the setting otherwise)
  • Confirm litellm_settings.callbacks includes datadog, so the truncation path runs before the prompt caching check on every call (the bug this PR fixes only shows with a logger that truncates the shared payload)
  • POST /model/new for a group with one deployment: anthropic/claude-haiku-4-5, weight: 1
  • POST /v1/chat/completions with a system turn of about 600 filler clauses carrying "cache_control": {"type": "ephemeral"} plus a short user turn; expect a 200 from that deployment with usage.cache_creation_input_tokens > 0
  • POST /model/new for the same group: openai/gpt-5.5 with weight: 20
  • Send three follow-ups with the same system turn and expect each x-litellm-model-id to be the Anthropic deployment with usage.cache_read_input_tokens > 0
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 038a7c6 passes /live-pr-risk

• 3d22ee8 passes /live-pr-risk

[greptile-apps]

Greptile Summary

This PR substantially expands live E2E coverage for router reliability behavior and prevents DataDog payload truncation from mutating logging data shared with later callbacks.

• Adds live retry, cooldown, fallback, prompt-cache affinity, and routing-strategy coverage.
• Centralizes deselection of configuration-specific E2E tests behind opt-in environment variables.
• Adds held-stream and router-settings harness support.
• Corrects reliability registry endpoint claims to match the exercised chat-completions coverage.
• Makes logging truncation return an isolated payload and updates DataDog to serialize that copy.

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issue or outstanding previous finding remains.

The control-plane prefix now fixes the earlier split-route issue, the RPM and recovery-deadline findings are resolved, and the two rule findings were correctly retracted. The new logging-copy change preserves DataDog truncation while leaving shared callback data intact, and no mounted-rule violation was found.

Important Files Changed

Filename	Overview
litellm/integrations/custom_logger.py	Returns a shallow payload copy with only oversized logging fields replaced, preserving the shared original object.
litellm/integrations/datadog/datadog.py	Uses the isolated truncated payload when constructing the DataDog event.
tests/e2e/conftest.py	Centralizes environment-controlled deselection for weekly, managed-files, prompt-caching, and Redis-chaos tests.
tests/e2e/transport.py	Adds held-stream transport support and correctly classifies router settings as a control-plane route.
tests/e2e/router/reliability_support.py	Adds shared helpers for deterministic live failures, cooldowns, prompt caching, and streaming requests.
tests/e2e/router/test_reliability_cooldowns_e2e.py	Covers cooldown propagation, continued benching, and recovery while anchoring recovery to the last observed failure.
tests/e2e/router/test_reliability_routing_strategies_e2e.py	Adds strategy-specific live routing cells and explicitly skips two cells whose required callback behavior is currently unavailable.
tests/e2e/coverage_registry/reliability.yaml	Aligns reliability registry endpoint claims and prompt-cache prerequisites with the tests that actually execute.

_{Reviews (6): Last reviewed commit: "fix(logging): keep partial logging paylo..." | Re-trigger Greptile}

[greptile-apps]

Split route misclassified In split deployments, this management request goes to the data plane and fails before prompt-cache routing is tested

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: 429 tests flake after RPM reset
  • Switched both 429 tests to use rpm_limit=0 (which blocks every request in the Lua check current_counter + increment > limit) and removed the prime call, so the failing deployment stays permanently 429 regardless of how long the cooldown recovery poll takes.

Or push these changes by commenting:

@cursor push 2ab8b97db2

Preview (2ab8b97db2)

diff --git a/tests/e2e/router/reliability_support.py b/tests/e2e/router/reliability_support.py
--- a/tests/e2e/router/reliability_support.py
+++ b/tests/e2e/router/reliability_support.py
@@ -6,7 +6,7 @@
 `client.proxy`. Every failure is a real one from a real deployment: a bad base
 URL yields a connection error, a 1ms deadline a timeout, a bogus key a 401, an
 Azure content filter a policy refusal, and a 500 or a 429 comes from this same
-proxy fronting a group that cannot answer (or a key that is out of rpm), so the
+proxy fronting a group that cannot answer (or a key with rpm_limit=0), so the
 outer deployment sees exactly the status a customer's own upstream would send.
 Each test wires its reroute per request through a `router_settings_override` in
 the /chat/completions body, so a single long-lived proxy serves every
@@ -158,7 +158,7 @@
 def _nested_proxy_params(upstream_group: str, upstream_key: str, cooldown_time: float | None) -> LiteLLMParamsBody:
     """A deployment whose upstream is this same proxy serving `upstream_group` with
     `upstream_key`: whatever that group answers (a 500 from an unreachable base, a
-    429 from a key out of rpm) arrives as a real provider status, with the inner
+    429 from an rpm_limit=0 key) arrives as a real provider status, with the inner
     proxy's and the client's own retries off so it arrives at once."""
     return LiteLLMParamsBody(
         model=f"openai/{upstream_group}",
@@ -187,8 +187,8 @@
 def create_always_rate_limited_deployment(
     proxy: ProxyClient, name: str, upstream_group: str, upstream_key: str, cooldown_time: float | None = None
 ) -> str:
-    """Fronts a healthy upstream group with a key that is out of rpm, so every call
-    is a real 429, benched on its first RateLimitError."""
+    """Fronts a healthy upstream group with an rpm_limit=0 key, so every call is a
+    real 429, benched on its first RateLimitError."""
     return _register_benched_on_first_failure(
         proxy, name, _nested_proxy_params(upstream_group, upstream_key, cooldown_time), "RateLimitErrorAllowedFails"
     )

diff --git a/tests/e2e/router/test_reliability_cooldowns_e2e.py b/tests/e2e/router/test_reliability_cooldowns_e2e.py
--- a/tests/e2e/router/test_reliability_cooldowns_e2e.py
+++ b/tests/e2e/router/test_reliability_cooldowns_e2e.py
@@ -21,8 +21,8 @@
 The failures are the same real ones the retry tests use: a 1ms deadline and a
 bogus key on the real backend, and this proxy standing in as the upstream for
 the 500 (fronting a group whose only deployment is unreachable) and the 429
-(fronting a healthy group with a key that already spent its one request per
-minute).
+(fronting a healthy group with a key whose rpm_limit is 0, so every call is
+blocked without depending on a per-minute window that could reopen mid-test).
 """
 
 from __future__ import annotations
@@ -161,19 +161,14 @@
     def test_429_trips_cooldown_then_recovers(
         self, client: ComplexityRouterClient, resources: ResourceManager, scoped_key: str
     ) -> None:
-        spent_key = client.proxy.generate_key(
-            KeyGenerateBody(models=[CHEAP_OPENAI_MODEL], rpm_limit=1, user_id="e2e-test-user")
+        blocked_key = client.proxy.generate_key(
+            KeyGenerateBody(models=[CHEAP_OPENAI_MODEL], rpm_limit=0, user_id="e2e-test-user")
         )
-        resources.defer(lambda: client.proxy.delete_key(spent_key))
-        primed = chat_override(client.proxy, spent_key, CHEAP_OPENAI_MODEL, f"say hi {unique_marker()}")
-        assert primed.status_code == 200, (
-            f"the one request the rpm-limited key allows should have succeeded, got {primed.status_code}: "
-            f"{primed.body[:300]}"
-        )
+        resources.defer(lambda: client.proxy.delete_key(blocked_key))
 
         group = f"reliability-cooldown-429-{unique_marker()}"
         failing = create_always_rate_limited_deployment(
-            client.proxy, group, CHEAP_OPENAI_MODEL, spent_key, cooldown_time=COOLDOWN_SECONDS
+            client.proxy, group, CHEAP_OPENAI_MODEL, blocked_key, cooldown_time=COOLDOWN_SECONDS
         )
         resources.defer(lambda: client.proxy.delete_model(failing))
         backup = create_zero_weight_backup_deployment(client.proxy, group)

diff --git a/tests/e2e/router/test_reliability_retries_e2e.py b/tests/e2e/router/test_reliability_retries_e2e.py
--- a/tests/e2e/router/test_reliability_retries_e2e.py
+++ b/tests/e2e/router/test_reliability_retries_e2e.py
@@ -12,9 +12,10 @@
 The failures are real. A timeout is a 1ms deadline on the real backend and a 401
 is a bogus key on it. A 500 and a 429 come from this same proxy standing in as
 the upstream: the failing deployment fronts a group of this proxy whose only
-deployment is unreachable (a real 500), or a healthy group called with a key that
-has already spent its one request per minute (a real 429), so the router sees the
-same statuses a customer's provider would send.
+deployment is unreachable (a real 500), or a healthy group called with a key
+whose rpm_limit is 0 (a real 429, blocked without depending on a per-minute
+window that could reopen mid-test), so the router sees the same statuses a
+customer's provider would send.
 
 The context-window retry cell has no test on purpose: the router refuses to
 retry a 400-class error, and a context-window refusal is one, so the documented
@@ -111,18 +112,13 @@
     def test_429_on_first_deployment_succeeds_on_retry(
         self, client: ComplexityRouterClient, resources: ResourceManager, scoped_key: str
     ) -> None:
-        spent_key = client.proxy.generate_key(
-            KeyGenerateBody(models=[CHEAP_OPENAI_MODEL], rpm_limit=1, user_id="e2e-test-user")
+        blocked_key = client.proxy.generate_key(
+            KeyGenerateBody(models=[CHEAP_OPENAI_MODEL], rpm_limit=0, user_id="e2e-test-user")
         )
-        resources.defer(lambda: client.proxy.delete_key(spent_key))
-        primed = chat_override(client.proxy, spent_key, CHEAP_OPENAI_MODEL, f"say hi {unique_marker()}")
-        assert primed.status_code == 200, (
-            f"the one request the rpm-limited key allows should have succeeded, got {primed.status_code}: "
-            f"{primed.body[:300]}"
-        )
+        resources.defer(lambda: client.proxy.delete_key(blocked_key))
 
         group = f"reliability-retry-429-{unique_marker()}"
-        failing = create_always_rate_limited_deployment(client.proxy, group, CHEAP_OPENAI_MODEL, spent_key)
+        failing = create_always_rate_limited_deployment(client.proxy, group, CHEAP_OPENAI_MODEL, blocked_key)
         resources.defer(lambda: client.proxy.delete_model(failing))
         backup = create_zero_weight_backup_deployment(client.proxy, group)
         resources.defer(lambda: client.proxy.delete_model(backup))

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_reliability_module_cells (3d22ee8) with litellm_internal_staging (a969319)}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3d22ee8. Configure here.}