docs(auto-routing): document tier fallback when a tier cannot serve

blog/autorouter_cost_quality_benchmark/index.md

@@ -34,6 +34,7 @@ Already testing it? Share your results in [discussion #32172](https://github.com
 - **40.4% cheaper at 97.1% of frontier quality**, on 220 prompts from six public benchmarks replayed through a live proxy
 - **74.5% cheaper at 87.3% of frontier quality**, on RouterArena's full 8,399-query set
 - **Around 65% cheaper** on simulated real chat and developer traffic, where most requests are short
+- **46% cheaper than a cached frontier baseline** on multi-turn developer conversations, with prompt caching switched on for both arms
 
 | Evaluation | Sample | Quality retained vs Opus-5 | Cost savings vs Opus-5 | Routing mix (haiku/sonnet/opus) |
 | --- | --- | --- | --- | --- |
@@ -87,11 +88,23 @@ The split runs opposite to intuition:
 
 A question we keep hearing: do I have to choose between auto-routing and prompt caching?
 
-No. Session affinity already lets you keep a conversation on the model its first turn picked, so the cache holds for the rest of that conversation. What it costs you is the routing decision on every turn after the first: once a session is pinned, a one-line follow-up stays on whatever model the opening turn earned. Closing that gap is the first item below.
+No, and on multi-turn traffic you want both. Every benchmark above prices requests from cold, which is the wrong baseline for anyone running long conversations, so we simulated 1,011 multi-turn DevGPT conversations; 11,808 turns, every conversation at least three turns deep, against a baseline that sends each turn to `claude-opus-5` with prompt caching already switched on.
+
+| Configuration | $/1k turns | vs cached Opus-5 |
+| --- | --- | --- |
+| `claude-opus-5`, prompt caching on | 12.85 | baseline |
+| Auto Router, prompt caching off | 12.03 | 6% cheaper |
+| Auto Router, prompt caching on | **6.96** | **46% cheaper** |
+
+The ordering matters more than any single number here. Routing with the cache switched off is close to a wash against cached Opus-5 and loses outright on 31% of conversations, because every turn resends the whole accumulated transcript at full input price and cost climbs with the square of conversation depth. Switch caching back on, hold the routing decisions identical, and the same sessions land 46% under the cached baseline while beating it on 80% of individual conversations. Toggling only the cache moves the bill by 42%, so on multi-turn traffic caching is the larger lever and routing compounds it rather than competing with it.
+
+Session affinity is what lets the two stack. A conversation pins to the model its first turn selected and the prefix stays warm on that model for the rest of the session, while the router keeps scoring each new turn and can move a session up a tier as the work gets harder, never back down; de-escalating repays its cold prefix too slowly to be worth taking. Moving up does start cold on the new model, and 72% of these conversations escalate at least once, so that write is the cost an escalation has to clear before it is worth making.
+
+Two caveats on the 46%. This baseline already has caching switched on, so the number is not comparable with the roughly 65% figures above, which price both arms from cold. And it covers multi-turn conversations only; the deepest 1% of sessions carry enough spend to pull the all-conversation figure down to 28%, since a long session tends to reach the top tier anyway and pays a cold prefix when it gets there.
 
 ## What's next
 
-- **Prompt caching that survives a tier change.** Affinity pins a conversation today because switching models means a cold cache. We are testing a background refresher that keeps the prefix warm on every tier, so a session can move without paying the write cost twice
+- **Keeping a session's caches alive.** A background refresher replays a session's stored prefix against the tiers that session has already used, so an idle gap longer than the provider cache TTL no longer costs the session its cache, and coming back to a tier it has visited is a read rather than a fresh write. The first move to a new tier still starts cold; that write is the cost the escalation decision weighs
 - **Routing decisions you can inspect.** Surfacing why a request landed on the tier it did, not just which model served it. That same signal feeds back into the classifier, so it improves against real traffic rather than benchmarks
 
 ## Try it

docs/proxy/auto_routing.md

@@ -170,6 +170,18 @@ A tier value can be a single model name or a list.
 - **List:** router random-picks per request (uniform), same idea as simple-shuffle. Empty pools raise at config load rather than falling through to `default_model`.
 - **List + `adaptive: true`:** Thompson-sample across the pool. Cold requests sample only inside the classified tier so cost weights do not collapse initial traffic on the cheapest model. Models configured in multiple tiers use their minimum distance from the classified tier. Feedback from a later turn attributes back to the model that actually served the previous response.
 
+## Tier fallback
+
+Classifying a request is half the job; the tier still needs something that can answer it. Resolution takes the first of these that works: a live model in the classified tier, then the next tier up and the one above that, then `default_model`, then the classified tier anyway as a best effort. A pool member counts as live when the router knows that model group and it has at least one deployment out of cooldown, so a dead model is passed over for its peers before the tier itself is given up on.
+
+Resolution never falls to a cheaper tier. That is the model the classifier already ruled out, so a COMPLEX request is not answered by the SIMPLE model just because SIMPLE happens to be healthy. The last step is best effort rather than an error because cooldowns expire and the health view is a snapshot; a request that might succeed is sent instead of failed.
+
+Two facts ride on the routing decision, because neither survives in the routed model name alone. `tier_fallback_from` records the tier the request was classified into whenever the ladder climbed, so a cheap request sitting on an expensive model is explainable after the fact. `resolved_by` records how the model was obtained when it did not come from a tier pool at all: `default_model` when the ladder was exhausted, `best_effort` when nothing reported a live deployment and the tier was served anyway. The two are independent, so an exhausted ladder can carry both, and the ordinary path carries neither.
+
+Two cases are deliberately not routed around. A tier configured as an empty pool (`SIMPLE: []`) raises, since that is a config error rather than a gap to work around. And a routing plugin that narrows a tier to zero candidates raises, since refusing every candidate is a policy decision and climbing past it would serve exactly what the plugin denied; plugins run against whichever tier the ladder settles on, so they always vet what is served. If the health view itself cannot be read, the pool is treated as live and resolution behaves as it did before, rather than declaring every tier dead and pushing traffic to the top.
+
+The adaptive path keeps its own selection, which already scores every pool model with a tier-distance penalty.
+
 ## Session affinity
 
 On by default. Pins the first-turn model for a session and skips reclassification on later turns, so provider-side prompt caches keyed to that model do not get invalidated when a follow-up ("thanks!") would otherwise classify into a different tier. It also keeps a multi-turn session on a single model, which avoids provider errors when conversation history produced by one model (for example an Anthropic `thinking` block) is replayed to a different model on a later turn.
@@ -197,6 +209,8 @@ Every routing decision emits one greppable line naming its cause. `cause=` is gr
 
 ```
 ComplexityRouter: routing decision cause=complexity_scorer,      tier=SIMPLE,     score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini
+ComplexityRouter: routing decision cause=complexity_scorer,      tier=MEDIUM,     score=-0.150, signals=['short (7 tokens)'],                     routed_model=gpt-4o, tier_fallback_from=SIMPLE
+ComplexityRouter: routing decision cause=complexity_scorer,      tier=SIMPLE,     score=-0.150, signals=['short (7 tokens)'],                     routed_model=claude-sonnet-5, resolved_by=default_model
 ComplexityRouter: routing decision cause=literal_keyword_match,  tier=REASONING,                                                                    routed_model=gpt-5.5
 ComplexityRouter: routing decision cause=semantic_keyword_match, tier=REASONING,                                                                    routed_model=gpt-5.5
 ComplexityRouter: routing decision cause=llm_classifier,         tier=COMPLEX,    score=1.000, signals=['llm-classifier:COMPLEX'],                  routed_model=claude-sonnet-5