feat(spend): tell a mid-conversation switch from a conversation's first turn - #35510
feat(spend): tell a mid-conversation switch from a conversation's first turn#35510tin-berri wants to merge 1 commit into
Conversation
…st turn The auto-router savings number reads a cold cache the same way whichever reason it is cold for, and the two want opposite arithmetic. A switch leaves the new model cold, so the write is a cost the switch caused and the baseline, already warm on the model it never left, would have paid only a read. A first turn has nothing cached anywhere, so the baseline would have written the same prompt and the write belongs on both arms. Charging every cold cache as a switch, which is all the rollup row could support, understated a genuine first turn badly: on a 20k prompt with a 1k completion it reported +$0.005 against a true +$0.12, and a whole conversation only converged as it lengthened, reaching half its real value at five turns. Worse, the write premium is fixed by prompt size while the saving grows with completion length, so under roughly 750 completion tokens on a 20k prompt the understated number crossed zero and a profitable route rendered as a loss. The discriminator is what the session was served last. The complexity router records the model it picked against the proxy-derived session id, reads it back on the next turn and carries it on the routing decision, where the existing per-attempt writer records it with the baseline so a fallback clears both together. Three states, not two: no session at all keeps the conservative rule and under-claims, a tracked session with nothing recorded is a real first turn, and a tracked session served something else before is the switch. Session identity moves to router_utils/session_identity.py, since the router now needs the same reader the complexity router already had. That reader is wider than its docstring claimed: the proxy writes the id it derives from an `x-*-session-id` header or from Anthropic's `metadata.user_id` into the same `metadata.session_id` the complexity router was reading, so header-carrying clients were always covered. Observation only. Nothing is pinned, no candidate pool narrows, and `session_affinity` is untouched; a request naming no session never touches the cache at all.
Greptile SummaryThis PR distinguishes first turns from mid-conversation model switches when calculating auto-router savings by persisting each tracked session's previously selected model.
Confidence Score: 4/5The PR should not merge until cache failures remain conservative and session state is updated only for models that actually serve a request. The new discriminator can overstate savings when a cache failure masquerades as a first turn, and an unsuccessful routed attempt can poison the previous-model state used by the next request. Files Needing Attention: litellm/router_strategy/complexity_router/complexity_router.py
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/complexity_router/complexity_router.py | Adds previous-model cache reads and writes, but read failures become false first turns and models are persisted before successful service. |
| litellm/proxy/spend_tracking/savings.py | Adds the three-state switch discriminator and adjusts cache-write baseline pricing; its correctness depends on reliable upstream session state. |
| litellm/router_utils/session_identity.py | Centralizes session and authenticated-caller extraction and scopes cache keys by router namespace, caller, and session. |
| litellm/router.py | Propagates and clears the new routing metadata fields across attempts. |
| litellm/proxy/db/db_spend_update_writer.py | Passes the new previous-model and session-tracked metadata into savings computation. |
| tests/test_litellm/router_strategy/test_complexity_router.py | Expands cache and session-state coverage but does not cover cache failures or terminally failed provider attempts. |
| tests/test_litellm/proxy/spend_tracking/test_savings.py | Covers first-turn, switch, same-model, unresolved-model, and untracked-session arithmetic. |
Reviews (1): Last reviewed commit: "feat(spend): tell a mid-conversation swi..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| try: | ||
| previous_model = await self.litellm_router_instance.cache.async_get_cache(key=cache_key) | ||
| except Exception as e: # noqa: BLE001 # a dashboard metric must not fail a live request | ||
| verbose_router_logger.debug("complexity router: could not read the session's previous model (%s)", e) | ||
| return None | ||
| return previous_model if isinstance(previous_model, str) else None |
There was a problem hiding this comment.
When a tracked session's cache read fails, _previous_model_for_session returns the same None used for a cache miss while session_tracked remains true. The savings calculation therefore treats unknown session history as a confirmed first turn instead of applying the conservative switch calculation, causing autorouter_savings_spend to be overstated.
Knowledge Base Used:
| if response is not None: | ||
| await self._remember_model_for_session(previous_model_key, response.model) |
There was a problem hiding this comment.
When a selected provider attempt fails before serving a response, this pre-routing write still records its model as the session's last-served model. After a terminal failure, no later hook corrects the entry, so the next request compares against a model that never served the conversation and reports incorrect auto-router savings.
Knowledge Base Used:
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| input=input, | ||
| specific_deployment=specific_deployment, | ||
| previous_model=previous_model, | ||
| session_tracked=previous_model_key is not None, |
There was a problem hiding this comment.
Low: Caller-controlled session IDs can inflate savings
previous_model_key exists whenever the caller supplies a session ID, so rotating IDs makes every request a tracked cache miss with previous_model=None. The savings code interprets that state as a genuine first turn and applies the larger calculation, allowing an authenticated caller to inflate dashboard savings. Treat cache misses conservatively unless the session identity and first-turn state are server-issued or otherwise verifiable; misses can also result from eviction or cache restarts.
PR overviewThis PR updates the complexity router’s spend tracking to distinguish a conversation’s first turn from a model switch later in the conversation. That distinction is used when calculating reported savings. One issue remains open: authenticated callers can manipulate session identifiers to make requests appear to be first turns, inflating dashboard savings figures. The impact is limited to the integrity of savings reporting, but no issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 5/10 |
1 participant
TLDR
Problem this solves:
How it solves it:
Relevant issues
d539fa6, including the region where the number goes negativelitellm_pre_call_utils, so no caller has to opt inLinear ticket
Resolves LIT-5087
Pre-Submission checklist
Screenshots / Proof of Fix
Not yet captured against a live proxy, and the gap is stated rather than glossed. The pricing arithmetic and the routing-side recording are covered by unit tests only; what is still owed is real auto-routed traffic across two turns of one session, then
/user/daily/activityshowing the first turn's largerautorouter_savings_spend, then the dashboard athttp://localhost:4000/ui/?page=cost-optimization. The available Anthropic key is out of credit, which is the same blocker #35402 is carrying.The numbers the change turns on, computed from the shipped cost map on this branch, 20k prompt and 1k completion,
anthropic/claude-opus-5as baseline againstclaude-haiku-4-5selected:Before this change every one of those requests was priced at the
+0.0050row.Type
🆕 New Feature
Changes
Auto-router savings are measured against what the traffic would have cost on one model, and that counterfactual depends on whether the model had a warm cache. A switch leaves the newly chosen model cold, so the write is a cost the switch caused and the baseline, which never left the model it was on, would have paid only a read. A conversation's first turn has nothing cached anywhere, so the baseline would have written the same prompt and the write belongs on both arms at each model's own rate. The rollup row cannot tell those apart, so every cold cache was charged as a switch.
That understated a genuine first turn to roughly 4% of its value, and a whole conversation only converged as it lengthened, reaching about half its real value at five turns. The sharper edge is that the write premium is fixed by prompt size while the saving grows with completion length, so below roughly 750 completion tokens on a 20k prompt the understated number crossed zero; single-turn traffic with a large cached system prompt and a short answer, an ordinary agent shape, could show the router losing money on requests that each genuinely saved.
The discriminator. What the session was served last is the one bit that answers it. The complexity router records the model it picked against the session id and reads it back on the next turn, carrying it on the routing decision beside the baseline. The existing per-attempt writer records both together, so a fallback that re-enters the hook clears them as a pair rather than leaving one behind.
Three states, not two. An absent previous model means two opposite things, and collapsing them inverts the fallback. A request that named no session has no discriminator at all and keeps the conservative rule, under-claiming rather than inflating a savings figure. A tracked session with nothing recorded is a real first turn. A tracked session whose last model is the one picked now stayed put, which is also not a switch. Only a tracked session served something else before is.
session_trackedtherefore travels withprevious_modelfrom the routing hook through the spend log to the pricing.Where session identity comes from. The proxy already derives it before routing, so no caller opts in:
get_chain_id_from_headerspicks upx-litellm-trace-id,x-litellm-session-idor anyx-<vendor>-session-id, and Anthropic'smetadata.user_idis read for clients that send it natively. Both land inmetadata["session_id"], which is what the complexity router was already reading for session affinity; that reader was never narrower than the proxy-derived signal, contrary to what its docstring implied. The reader moves torouter_utils/session_identity.pynow that more than one caller wants it, and the entry is namespaced by the authenticated key's hash, the same trust boundary session affinity uses, so two callers reusing one session id cannot read each other's models.Scope. Observation only. Nothing is pinned, no candidate pool narrows, and
session_affinityis a separate feature that is unaffected and can stay off. A request naming no session never touches the cache, so direct Router use and header-less callers pay nothing. Every cache access degrades to the conservative rule on failure, because a dashboard's counterfactual is not worth failing a live request over. The semantic auto-router does not set the field and is unchanged.This is stacked on #35402 and is based on that branch, so review the last commit rather than the full diff.
Final Attestation