Skip to content

docs(auto_routing): session_affinity is on by default #583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
krrish-berri-2 merged 1 commit into from
Jul 16, 2026
Merged

docs(auto_routing): session_affinity is on by default #583

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/proxy/auto_routing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,14 +172,16 @@ A tier value can be a single model name or a list.

## Session affinity

Opt-in. Pins the first-turn model for a session and skips reclassification on later turns, so provider-side prompt caches keyed to that model do not get invalidated when a follow-up ("thanks!") would otherwise classify into a different tier.
On by default. Pins the first-turn model for a session and skips reclassification on later turns, so provider-side prompt caches keyed to that model do not get invalidated when a follow-up ("thanks!") would otherwise classify into a different tier. It also keeps a multi-turn session on a single model, which avoids provider errors when conversation history produced by one model (for example an Anthropic `thinking` block) is replayed to a different model on a later turn.

Set `session_affinity: false` to reclassify every turn instead.

```yaml
session_affinity: true
session_affinity: true # default; set false to reclassify every turn
session_affinity_ttl_seconds: 3600
```

`session_id` is read from request metadata. When `adaptive: true` is also set, a pinned turn still stamps the adaptive bandit's chosen-model metadata key so reward feedback keeps working.
`session_id` is read from request metadata; when no `session_id` is resolvable the router classifies every turn as usual. When `adaptive: true` is also set, a pinned turn still stamps the adaptive bandit's chosen-model metadata key so reward feedback keeps working.

## Custom technical keywords

Expand Down