docs(auto_routing): document the classifier context window and assistant turns - #747
Conversation
…ant turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| ### Assistant turns in the context window | ||
|
|
||
| `classifier_context_include_assistant_turns` is off by default and puts the model's own replies in the window. It exists for the conversation where difficulty is stated by the assistant rather than by the user: the assistant answers "here is the plan, it is complex, should I execute?", the user answers "yes", and with user turns alone the router rates the word "yes" and picks the cheapest tier. With it on, the classifier rates the work the current message approves, judged in the conversation it continues. | ||
|
|
||
| ```yaml | ||
| classifier_type: llm | ||
| classifier_llm_config: | ||
| model: claude-haiku-4-5-20251001 | ||
| classifier_context_include_assistant_turns: true | ||
| classifier_context_window_size: 3 | ||
| classifier_context_per_turn_chars: 200 | ||
| ``` | ||
|
|
||
| Enabling it changes what `classifier_context_window_size` counts: the last N turns of the conversation across both roles rather than the last N user turns, so budget accordingly if a chatty exchange should still carry several user asks. Turns are labeled by role in the payload only when this is on, which keeps the prompt of every existing deployment unchanged. Assistant replies share `classifier_context_per_turn_chars` with user turns, so raise it if replies truncate before the part that states the difficulty. | ||
|
|
||
| It ships off by default for two reasons: turning it on shifts tier decisions, and therefore spend, on an already-deployed router, and assistant text becomes net-new egress to the classifier deployment, which may be a different provider than the routed model. Assistant text reaches the classifier payload and nothing else; `keyword_tier_rules`, escalation keywords, the heuristic scorer, and semantic matching still read only the human ask, so an assistant echoing an escalation keyword back cannot pick the tier. |
There was a problem hiding this comment.
🟡 Keyword rules description now reads as part of the assistant-turns section
The keyword-rules explanation is left after two newly inserted subsections (### Classifier context window at docs/proxy/auto_routing.md:154 and ### Assistant turns in the context window at line 188), so the third classification method now appears to belong to the assistant-turns topic instead of standing alongside the other two.
Impact: Readers and the page's table of contents present keyword routing as a sub-topic of assistant turns, making one of the three routing methods hard to find.
How the section ordering breaks the three-method structure
The ## Classification section is written as three bolded siblings: Heuristic scorer (docs/proxy/auto_routing.md:131), LLM classifier (line 145), and Keyword rules (line 205). Inserting two ### headings between the LLM classifier paragraph and the keyword-rules paragraph means everything from line 205 onward (including the semantic_keyword_matching YAML at lines 209-218) renders inside the Assistant turns in the context window heading scope. Moving the two new ### subsections after the keyword-rules block, or promoting keyword rules to its own heading, restores the parallel structure.
Prompt for agents
In docs/proxy/auto_routing.md, the `## Classification` section presents three peer methods as bolded lead-ins: heuristic scorer, LLM classifier, and keyword rules. The PR inserts `### Classifier context window` and `### Assistant turns in the context window` between the LLM classifier paragraph and the keyword rules paragraph, which pushes the keyword rules prose and its YAML example under the assistant-turns heading in both the rendered page and the table of contents. Reorder so the keyword rules content stays a peer of the other two methods, for example by placing the two new `###` subsections after the keyword rules block, or by giving each classification method its own `###` heading so nesting is explicit.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
The auto-router page still described the LLM classifier as if it saw the current message alone, so nothing documented
classifier_context_window_size,classifier_context_per_turn_chars, orclassifier_context_include_assistant_turnsfrom #35185 and #35471, both in v1.96.x.Two new subsections under Classification. "Classifier context window" covers the defaults (last 3 qualifying turns, 200 chars each), which turns qualify (human-written user text only; tool output and complete
<system-reminder>blocks never count, and a turn matching the current ask is not quoted twice), the payload shape with the cacheable operator rubric in the system role and caller text quoted in the user role, andclassifier_context_window_size: 0as the off switch. "Assistant turns in the context window" covers the opt-in flag, why it is off by default (it shifts tier decisions and therefore spend on a deployed router, and sends assistant text to the classifier deployment), and that enabling it makes the window count the last N turns across both roles rather than the last N user asks.Two smaller additions: the Full config block now lists the three keys with their defaults, and the UI section notes that all three are exposed under the LLM Classifier radio in Models + Endpoints > Add Model > Auto Router, written only when the classifier type is LLM.
The window interacts with
session_affinity, now off by default on main: a router that turns affinity on only classifies turn one, so the window applies there and on requests with no resolvablesession_id, while a default router classifies every turn and the window applies throughout.classifier_tier_rubricfrom #35471 is deliberately not documented; it was removed again in #35504 before the release.Validated with
npm run build; no new warnings.Link to Devin session: https://app.devin.ai/sessions/1f8ebd08f159482b930fc1639e741cdb