blog: which LLM should classify your prompts? A 13-way bake-off

[tin-berri]

Adds a blog post sharing the internal classifier bake-off behind the Auto Router: 12 LLMs plus the built-in heuristic, scored on the same 100 stratified prompts with intrinsically-labeled tiers (no LLM judge).

Headline numbers: gemini-3.5-flash-lite tops the table at 65% tier accuracy with the tightest hosted p95 (0.67s) at ~$0.09 per 1k classifications; gpt-4o-mini is one point behind at half the price. The hosted top four are within noise of each other, every model fails hardest on the medium tier, reasoning models make poor classifiers, and the zero-cost heuristic floors at 45%.

The Try It config was verified against the actual ComplexityRouterConfig schema (classifier_type: llm + classifier_llm_config.model referencing a model_list entry, with automatic heuristic fallback on timeout). Follows the structure of the two existing auto-router benchmark posts, including the design-partner CTA and discussion #32172 link. Hero image generated in the same title-card style as the existing posts.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview	Aug 7, 2026 2:51am