feat(gateway): add smart_model_routing for auto-complexity classification - #37444
Closed
davidgut1982 wants to merge 2 commits into
Closed
feat(gateway): add smart_model_routing for auto-complexity classification#37444davidgut1982 wants to merge 2 commits into
davidgut1982 wants to merge 2 commits into
Conversation
davidgut1982
force-pushed
the
feat/smart-model-routing
branch
from
June 2, 2026 15:50
ba56415 to
d4fd707
Compare
Routes short/simple gateway messages to a cheap model before agent dispatch, cutting cost without affecting complex requests or manual /model overrides. Config-gated via smart_model_routing (default off). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
davidgut1982
force-pushed
the
feat/smart-model-routing
branch
from
June 2, 2026 15:55
d4fd707 to
98d86d9
Compare
combatsheep
approved these changes
Jun 2, 2026
combatsheep
left a comment
There was a problem hiding this comment.
Smart routing is well-contained: the override and Platform.LOCAL bypasses stay intact, and the classifier only rewrites runtime kwargs for short, non-complex messages. Approving.
TestDialecticLifecycleSmoke._await_thread did a single join(timeout=3.0) and then proceeded regardless of whether the background dialectic thread had actually finished. On a loaded CI runner (6 parallel test slices) the prewarm thread's completion can slip past that 3s window, so the join times out silently and the test reads _prefetch_result before the worker wrote it — the intermittent 'session-start prewarm must land in _prefetch_result' failure seen in CI run 26831695266 slice 3. Join in a loop up to a 30s ceiling and assert the thread is actually dead, so a genuine hang surfaces as a clear failure instead of a timing race. Cherry-picked from bb1c8b6 (merged to main after this branch forked). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
6 tasks
Contributor
|
We do not want this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
smart_model_routinggateway-level classification step that automatically routes short, simple messages to a configurable cheap model before agent dispatch — without requiring any user action or session state change.How it works (
gateway/run.py):GatewayRunner._apply_smart_routing(message, model, runtime_kwargs, smart_cfg)inspects the raw incoming message text:max_simple_chars(default 200) andmax_simple_words(default 40)complexity_keywords(e.g. "implement", "debug", "refactor", "diagnose", "architect")runtime_kwargs["model"]is replaced withcheap_modelbefore the agent startsThe gate is applied in
run_sync()after_resolve_session_agent_runtimeand is skipped when:/modeloverride (_session_model_overridesentry present)Platform.LOCAL(CLI sessions bypass smart routing)Configuration (
hermes_cli/config.py):All provider routing, API keys, and base URLs flow through unchanged; only the model slug in
runtime_kwargsis swapped.Why
Operators running Hermes on a fixed budget or with high message volume pay the same cost for "ping" as for "implement OAuth2". Smart routing cuts cost on trivial turns without changing the agent's behavior, memory, or session continuity.
Tests
6 tests covering: simple message routed to cheap model, complexity keyword blocks downgrade, message-too-long blocks downgrade, manual
/modeloverride bypasses smart routing,enabled: falsepassthrough, and platform-LOCAL bypass. All pass.Platforms tested
Linux (CT/LXC environment, Python 3.13).