Skip to content

feat(gateway): add smart_model_routing for auto-complexity classification - #37444

Closed
davidgut1982 wants to merge 2 commits into
NousResearch:mainfrom
davidgut1982:feat/smart-model-routing
Closed

feat(gateway): add smart_model_routing for auto-complexity classification#37444
davidgut1982 wants to merge 2 commits into
NousResearch:mainfrom
davidgut1982:feat/smart-model-routing

Conversation

@davidgut1982

Copy link
Copy Markdown
Contributor

What

Adds a smart_model_routing gateway-level classification step that automatically routes short, simple messages to a configurable cheap model before agent dispatch — without requiring any user action or session state change.

How it works (gateway/run.py):

GatewayRunner._apply_smart_routing(message, model, runtime_kwargs, smart_cfg) inspects the raw incoming message text:

  • If the text is shorter than max_simple_chars (default 200) and max_simple_words (default 40)
  • And contains none of complexity_keywords (e.g. "implement", "debug", "refactor", "diagnose", "architect")
  • Then runtime_kwargs["model"] is replaced with cheap_model before the agent starts

The gate is applied in run_sync() after _resolve_session_agent_runtime and is skipped when:

  • The session has an active /model override (_session_model_overrides entry present)
  • The source platform is Platform.LOCAL (CLI sessions bypass smart routing)

Configuration (hermes_cli/config.py):

smart_model_routing:
  enabled: false          # opt-in; must explicitly set true
  cheap_model: deepseek/deepseek-v4-flash
  max_simple_chars: 200
  max_simple_words: 40
  complexity_keywords:
    - implement
    - debug
    - refactor
    - diagnose
    - migrate
    - architect
    - explain
    - "why does"
    - "how does"
    - broken
    - failing
  simple_keywords:        # informational only; short + no complexity_keyword → cheap
    - status
    - show
    - check
    - list
    - restart
    - "what is"
    - ping
    - health

All provider routing, API keys, and base URLs flow through unchanged; only the model slug in runtime_kwargs is swapped.

Why

Operators running Hermes on a fixed budget or with high message volume pay the same cost for "ping" as for "implement OAuth2". Smart routing cuts cost on trivial turns without changing the agent's behavior, memory, or session continuity.

Tests

pytest tests/gateway/test_smart_model_routing.py -v

6 tests covering: simple message routed to cheap model, complexity keyword blocks downgrade, message-too-long blocks downgrade, manual /model override bypasses smart routing, enabled: false passthrough, and platform-LOCAL bypass. All pass.

Platforms tested

Linux (CT/LXC environment, Python 3.13).

@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have labels Jun 2, 2026
@davidgut1982
davidgut1982 force-pushed the feat/smart-model-routing branch from ba56415 to d4fd707 Compare June 2, 2026 15:50
Routes short/simple gateway messages to a cheap model before agent
dispatch, cutting cost without affecting complex requests or manual
/model overrides. Config-gated via smart_model_routing (default off).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@davidgut1982
davidgut1982 force-pushed the feat/smart-model-routing branch from d4fd707 to 98d86d9 Compare June 2, 2026 15:55

@combatsheep combatsheep left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smart routing is well-contained: the override and Platform.LOCAL bypasses stay intact, and the classifier only rewrites runtime kwargs for short, non-complex messages. Approving.

TestDialecticLifecycleSmoke._await_thread did a single join(timeout=3.0)
and then proceeded regardless of whether the background dialectic thread
had actually finished. On a loaded CI runner (6 parallel test slices) the
prewarm thread's completion can slip past that 3s window, so the join
times out silently and the test reads _prefetch_result before the worker
wrote it — the intermittent 'session-start prewarm must land in
_prefetch_result' failure seen in CI run 26831695266 slice 3.

Join in a loop up to a 30s ceiling and assert the thread is actually dead,
so a genuine hang surfaces as a clear failure instead of a timing race.

Cherry-picked from bb1c8b6 (merged to main after this branch forked).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@teknium1

teknium1 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

We do not want this

@teknium1 teknium1 closed this Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants