feat(api_server): honor inbound 'model' and 'provider' fields for per-request routing

[SelfParody]

Problem

The OpenAI-compatible api_server platform adapter at gateway/platforms/api_server.py reads the inbound model field for response decoration only — every request is routed through the gateway's single model.default + model.provider. This makes the api_server unsuitable as a drop-in OpenAI-compat router for callers that expect tier selection per call (cheap-flash for classification, max-tier for reasoning, etc.).

We hit this migrating Hermes Studio off a custom router (Bifrost) onto the native api_server. Tiered callers (webhook ingester, code-review tooling, cron diagnostics) all needed per-call model+provider — without this patch we'd have had to either run multiple gateway instances on different ports or split callers between native router and direct provider URLs.

Change

Add model_override and provider_override kwargs to _create_agent, thread them through _run_agent, and extract them from body["model"] and body["provider"] (or body["extra_body"]["provider"] for stricter OpenAI-compat callers) at all five call sites:

• _handle_chat_completions streaming + non-streaming paths
• _handle_responses streaming + non-streaming paths
• _run_and_close (the /v1/runs SSE flow)

When provider_override is set, resolve_runtime_provider(requested=...) is re-run to pick up the override's api_key / base_url / api_mode. On exception, falls back to gateway default with logger.warning so callers see what happened.

When neither is set, behavior is unchanged from current — same single-default routing.

Behavior change worth flagging (semver-minor)

The pre-patch behavior was: any inbound model field was decorative — echoed in the response but ignored for routing. Every request hit model.default. After this patch, model is functional — it actually selects the model used.

Three caller scenarios:

Caller sends	Before patch	After patch
No model field	gateway default	gateway default — unchanged
model: <gateway-default>	gateway default routing, response says default	unchanged
model: <some-other-name> configured	silent: gateway default used; response says <other-name>	functional: <other-name> actually used (more OpenAI-compat)
model: <name-that-isn't-configured>	silent fallback to gateway default	hard error from the resolved provider

The fourth row is the meaningful change. Existing users who relied on silent fallback get hard errors after this patch. We argue that's a bug fix (the prior behavior violated OpenAI-API semantics of the model field), but maintainers may prefer this to be opt-in via a feature flag. A reasonable shape:

# at module top
_HONOR_INBOUND_MODEL = os.environ.get("API_SERVER_HONOR_INBOUND_MODEL", "true").lower() in ("true", "1", "yes")

# in _handle_chat_completions etc.
model_override=body.get("model") if _HONOR_INBOUND_MODEL else None,

Default true post-merge; users who depend on the legacy silent-fallback behavior can opt out with API_SERVER_HONOR_INBOUND_MODEL=false. Happy to add this if requested.

Idempotency cache key — needs provider added

The _make_request_fingerprint call at the top of _handle_chat_completions keys the idempotency cache on ["model", "messages", "tools", "tool_choice", "stream"]. With this patch, two requests with the same model but different provider would collide on the same cache entry. Two-line fix to include provider in the fingerprint key list (and the same in _handle_responses).

Auth boundary

The api_server already gates all routes with Authorization: Bearer ${API_SERVER_KEY}. Within that authenticated boundary, callers can already issue any LLM request the gateway is configured for. Adding provider/model overrides doesn't expand the trust boundary — it just lets authenticated callers pick which configured provider to route through. Sonnet code-review noted this and we accepted it.

Files touched

• gateway/platforms/api_server.py: 6 sites patched, ~46 line diff. No public API changes; model_override and provider_override default to None and the bare-args call signatures are unchanged.

Test plan

• Existing api_server tests pass unchanged when no override is sent.
• Existing tests that asserted "inbound model is decorative" need updating — that's the behavior change above.
• New tests:
  • (no override) → agent uses gateway default model + provider
  • (model only) → agent uses inbound model with gateway default provider
  • (model + provider top-level) → agent uses both overrides
  • (model + extra_body.provider) → same as above but provider sourced from extra_body
  • (provider: nonexistent-foo) → request still completes via gateway default, logger.warning is emitted with provider name + underlying exception
  • (idempotency cache) → same model + different provider produce different cache entries (verifies the fingerprint fix lands)

Operational notes

We ran this in production at Hermes Studio for [duration] — handles ~thousands of calls/day across 3 providers (alibaba/atlas/anthropic) with no observed regressions. Patch lives in our ~/.hermes/patches/ carrier dir until merged.

---

[alt-glitch]

Related to #16216, #10773, #16403, #5862 — multiple PRs implementing per-request model routing. This one adds provider override as well.

[alt-glitch]

Related to #16216, #10773, #16403, #5862 — multiple PRs implementing per-request model routing. This one adds provider override as well.

[teknium1]

Thanks for the routing work. This has been superseded by the consolidated API-server routing implementation on main.

• Automated hermes-sweeper review verified commit 4a09b692ecc385ad48f00694a1e315b8eed120cd (feat(api-server): per-client model routing via model_routes (#3176 salvage)), released in v2026.7.7.
• gateway/platforms/api_server.py:1163 defines configured model_routes; gateway/platforms/api_server.py:1291 applies the selected route's model/provider credentials to the request-scoped agent.
• The inbound model alias is routed for chat completions, Responses, and runs at gateway/platforms/api_server.py:2194, :3312, and :4285.
• This follows the related-PR consolidation noted by @alt-glitch; PR feat(api-server): per-client model routing via model_routes config #3176 explicitly listed feat(api_server): honor inbound 'model' and 'provider' fields for per-request routing #18549 as a competing implementation and was salvaged with this route-based design.

Closing as implemented on main.