fix: deliver custom-provider extra_body (request_overrides) on gateway, Open WebUI, and TUI/CLI paths - #53765
Conversation
A `custom_providers` entry can carry an `extra_body` (e.g.
`chat_template_kwargs` to toggle a local vLLM model's thinking).
`resolve_runtime_provider()` correctly surfaces it as `request_overrides`
on the resolved runtime dict, but the gateway never plumbed it through to
the per-turn agent:
- `_resolve_runtime_agent_kwargs()` rebuilt the runtime dict from a fixed
key whitelist that omitted `request_overrides`.
- `_resolve_turn_agent_config()` rebuilt `runtime` from the same whitelist
and set `route["request_overrides"]` solely from `/fast` service-tier
overrides (`{}` otherwise).
- The per-turn `agent.request_overrides = turn_route.get(...)` assignment
then clobbered the value `_merge_custom_provider_extra_body()` applied at
agent construction.
Net: on the gateway, a custom provider's configured `extra_body` never
reached the model -- only `/fast` overrides survived. The CLI/TUI path
(which does not go through `_resolve_turn_agent_config`) and the auxiliary
client (which sends `extra_body` directly) were unaffected.
Fix: carry `request_overrides` through the runtime resolvers
(`_resolve_runtime_agent_kwargs`, `_try_resolve_fallback_provider`) and
merge the provider overrides into the per-turn route, layering any `/fast`
service-tier overrides on top (top-level keys, no collision with
`extra_body`).
Adds tests/gateway/test_turn_request_overrides.py.
Known follow-up: the mid-session `/model`-switch override path
(`_session_model_overrides` / `ModelSwitchResult`) does not yet carry
`request_overrides`.
Follow-up to the previous commit (which fixed the default/fallback provider path). A mid-session `/model` switch stores a per-session override bundle in `_session_model_overrides` that omitted `request_overrides`, and the two consumers (`_resolve_session_agent_runtime` fast path and `_apply_session_model_override`) only copied provider/api_key/base_url/api_mode. So switching *to* a custom provider via `/model` did not apply its `extra_body`. - `ModelSwitchResult` gains a `request_overrides` field, derived for the switched provider via `_get_named_custom_provider` / `_custom_provider_request_overrides` (the same overrides `resolve_runtime_provider` surfaces for the default path). - Both `/model` override-storage sites in slash_commands.py persist it. - Both consumers apply it; `_apply_session_model_override` also clears a stale value when switching to a provider that has none. Extends tests/gateway/test_turn_request_overrides.py (3 new cases).
…UI/CLI) Third in the series. The gateway rebuild path (previous two commits) carries a custom provider's `request_overrides` (`extra_body`, e.g. `chat_template_kwargs`) into the agent, but the *in-place* live switch used by the TUI dashboard and the CLI — `agent.switch_model()` -> `agent_runtime_helpers.switch_model()` — swapped model/provider/base_url/api_key without ever updating `request_overrides`. So a `/model` switch to a thinking-enabled custom provider in the TUI/CLI kept the previous provider's `extra_body`. `switch_model()` now re-derives the switched-to provider's `request_overrides` (via `_get_named_custom_provider`) and applies it in place, preserving non-provider overrides (`service_tier`/`speed` from `/fast`). Logic factored into `_apply_switched_provider_request_overrides` for testability. Adds tests/agent/test_switch_model_request_overrides.py.
The TUI dashboard appended a role:"system" model-switch marker into the conversation history after a /model switch. Strict OpenAI-compatible backends (vLLM/Qwen) reject a system message that is not at the beginning, so the NEXT turn failed with "HTTP 400: System message must be at the beginning." Mirror the gateway's pending-note approach: _append_model_switch_marker now stages session["pending_model_note"], and _run_prompt_submit prepends it to the next user turn (consumed once, not persisted). No mid-history system message is created, so strict backends accept the turn. Adds tests/tui_gateway/test_model_switch_marker.py; updates the existing tui-server switch test to assert the staged-note behavior.
The TUI `_make_agent` rebuild path resolves the switched provider via
`resolve_runtime_provider()` (so `runtime` carries `request_overrides`) but
passed only model/provider/base_url/api_key/api_mode to `AIAgent` —
dropping `request_overrides`. So after switching to a thinking-enabled
custom provider, a `/new` or session resume reverted to the default
provider's first-match `extra_body` merge (thinking turned back off).
Pass `request_overrides=runtime.get("request_overrides")` to `AIAgent` in
`_make_agent`. Explicit-field construction (no `**runtime` spread) avoids
the double-pass crash in NousResearch#53406.
Adds a regression test to tests/test_tui_gateway_server.py.
|
Note on the test suite — pre-existing failures, not from this PR Running
Evidence they aren't introduced here:
Flagging so the CI result isn't read as a regression. |
alt-glitch
added
type/bug
Related to #52432 (which patches only the gateway clobber sub-case) and #30224 (turn-refresh merge) — this PR is the broader fix for the same root cause (custom-provider |
…ftover The NousResearch#53765 conflict resolution kept commit 4's pending_model_note staging in the locked branch alongside upstream's role=user append (NousResearch#48338), leaving _append_model_switch_marker asymmetric and an orphaned consumer in _run_prompt_submit. Match upstream NousResearch#48338 exactly; remove the now-redundant commit-4 test.
…verrides # Conflicts: # gateway/run.py # tests/test_tui_gateway_server.py
|
Thanks for tracing the gateway route through to the live per-turn overwrite. The central premise remains valid on current main: Problems
Suggested changes
Automated hermes-sweeper review. |
teknium1
added
sweeper:risk-session-state
…name Addresses the hermes-sweeper review on NousResearch#53765. The in-place /model switch helper (_apply_switched_provider_request_overrides) derived a custom provider's extra_body by provider *name* only, while build-time matching in agent_init._merge_custom_provider_extra_body matches by provider key, base_url, AND model. So a different model selected at the same named endpoint could inherit an extra_body configured for another model. Reuse the shared agent_init._custom_provider_extra_body_for_agent matcher (provider key + base_url + model), sourcing custom_providers from the init-time agent._custom_providers cache (fresh-load fallback if absent). A stale extra_body is always cleared when no entry matches; non-provider overrides (service_tier / speed from /fast) are preserved. Tests: add nonmatching-model and endpoint-mismatch regressions; update the existing switch tests onto the model/base_url-aware matcher.
cfd4ef7 to
09115a6
Compare
|
Thanks for the review — good catch on the matching asymmetry. Addressed in Change: the in-place switch path ( Stale handling: a stale Tests: added I kept the switch path calling the matcher directly rather than routing through |
3 participants
What does this PR do?
A
custom_providersentry can carry anextra_body— e.g.{chat_template_kwargs: {enable_thinking: false}}to control a locally-served Qwen3 / Qwen3.6 model's "thinking" on vLLM.resolve_runtime_provider()correctly surfaces this asrequest_overrideson the resolved runtime, but several independent agent build/switch paths rebuild the runtime from a fixed field whitelist that omitsrequest_overrides, so the configuredextra_bodynever reaches the model.This threads
request_overridesthrough every path that was dropping it:api_serverpath,/modelmid-session switch,/modelswitch, andIt also fixes a related TUI bug: after a
/modelswitch the next turn failed on strict backends withHTTP 400: System message must be at the beginning(details below).Why this approach: the value is already resolved by
resolve_runtime_provider(); the fix carries it through the build/switch sites rather than re-deriving or special-casing. The messaging platforms are thin adapters over one sharedGatewayRunnerpath, so a single change covers all of them; onlyapi_serverand the TUI/CLI have their own build/switch code, handled explicitly.Related Issue
No separate issue — the root cause and fix are both here. Related prior work:
/modelswitch, and TUI/CLI.extra_bodymerge only.Type of Change
Changes Made
gateway/run.py— carryrequest_overridesthrough_resolve_runtime_agent_kwargsand_try_resolve_fallback_provider; merge it into the per-turn route in_resolve_turn_agent_configunder any/fastservice-tier overrides; apply it in_apply_session_model_overrideand the_resolve_session_agent_runtimefast path. (Also fixes the Open WebUI path, sinceapi_server._create_agentbuildsAIAgent(**runtime_kwargs).)hermes_cli/model_switch.py— add arequest_overridesfield toModelSwitchResult, derived for the switched-to provider via_get_named_custom_provider.gateway/slash_commands.py— persistrequest_overridesin both/modelsession-override bundles.agent/agent_runtime_helpers.py—switch_model()re-derives and applies the switched provider'srequest_overridesin place, preserving non-provider overrides (service_tier/speed).tui_gateway/server.py— (a) stage the model-switch note for the next user turn instead of appending a mid-conversationrole:"system"message (mirrors the gateway's pending-note pattern; fixes theHTTP 400: System message must be at the beginningon strict OpenAI-compatible backends like vLLM/Qwen); (b) passrequest_overridestoAIAgentin_make_agentso rebuild/resume keeps the switched provider's settings.tests/gateway/test_turn_request_overrides.py,tests/agent/test_switch_model_request_overrides.py,tests/tui_gateway/test_model_switch_marker.py, plus a_make_agentrebuild test and an updated marker test intests/test_tui_gateway_server.py.How to Test
custom_providersentry for a vLLM-served Qwen3 model withextra_body: {chat_template_kwargs: {enable_thinking: true}}; set it asmodel.provider, or/model-switch to it mid-session.chat/completionsrequest.Before:
extra_bodyis absent —chat_template_kwargsis never sent and the model doesn't think.After: the provider's
extra_bodyis sent and thinking toggles as configured. Verified on the wire in all three interfaces.#53406 (double-pass) safety: the gateway
AIAgent(**turn_route["runtime"], …, request_overrides=…)sub-dict excludesrequest_overrides(it ridesroute["request_overrides"]only), andapi_server._create_agentuses**runtime_kwargswith no explicitrequest_overrides=— sorequest_overridesis passed exactly once.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AScreenshots / Logs
On-the-wire payload (custom provider pinned
enable_thinking:false, vLLM server defaultenable_thinking:true, so the provider flag must win):The model's behavior matched the wire in all three interfaces (Telegram, Open WebUI, TUI dashboard).