fix(agent): also send chat_template_kwargs.enable_thinking=false for llama.cpp/vLLM

[strictlyskyler]

What does this PR do?

The custom-provider thinking opt-out in _build_api_kwargs only sets extra_body.think=false, which is Ollama's field. llama-server and vLLM ignore think and look for chat_template_kwargs.enable_thinking instead.

Net effect: setting agent.reasoning_effort: none on a custom provider pointed at llama.cpp / vLLM has no effect. Thinking-capable models (Qwen3, etc.) still emit <think> blocks, and any subsequent assistant-prefill turn (including Hermes' own thinking-only continuation path in run_agent.py) hits llama.cpp's prefill-incompatibility 400:

HTTP 400: Assistant response prefill is incompatible with enable_thinking.

The fix sends both fields. Different OpenAI-compat backends use different keys and ignore unknown body fields by convention, so adding chat_template_kwargs.enable_thinking is additive for Ollama.

Related Issue

No issue filed; this is a small, surgical fix to behaviour added in the same block.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• run_agent.py: also set extra_body["chat_template_kwargs"]["enable_thinking"] = False when the existing custom-provider opt-out fires. Updated the surrounding comment.
• tests/run_agent/test_run_agent.py: 3 new tests next to the existing test_ollama_think_* cluster covering both the on-paths and the negative case.

How to Test

Reproduction against llama.cpp b8808-408225bb1 with Qwen3-14B (Bartowski GGUF):

# Without the fix: Hermes sends only think=false; llama.cpp ignores it,
# thinking stays on, prefill 400 fires on the next turn.
curl -s $LLAMA/v1/chat/completions -d '{
  "messages":[
    {"role":"user","content":"hi"},
    {"role":"assistant","content":"x"}
  ],
  "think": false,
  "chat_template_kwargs": {"enable_thinking": true}
}'
# → 400 "Assistant response prefill is incompatible with enable_thinking."

# With the fix: chat_template_kwargs.enable_thinking=false is also sent.
curl -s $LLAMA/v1/chat/completions -d '{
  ...,
  "chat_template_kwargs": {"enable_thinking": false}
}'
# → 200

Test suite:

pytest tests/run_agent/test_run_agent.py -k "ollama_think or chat_template_kwargs or non_custom_provider_unaffected" -q
# 6 passed

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(agent):)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've added tests for my changes
• I've tested on my platform: Ubuntu 22.04 (WSL2) + llama.cpp b8808-408225bb1 (remote) + Qwen3-14B Q4_K_M

Documentation & Housekeeping

• N/A — no public-API or config-key changes; comment updated in-place
• N/A — no cross-platform impact (pure body-shape change in HTTP client)

Notes for reviewers

• Verified on llama.cpp + vLLM (both honour chat_template_kwargs).
• Not retested on Ollama with chat_template_kwargs co-present, but it's an unknown OpenAI-extension field so should be ignored. Happy to gate by base URL if you'd prefer a more conservative landing.
• This is adjacent to fix(agent): disable thinking on assistant-prefill continuation #11324 (assistant-prefill + enable_thinking conflict). That PR fixes the prefill code path for Ollama; this PR fixes the request-build path so the opt-out actually reaches llama.cpp / vLLM.

[alt-glitch]

Related to #12914 (scope think=False to Ollama only) and #11237 (think=False incorrectly sent to all custom providers). This PR takes the complementary approach: send both fields so all backends work.

[jcjc81]

Confirming this bug still bites on current main, plus two notes that might help this land — one rebase, one design.

1. The target moved — this PR is now stale against main.

The think=False opt-out is no longer in run_agent.py. It was extracted into the custom provider profile at plugins/model-providers/custom/__init__.py. The disable path there today is:

# plugins/model-providers/custom/__init__.py  (build_api_kwargs_extras)
if _effort == "none" or _enabled is False:
    extra_body["think"] = False          # ← Ollama-only flag; vLLM/llama.cpp ignore it
elif _effort:
    top_level["reasoning_effort"] = _effort

So the exact bug this PR describes is intact — a custom provider pointed at vLLM/llama.cpp with reasoning_effort: none still only sends think=False, which those backends ignore (per vLLM docs: Qwen3 reasoning is on by default and only chat_template_kwargs.enable_thinking=False disables it). The fix just needs to move to that profile method.

2. Suggest making it backend-aware, to resolve the tension with reasoning_effort.

The custom profile is shared across Ollama, vLLM, llama.cpp and OpenAI-compatible reasoning endpoints like GLM-5.2 / Volcengine ARK. The profile's own code comment notes it deliberately avoids backend-specific flags because they "risk a 400 on GLM/vLLM endpoints that don't recognize it," which is why the enable path routes to top-level reasoning_effort.

Sending chat_template_kwargs.enable_thinking unconditionally re-introduces exactly that risk from the other direction: it's a vLLM/llama.cpp chat-template concept, and GLM/ARK may reject or ignore it. So "send both fields always" trades one silently-broken backend for another.

A backend-aware split satisfies both camps:

server_type = detect_local_server_type(base_url) if base_url else None
if _effort == "none" or _enabled is False:
    extra_body["think"] = False                       # Ollama
    if server_type in ("vllm", "llamacpp"):
        extra_body["chat_template_kwargs"] = {"enable_thinking": False}   # vLLM/llama.cpp
elif _effort:
    top_level["reasoning_effort"] = _effort           # GLM/ARK etc. unchanged

Scope: this addresses the disable path (reasoning_effort: none) that #12427 reports. The enable path stays as-is here — a fuller fix would also clamp unsupported efforts like max→high and send enable_thinking=True on detected vLLM, but that's a deliberate follow-up, out of scope for this PR.

Good news on feasibility: no plumbing change is needed. The transport already passes base_url into this method — chat_completions.py calls profile.build_api_kwargs_extras(..., base_url=params.get("base_url"), ...) — the custom profile just doesn't consume it yet. detect_local_server_type() is in agent/model_metadata.py. Heads-up on today's main: detect_local_server_type() does a live HTTP probe per call (up to four endpoints, 2s timeout each) with no memoization, so don't call it per request. Rather than caching forever on the long-lived profile — which would go stale if the server restarts as a different type — reuse the existing short-lived precedent: _LOCAL_CTX_PROBE_CACHE in model_metadata.py already memoizes local-server probes with a 30s TTL keyed by (model, base_url). The open caching PRs #29988 (process-lifetime cache for detect_local_server_type) / #37905 (drop idle probes) would make repeated calls cheap once they land.

One caveat worth a test: vLLM issues #35574 and #17357 show enable_thinking=False has version-specific edges on some Qwen3.5 builds (empty content, reasoning leaking into reasoning_content). Scoping the flag to detected vLLM/llama.cpp keeps that blast radius off GLM/ARK.

3. How this sits next to the extra_body passthrough work — complementary, not duplicate.

There's a parallel line of work making a custom_providers[].extra_body reach the model on every path (#52333 merged the init-time named-provider merge; #53765, opened today, threads request_overrides through the gateway / Open WebUI / TUI / /model-switch sites; #8160 is the original request; #55276 tracks reasoning_effort being silently dropped for custom/zai). Worth heading off the obvious "isn't this already covered?" question: it isn't — the two are different layers.

• extra_body passthrough (fix: deliver custom-provider extra_body (request_overrides) on gateway, Open WebUI, and TUI/CLI paths #53765 et al.) is the plumbing. It lets a user hand-write {chat_template_kwargs: {enable_thinking: false}} in config — but only if they know that's vLLM's exact incantation, and they'd have to special-case it per backend.
• This PR is the abstraction on top. It makes the provider-agnostic reasoning_effort: none knob actually work on vLLM/llama.cpp, deriving the right backend flag so the user never has to know chat_template_kwargs exists. That's the same "one shared knob, backend adapts" contract the profile already honors for GLM/ARK via reasoning_effort.

So this belongs in the profile regardless of where the passthrough lands. The enable-path piece I scoped out above (effort clamping + enable_thinking=True on detected vLLM) is exactly what #55276 describes — happy to fold this into that as the umbrella if a maintainer prefers one PR for the whole custom/vLLM reasoning story.

One adjacent gotcha to flag, not block on: #34027 (closed not_planned) documents that enable_thinking=false breaks the streaming server-side tool-call loop on the api_server path (model emits the tool name as plain text). Different code path from this PR — that's Hermes-as-server, this is Hermes-as-client — but it confirms the flag isn't free, which is another reason to keep it scoped to detected local backends rather than sent unconditionally.

Happy to help with the rebase to the profile method + a detect_local_server_type-routed test if that's useful. (The disable-path failure is confirmed from vLLM's own docs — reasoning is on by default, only enable_thinking=False in chat_template_kwargs disables it — and matches your llama.cpp repro above; I haven't re-run it against a live vLLM server myself.)

[teknium1]

Thanks for identifying the vLLM/llama.cpp reasoning-disable gap. The issue remains on current main, but the implementation has moved.

Problems

• The changed run_agent.py implementation no longer exists: AIAgent._build_api_kwargs() is a forwarder at run_agent.py:5328-5331. Registered custom providers take the profile path at agent/chat_completion_helpers.py:849-888; the live disable branch is plugins/model-providers/custom/__init__.py:52-60.
• provider=custom also covers non-vLLM endpoints (plugins/model-providers/custom/__init__.py:3-10). Current docs treat chat_template_kwargs.enable_thinking as a server-specific vLLM shape (website/docs/integrations/providers.md:1209-1223), so emitting it unconditionally is too broad.

Suggested changes

• Move the disable-path handling into CustomProfile.build_api_kwargs_extras() and use its already-forwarded base_url context (agent/transports/chat_completions.py:565-573) to scope the vLLM/llama.cpp field.
• Replace the old-builder tests with profile-dispatch tests for vLLM/llama.cpp and a non-matching custom endpoint.

Automated hermes-sweeper review.