fix: emit chat_template_kwargs for llama.cpp/vLLM thinking control

[mlaihk]

Summary

The custom provider's build_api_kwargs_extras() emits extra_body.think=False when reasoning is disabled. This is an Ollama-only flag — llama.cpp and vLLM ignore it entirely, defaulting to thinking-on, which results in empty content responses and retry storms.

Fix

Emit chat_template_kwargs={enable_thinking: false} alongside the existing think=False so each backend picks up the flag it understands:

• Ollama — reads extra_body.think (unchanged)
• llama.cpp / vLLM — reads chat_template_kwargs.enable_thinking (new)

Both flags are sent in extra_body, which the OpenAI Python client serialises into the top-level request body. No conditional provider detection needed.

Testing

Verified against llama.cpp server (Qwen3.6-35B-A3B-MTP) running on both Vulkan and ROCm backends:

Parameter	Thinking	Content
(default, no params)	ON	Empty
chat_template_kwargs.enable_thinking: false	OFF	Answer
reasoning_effort: none	ON (ignored)	Empty

chat_template_kwargs is the only parameter that reliably disables thinking on llama.cpp.

Motivation

Without this fix, Hermes configured with the custom provider against llama.cpp/vLLM backends produces empty responses when thinking is disabled (agent.reasoning_effort: none), because the server still produces reasoning tokens that consume the entire token budget.

[jcjc81]

Nice catch on the Ollama-only think field — llama.cpp/vLLM silently ignoring it is a real gap.

While independently building a fix for the same file/same reasoning-control block, I ran into two adjacent problems this PR doesn't cover yet, both on the enable path rather than disable:

1. vLLM rejects Hermes-only effort levels with HTTP 400

vLLM validates reasoning_effort against the OpenAI enum {none, low, medium, high} and throws a non-retryable 400 (literal_error) on Hermes-only levels (minimal, xhigh, max):

error_type=BadRequestError provider=custom base_url=http://<vllm-host>:8000/v1 model=Qwen3.6-27B
summary=HTTP 400: 1 validation error:
  {'loc': ('body', 'reasoning_effort'), 'msg': "Input should be 'none', 'low', 'medium' or 'high'"...

This fires whenever a user has agent.reasoning_effort: xhigh (or max/minimal) set and switches to a vLLM-backed custom provider — currently a hard crash, not a graceful clamp. GLM-5.2/ARK (same provider: custom route) legitimately accepts max/xhigh natively, so the clamp needs to be scoped to detected vLLM/llama.cpp only, not applied universally.

2. chat_template_kwargs.enable_thinking is needed on enable too, not just disable

Your fix correctly adds enable_thinking: False when reasoning is disabled. But the same key is required in the other direction: when reasoning is enabled with an effort level, chat_template_kwargs.enable_thinking: True must also be sent — reasoning_effort alone only controls intensity, it doesn't make the chat template actually emit reasoning content on models where thinking isn't the server-side default. Confirmed live against a vLLM box (Qwen3.6-27B): sending reasoning_effort without enable_thinking: True produces a normal non-reasoning completion.

Backend detection

Both fixes need to know whether the endpoint is vLLM/llama.cpp before touching chat_template_kwargs, since GLM/ARK (also provider: custom) would 400 or ignore-unexpectedly on a field it doesn't understand. agent/model_metadata.py already has detect_local_server_type() for exactly this — a lightweight, best-effort /version+/models probe that degrades to None on any failure (never blocks the request).

I have a working patch that:

• Clamps xhigh/max → high, minimal → low for detected vLLM/llama.cpp (GLM/ARK/Ollama/unknown pass through verbatim)
• Emits chat_template_kwargs.enable_thinking True/False symmetrically on both enable and disable
• 35 tests covering both backends, the clamp table, detection-failure fallback, and non-templated passthrough
• Live-validated against a real vLLM box: previously-400ing xhigh/minimal now return HTTP 200 with populated reasoning content

Happy to either (a) extend this PR with the clamp + enable-path fix so it ships as one coherent change, or (b) open a separate PR scoped to just the clamp/enable-path if you'd rather keep this one minimal and disable-only. Your call — didn't want to just force-push over your branch. Let me know which you'd prefer.

[teknium1]

Thanks for identifying the vLLM/llama.cpp disable-thinking gap. Current main still lacks this automatic field: CustomProfile emits think=False at plugins/model-providers/custom/__init__.py:65, while the vLLM documentation identifies chat_template_kwargs.enable_thinking: false as the applicable shape (website/docs/integrations/providers.md:1216-1229).

Problems

• The new field at plugins/model-providers/custom/__init__.py:41 is emitted for every custom endpoint. That profile also serves GLM/ARK (plugins/model-providers/custom/__init__.py:40-46), and the docs explicitly describe custom request bodies as endpoint-specific (website/docs/integrations/providers.md:1204-1216). Please scope this to compatible backends.
• The branch conflicts with current main. Preserve the current top_level["reasoning_effort"] = "none" behavior introduced by 8662254ab; the PR's older return shape would otherwise lose it.
• Please add coverage alongside tests/plugins/model_providers/test_custom_profile.py:51-71 for targeted vLLM/llama.cpp emission and unaffected non-target custom endpoints.

Suggested changes

• Resolve the current reasoning-control block first, then gate chat_template_kwargs using the existing cached local-server detection in agent/model_metadata.py:683-762.

Automated hermes-sweeper review.

[teknium1]

provider=custom also covers GLM/ARK and arbitrary OpenAI-compatible endpoints on current main. Please gate this backend-specific field to compatible vLLM/llama.cpp endpoints rather than changing every custom request's wire shape; the docs describe custom extra_body fields as endpoint-specific.

[GottZ]

This was generated by AI during triage.

Summary

Two open PRs address the same reasoning-disable gap: #12427 adds the llama.cpp/vLLM request field in the former run_agent.py path with tests, while #64153 adds it in the current custom-provider profile but applies it to every custom endpoint.

Related pull requests

• fix(agent): also send chat_template_kwargs.enable_thinking=false for llama.cpp/vLLM #12427 duplicate — (+49/-6) — n/a: Adds chat_template_kwargs.enable_thinking=false beside think=false and covers disabled and enabled reasoning paths, but the diff targets the obsolete run_agent.py implementation. Despite the automated keep_open verdict, fix: emit chat_template_kwargs for llama.cpp/vLLM thinking control #64153 now patches the live profile location, so fix(agent): also send chat_template_kwargs.enable_thinking=false for llama.cpp/vLLM #12427 is relevant as the earlier duplicate and as test-design input.
• fix: emit chat_template_kwargs for llama.cpp/vLLM thinking control #64153 related — (+4/-1) — n/a: Adds the required llama.cpp/vLLM disable-thinking field in plugins/model-providers/custom/init.py, directly addressing the reported cause, but the visible COMMENTED keep_open review requires backend scoping, preservation of the reasoning_effort="none" behavior from 8662254, rebasing, and targeted profile tests.

Duplicates

#12427 and #64153 implement substantially the same disable-thinking change; #12427 targets the superseded builder location, while #64153 targets the current custom-provider profile.

Suggested consolidation

Keep #64153 open with the salvage path recorded in its keep_open review: rebase onto main, scope emission to compatible vLLM/llama.cpp endpoints, preserve the behavior introduced by 8662254, and add targeted and unaffected-endpoint tests. Close #12427 as a duplicate of #64153 despite its automated keep_open verdict, because its diff changes the obsolete run_agent.py path; carry its reasoning-path test cases into #64153 where applicable.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    subgraph Dup12427 ["PRs duplicating each other"]
        P12427["PR #12427 (open)"]
        P64153["PR #64153 (open)"]
    end
    class P12427 open
    class P64153 open
    class P64153 target
    click P12427 "https://github.com/NousResearch/hermes-agent/pull/12427"
    click P64153 "https://github.com/NousResearch/hermes-agent/pull/64153"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 2 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 5 kB of PR diffs, 5 kB of issue/PR text, 11 kB of discussion (4 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.