Skip to content

fix(providers): probe ollama thinking capability before sending reasoning_effort on custom endpoints - #108070

Open
tomRumi wants to merge 1 commit into
NousResearch:mainfrom
tomRumi:fix/custom-ollama-thinking-400
Open

tomRumi wants to merge 1 commit into
NousResearch:mainfrom
tomRumi:fix/custom-ollama-thinking-400

Conversation

@tomRumi

@tomRumi tomRumi commented Sep 11, 2026

Copy link
Copy Markdown

Problem

Ollama's /v1/chat/completions rejects top-level reasoning_effort with HTTP 400 "<model>" does not support thinking whenever the target model lacks the thinking capability (verified live: granite4:3b, ministral-3:8b; their /api/show capabilities are [completion, tools] / [completion, vision, tools]).

CustomProfile forwarded any configured effort verbatim, so a global agent.reasoning_effort: high broke every non-thinking local model: three retries, failed turn, and no hint in the UI — the error only surfaced in errors.log. The workaround (agent.reasoning_overrides.<model>: disabled) must be re-done by hand for every non-thinking model the user ever pulls.

The main conversation path and the auxiliary path (session-title generation, which resolves through the same profile) both hit it:

ERROR agent.conversation_loop: API call failed after 3 retries.
HTTP 400: "granite4:3b" does not support thinking | provider=custom:ollama-(local) model=granite4:3b

Fix

CustomProfile reuses the existing hermes_cli.models_local.ollama_model_supports_thinking() probe (native /api/show capabilities — the same authority OllamaCloudProfile and ReasoningParamsMixin._ollama_supports_thinking_cached() already rely on) for Ollama-shaped custom endpoints before emitting an effort:

probe result behaviour
thinking in capabilities effort forwarded verbatim (unchanged for qwen3, deepseek-r1, …)
probed OK, no thinking reasoning_effort omitted — request cannot 400
probe unreachable / raises fail open — effort sent (worst case is the pre-fix behaviour; a transient probe failure must not silently drop the user's reasoning config)
non-Ollama custom endpoint (GLM/ARK, Groq, vLLM, llama.cpp) no probe, no latency, no behaviour change (_looks_like_ollama_endpoint gate; #57601 passthrough contract intact)

Definitive results are cached per (model, base_url) on the profile singleton — one /api/show per model per process, not per request build; failed probes retry after a 60 s TTL (mirrors the _cached_probe policy in reasoning_params.py). The explicit-disable path (reasoning_effort: none + think: false, #14820/#25758) is untouched and stays probe-free.

Testing

  • 9 new invariant tests in tests/plugins/model_providers/test_custom_profile.py: omit-on-non-thinking, verbatim-on-thinking, fail-open on None-returning and raising probes, definitive caching (+ distinct key per model), no probe on non-Ollama endpoints, disable path still probe-free.
  • Full file: 49 passed (tests/plugins/model_providers/test_custom_profile.py + test_ollama_cloud_profile.py), existing 40 assertions unchanged and green.
  • E2E against a live ollama 0.12.10 on this machine: with agent.reasoning_effort: high and no overrides, granite4:3b now answers a one-shot hermes chat -q turn (previously: 3x retry, then HTTP 400 failure).

Notes for reviewers

  • Ollama's native /api/chat happily ignores thinking params; this only concerns the OpenAI-compat /v1 route Hermes uses for custom providers.
  • An alternative "catch the 400 and retry without the field" recovery exists in spirit (_ephemeral_reasoning_off) but only covers disable-rejected routes; probing before sending is cheaper than a failed turn + retry cycle, and matches how Ollama Cloud is already handled.

…ning_effort on custom endpoints

Ollama's /v1/chat/completions rejects top-level reasoning_effort with
HTTP 400 '"<model>" does not support thinking' when the model lacks the
thinking capability (e.g. granite4:3b, ministral-3:8b). CustomProfile
forwarded any configured effort verbatim, so a global
agent.reasoning_effort: high broke every non-thinking local model with
three retries and a failed turn. Users had to discover the error in logs
and hand-write agent.reasoning_overrides per model — re-doing it for
every model they ever pull.

CustomProfile now reuses the existing ollama_model_supports_thinking()
probe (native /api/show capabilities — the same authority the Ollama
Cloud profile and ReasoningParamsMixin rely on) for Ollama-shaped
custom endpoints:

- capability absent -> omit reasoning_effort (request cannot 400)
- capability present -> effort forwarded verbatim (qwen3, deepseek-r1...)
- probe unreachable -> fail open, send the effort (worst case is the
  previous behaviour; a transient probe failure must not silently drop
  the user's reasoning config
- non-Ollama custom endpoints (GLM/ARK, Groq, vLLM, llama.cpp) are
  untouched — no probe, no latency, NousResearch#57601 passthrough contract intact
- definitive results cached per (model, base_url) on the profile
  singleton (one /api/show per model, not per request build); failures
  retried after a 60s TTL

Also covers the auxiliary path (session titles etc.), which builds its
kwargs through the same profile and 400'd the same way.
EOF
)
@tomRumi
tomRumi force-pushed the fix/custom-ollama-thinking-400 branch from b6e2ecd to 0411a22 Compare September 11, 2026 09:12
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins provider/ollama Ollama / local models labels Sep 11, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Related: this is the fourth open fix for #59660 alongside #95854, #81574 and #86197 (plus closed #63315). All gate reasoning_effort on the Ollama /api/show thinking capability but differ in where the probe lives (this PR reuses hermes_cli.models_local.ollama_model_supports_thinking). Flagging the cluster so a maintainer can pick one mechanism.

@alt-glitch alt-glitch added the area/local-models Local model inference/runtimes: llama.cpp, Ollama, LM Studio, MLX/vLLM, GGUF, VRAM and offload label Sep 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/local-models Local model inference/runtimes: llama.cpp, Ollama, LM Studio, MLX/vLLM, GGUF, VRAM and offload comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have provider/ollama Ollama / local models type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants