Skip to content

fix(agent): disable thinking on assistant-prefill continuation - #11324

Closed
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/assistant-prefill-thinking-conflict
Closed

LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/assistant-prefill-thinking-conflict

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

Summary

  • strip explicit reasoning payloads from assistant-prefill continuation requests
  • force think=false for custom OpenAI-compatible backends on those continuation calls
  • add regression coverage for both reasoning-extra-body and custom backend paths

Why

Fixes #11213.

Hermes uses a trailing assistant prefill when a model returns reasoning without visible text. Some OpenAI-compatible local backends reject that continuation request if Hermes also keeps reasoning/thinking enabled, returning:

Assistant response prefill is incompatible with enable_thinking.

The logged failures reproduce as api_calls=2, which lines up with the thinking-only continuation path.

Testing

  • python3 -m pytest -o addopts= tests/run_agent/test_run_agent.py -k "assistant_prefill_continuation or ollama_think_false or reasoning_config"
  • python3 -m pytest -o addopts= tests/run_agent/test_provider_parity.py -k "BuildApiKwargsCustomEndpoint or BuildApiKwargsOpenRouter or BuildApiKwargsAIGateway"

@RuckVibeCodes RuckVibeCodes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[gus-first-pass] Found issues affecting reasoning payloads. Please address these before merge. Inline comments provided for clarification.

@strictlyskyler

Copy link
Copy Markdown

llama-server ignores think:false (it's Ollama's field). The matching field on llama.cpp / vLLM is chat_template_kwargs.enable_thinking.

Reproduction (llama.cpp b8808-408225bb1, Qwen3-14B):

# last msg is assistant + thinking on → 400
curl -s $LLAMA/v1/chat/completions -d '{
  "messages":[
    {"role":"user","content":"hi"},
    {"role":"assistant","content":"x"}
  ],
  "think": false,
  "chat_template_kwargs": {"enable_thinking": true}
}'
# → "Assistant response prefill is incompatible with enable_thinking."

# adding the right field → 200
curl -s $LLAMA/v1/chat/completions -d '{
  ...,
  "chat_template_kwargs": {"enable_thinking": false}
}'

Suggested addition inside the existing block:

if self.provider == "custom":
    _extra_body["think"] = False
    _extra_body.setdefault("chat_template_kwargs", {})["enable_thinking"] = False

Verified on llama.cpp + vLLM. Not tested on Ollama with chat_template_kwargs co-present.

I've opened #12427 with the request-build-path fix and tests; happy to fold it into this PR if you'd prefer.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Apr 25, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for isolating the assistant-prefill/thinking conflict. This is now implemented on main by a later, broader fix.

  • Automated hermes-sweeper review verified that agent/conversation_loop.py:902-913 removes reasoning-only assistant turns from the per-call API message list before provider request construction.
  • run_agent.py:3705-3770 defines that state as an assistant turn with reasoning but no visible content or tool calls, covering the continuation scenario described here.
  • Commit 2b728e12748e3a30273acdbef36ecad15a04f2b9 (fix(agent): drop thinking-only assistant turns before provider call, PR fix(agent): drop thinking-only assistant turns before provider call #16959) added this behavior and dedicated regression coverage in tests/run_agent/test_thinking_only_sanitizer.py.

Closing as implemented on main. The cross-referenced #12427 concerns a separate generic custom-backend thinking-option mapping path.

@teknium1 teknium1 closed this Jul 12, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Assistant Response Prefill and Thinking Flag Conflict

5 participants