Conversation
RuckVibeCodes
left a comment
There was a problem hiding this comment.
[gus-first-pass] Found issues affecting reasoning payloads. Please address these before merge. Inline comments provided for clarification.
|
Reproduction (llama.cpp # last msg is assistant + thinking on → 400
curl -s $LLAMA/v1/chat/completions -d '{
"messages":[
{"role":"user","content":"hi"},
{"role":"assistant","content":"x"}
],
"think": false,
"chat_template_kwargs": {"enable_thinking": true}
}'
# → "Assistant response prefill is incompatible with enable_thinking."
# adding the right field → 200
curl -s $LLAMA/v1/chat/completions -d '{
...,
"chat_template_kwargs": {"enable_thinking": false}
}'Suggested addition inside the existing block: if self.provider == "custom":
_extra_body["think"] = False
_extra_body.setdefault("chat_template_kwargs", {})["enable_thinking"] = FalseVerified on llama.cpp + vLLM. Not tested on Ollama with I've opened #12427 with the request-build-path fix and tests; happy to fold it into this PR if you'd prefer. |
|
Thanks for isolating the assistant-prefill/thinking conflict. This is now implemented on
Closing as implemented on main. The cross-referenced #12427 concerns a separate generic custom-backend thinking-option mapping path. |
Summary
think=falsefor custom OpenAI-compatible backends on those continuation callsWhy
Fixes #11213.
Hermes uses a trailing assistant prefill when a model returns reasoning without visible text. Some OpenAI-compatible local backends reject that continuation request if Hermes also keeps reasoning/thinking enabled, returning:
Assistant response prefill is incompatible with enable_thinking.The logged failures reproduce as
api_calls=2, which lines up with the thinking-only continuation path.Testing
python3 -m pytest -o addopts= tests/run_agent/test_run_agent.py -k "assistant_prefill_continuation or ollama_think_false or reasoning_config"python3 -m pytest -o addopts= tests/run_agent/test_provider_parity.py -k "BuildApiKwargsCustomEndpoint or BuildApiKwargsOpenRouter or BuildApiKwargsAIGateway"