server : support preserving reasoning_content in assistant message

[ngxson]

Ref: #18936 (comment)

Changes included in this PR

• use json_fwd in chat.h to avoid using template trick
• deduplicate code between common_chat_msgs_to_json_oaicompat and common_chat_msg::to_json_oaicompat()
• force clear_thinking = false for GLM 4.7 if it is not specified
• report the supports_preserve_reasoning to server /props

(Web UI support is TBD)

Changes in API

The /chat/completions API now accepts reasoning_content for assistant message:

{
  "messages": [
    {
      "content": "Hello, world!",
      "role": "user"
    },
    {
      "content": "Hey there!",
      "role": "assistant",
      "reasoning_content": "This is my reasoning."
    },
    {
      "content": "Hello, world!",
      "role": "user"
    }
  ],
  "stream": false,
  "max_tokens": 64
}

If the template supports it, the reasoning will be put back into the message (testing with GLM 4.7)

[gMASK]<sop><|user|>Hello, world!<|assistant|><think>This is my reasoning.</think>Hey there!<|user|>Hello, world!<|assistant|><think>

Otherwise, it will be ignored.

To know if the template supports it, /props endpoint will indicate:

{
  "chat_template_caps": {
    ...
    "supports_preserve_reasoning": true,
    ...
  }
}

[pwilkin]

Just as a general notion: I am not a fan of splitting reasoning handling into "enable_reasoning", "clear_thinking" and the passive "supports_preserve_reasoning". I think this is a bit messy. Don't have a clear idea of how to handle this yet, but I guess we should (a) detect whether model supports reasoning (b) enable reasoning by default if it does (c) pass reasoning traces if the template supports it (d) accept explicit overrides, but I'm not sure if the explicit overrides are something we should handle on the level of flags or just allow passing it in template_kwargs.

[pwilkin]

Should just remove those at this point, we're not going back to Minja.

[pwilkin]

Likewise, I'd just remove this. The code files are littered with comments like this that are left and then never removed.

[pwilkin]

Ditto.

[pwilkin]

I guess you could argue that GPT-OSS does, but don't know if anyone properly supports that.

[aldehir]

Might need a couple more capability checks for thinking at the message level and "type": "thinking" in content parts for gpt-oss and ministral 3 respectively.

The current logic for these models transforms reasoning_content to their expected field at init.

[ngxson]

for gpt-oss, it seems like reasoning is only allowed to be added if add_generation_prompt = false, so not usable in llama.cpp use case I think:

{%- elif loop.last and not add_generation_prompt %}
    {#- Only render the CoT if the final turn is an assistant turn and add_generation_prompt is false #}
    {#- This is a situation that should only occur in training, never in inference. #}
    {%- if "thinking" in message %}
        {{- "<|start|>assistant<|channel|>analysis<|message|>" + message.thinking + "<|end|>" }}
    {%- endif %}

[aldehir]

Line 293:

            {%- elif message.thinking and not future_final_message.found %}
                {{- "<|start|>assistant<|channel|>analysis<|message|>" + message.thinking + "<|end|>" }}
            {%- endif %}

[ngxson]

Just as a general notion: I am not a fan of splitting reasoning handling into "enable_reasoning", "clear_thinking" and the passive "supports_preserve_reasoning"

@pwilkin I'm not splitting but they are indeed different notions:

• enable_reasoning: I think you mean enable_thinking. This flag means to add a trailing </think> to the formatted chat, it does not overlap with supports_preserve_reasoning (one is user-controlled and one is read-only). For example: I can enable thinking in older messages in the conversation, then the next message, I put back the reasoning_content while disable enable_thinking, this forces the model to read the reasoning from the earlier message in the conversation.
• supports_preserve_reasoning: as explained above; However, this is NOT a flag that you can enable or disable, it's simply an indication for whether put back the reasoning_content into history is accepted by the template
• clear_thinking: it is not a llama.cpp notion, just mentioned here because GLM 4.7 template have it; other models can have other naming for this.

(a) detect whether model supports reasoning (b) enable reasoning by default if it does (c) pass reasoning traces if the template supports it (d) accept explicit overrides

• (a) Hmm, could you point me to the code where we detect if a model supports reasoning?
• (b) Aren't we already enabled reasoning by default if model support it?
• (c) You mean reasoning traces parsing (enable_thinking) or preserving reasoning trace inside history (supports_preserve_reasoning)?
• (d) I think it's what this PR is made to do

Edit: I think this PR already provide the 4 points a,b,c,d that you brought up

[pwilkin]

@ngxson yeah, you're right. I was somehow confused that we're already passing the reasoning_content to the template.

[aldehir]

The API supports it, but the WebUI does not. I assume this is setting up the foundation to add first class support in the WebUI.

By support, I mean it'll pass the reasoning in the message objects fed to the template.