Skip to content

studio: re-emit prefilled <think> opener on the safetensors stream (fixes #6815) #6832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pjordanandrsn wants to merge 2 commits into from
Closed

studio: re-emit prefilled <think> opener on the safetensors stream (fixes #6815) #6832

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b7d3dcd
studio: re-emit prefilled <think> opener on the safetensors stream
pjordanandrsn Jul 3, 2026
e8db9ce
studio: carry the synthetic <think> opener in every cumulative snapshot
pjordanandrsn Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions studio/backend/core/inference/inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -982,7 +982,23 @@ def _generate_chat_response_inner(
formatted_prompt = self.format_chat_prompt(messages, system_prompt)

# Step 3: generate
yield from self.generate_stream(

# Newer hybrid-thinking templates (Qwen3.5 / Qwen3.6 style) *prefill* the reasoning
# opener in the generation prompt (`...<|im_start|>assistant\n<think>\n`), so the
# model's own output stream is `reasoning...</think>answer` with no opening tag.
# The shared stream protocol marks reasoning with literal <think>...</think> tags
# (the harmony parser above emits them for gpt-oss, and llama-server handles the
# prefilled-opener convention natively on the GGUF path). Without re-emitting the
# opener here, everything the model thinks renders as answer text on the
# safetensors path (#6815). enable_thinking=False prompts end with an *empty*
# `<think>\n\n</think>` block, not the bare opener, so they are unaffected.
# This generator yields *cumulative* snapshots (consumers diff each value
# against the previous one), so the synthetic opener must remain part of
# every subsequent snapshot rather than be emitted once on its own.
think_prefix = "<think>" if formatted_prompt.rstrip().endswith("<think>") else ""
if think_prefix:
yield think_prefix
for cumulative in self.generate_stream(
formatted_prompt,
temperature,
top_p,
Expand All @@ -992,7 +1008,8 @@ def _generate_chat_response_inner(
repetition_penalty,
cancel_event = cancel_event,
_adapter_state = _adapter_state,
)
):
yield think_prefix + cumulative

def _generate_vision_response(
self,
Expand Down