Fix Studio safetensors tool reprompt reasoning leakage

[Lyxot]

Fixes #7122

Summary

This updates the Studio safetensors tool loop so the plan-without-action reprompt heuristic classifies visible answer text when a response contains both reasoning and answer content. If the model emits only reasoning, the loop still falls back to that reasoning text so stalled tool-use attempts can be nudged as before.

The regression coverage adds cases for explicit reasoning spans, prefilled reasoning before a generated </think> close marker, re-emitted explicit <think>...</think> spans before visible text, later generated think blocks after a prefilled close, reasoning-only prefilled stalls with and without a close marker, reasoning-only tool-use stalls, and re-prompt history matching the same text surface used by the classifier.

Root Cause

Safetensors reasoning and answer text share the cumulative text stream. The reprompt heuristic was evaluating the raw accumulated text, so a phrase such as Let me ... inside private reasoning could be treated as a visible plan to call a tool even when the model had already produced a normal visible answer. For safetensors templates that prefill an opening <think>, the tool-loop classifier also needs the same prefilled-mode boundary as the route reasoning extractor: text before the first generated close marker is private reasoning, and later generated <think>...</think> blocks remain private reasoning.

Validation

python -m pytest studio/backend/tests/test_safetensors_tool_loop.py -q

318 passed

python -m pytest studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_reasoning_intent_does_not_reprompt_a_visible_answer studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_prefilled_reasoning_intent_does_not_reprompt_a_visible_answer studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_prefilled_reasoning_with_reemitted_think_does_not_reprompt studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_prefilled_reasoning_with_later_think_does_not_reprompt studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_reasoning_only_intent_still_reprompts_and_uses_a_tool studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_prefilled_no_close_reasoning_intent_still_reprompts studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_prefilled_reasoning_prefix_is_kept_for_reasoning_only_reprompt studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_reprompt_history_uses_visible_intent_text studio/backend/tests/test_safetensors_tool_loop.py::TestLoopRePrompt::test_intent_signal_triggers_reprompt -q

9 passed

ruff check studio/backend/core/inference/safetensors_agentic.py studio/backend/core/inference/inference.py studio/backend/core/inference/orchestrator.py studio/backend/routes/inference.py studio/backend/tests/test_safetensors_tool_loop.py

All checks passed!

git diff --check HEAD^..HEAD

passed

[gemini-code-assist]

Code Review

This pull request introduces the _reprompt_intent_text helper function to extract visible text from assistant responses containing <think> or [THINK] reasoning blocks. This ensures that the plan-without-action classifier evaluates only the visible content when present, preventing unnecessary re-prompts when a visible answer is already provided alongside reasoning. Unit tests have been added to verify this behavior. There are no review comments, so no additional feedback is provided.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Copilot]

Pull request overview

This PR adjusts Studio’s safetensors tool-loop “plan-without-action” re-prompt heuristic to ignore intent phrases that appear only inside private reasoning spans, preventing unintended re-prompts and resulting reasoning/content leakage into persisted assistant messages.

Changes:

• Add _reprompt_intent_text() to classify intent using visible answer text (falling back to reasoning only when no visible text exists).
• Update the safetensors re-prompt gate to use the classified intent text.
• Add regression tests covering “reasoning intent + visible answer” and “reasoning-only intent still reprompts and uses a tool”.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
studio/backend/core/inference/safetensors_agentic.py	Introduces visible-vs-reasoning intent extraction and uses it in the safetensors re-prompt gate.
studio/backend/tests/test_safetensors_tool_loop.py	Adds tests to prevent reprompt-trigger leakage when visible answer text is present and to preserve reprompt behavior for reasoning-only stalls.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6939f0504e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 152569dfdd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38b65727e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85598ca0c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[oobabooga]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 9c5e3693e1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Imagineer99]

Looks right overall, and test_safetensors_tool_loop.py passes locally: 315 passed.

One edge case to harden before merge: with reasoning_prefilled=True, text after the first </think> can still contain a later <think>...</think> block. That later private reasoning should be stripped before the reprompt classifier runs.

[Lyxot]

This edge case is already handled now: when reasoning_prefilled=True, _reprompt_intent_text() first trims through the initial generated </think>, then scans the remaining suffix for later explicit <think>...</think> spans. If visible answer text exists, only that visible text is passed to the reprompt classifier.

The regression test covering this is test_prefilled_reasoning_with_later_think_does_not_reprompt, using:

private prefilled planning</think><think>Let me prepare the requested summary carefully.</think>This is the final visible answer.

So the later private think block is stripped and does not trigger a reprompt.

[Imagineer99]

This edge case is already handled now: when reasoning_prefilled=True, _reprompt_intent_text() first trims through the initial generated </think>, then scans the remaining suffix for later explicit <think>...</think> spans. If visible answer text exists, only that visible text is passed to the reprompt classifier.

The regression test covering this is test_prefilled_reasoning_with_later_think_does_not_reprompt, using:

private prefilled planning</think><think>Let me prepare the requested summary carefully.</think>This is the final visible answer.

So the later private think block is stripped and does not trigger a reprompt.

Confirmed on latest head: this now handles the close-then-later-<think> case, and the added regression test covers it.
Validated locally. Looks merge-ready once CI is green.

[oobabooga]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: a18a547809

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[oobabooga]

Nice fix for <think> / [THINK] models. I verified it live: reasoning intent plus a visible answer reprompts on main but not here.

One blocker before it closes #7122: it doesn't cover the model the issue was filed against. Gemma-4 E-series puts reasoning in a plain-text channel, not <think>:

<|channel>thought ...First, ... <channel|>visible answer

_reprompt_intent_text only strips <think>/[THINK], so for gemma it returns the whole stream, reasoning included. Confirmed live on gemma-4-E2B-it (same loop MLX uses), same prompt both ways:

MAIN (no fix):  re-prompt 1/3, 2/3
BRANCH (fix):   re-prompt 1/3, 2/3   # identical, classifier still sees "thought\n..."

Could you extend the classifier to also strip <|channel>thought ... <channel|> spans (and add a gemma-channel test, since every new test is <think>-only), or drop the Fixes #7122 wording so the gemma case stays open?

[Lyxot]

I checked this against #7121: that PR normalizes Gemma native <|channel>thought ... <channel|> output into canonical <think>...</think> before the safetensors/MLX stream reaches this classifier. With #7121 applied first, _reprompt_intent_text() receives <think>...</think>visible answer, strips the reasoning, and does not reprompt.

So I think the clean path is to land #7121 first, then update this PR on top of it. If #7121 does not land first, we should either add a small native-channel fallback here or remove Fixes #7122.

[oobabooga]

#7121 is merged now, and you were right about it. With it in main, gemma's <|channel>thought ... <channel|> is normalized to <think>...</think> before it reaches this classifier, so _reprompt_intent_text strips it and only the visible answer is scored.

I tested the two together on gemma-4-E2B-it with the #7122 repro. The reprompts are gone: the classifier sees the answer ('Learn to tune your guitar...', 'Tokyo') instead of the reasoning, so it doesn't nudge. Without the fix the loop keeps firing them.

CI on this PR merged with current main is green except for test_sidebar_account_block_uses_leading_tight, which already fails on main and is unrelated.

Good to merge, and Fixes #7122 holds with #7121 in. Thanks for the channel normalization.