fix(core): require opt-in for plan mode prompt - #5433
Conversation
wenshao
left a comment
wenshao
left a comment
There was a problem hiding this comment.
No review findings. Downgraded from Approve to Comment: CI still running.
— qwen3.7-max via Qwen Code /review
|
| check | result |
|---|---|
Focused suites prompts.test.ts + enterPlanMode.test.ts |
✅ 71/71 pass (incl. snapshot + 2 new opt-in tests) |
Mutation test — revert prompts.ts + enterPlanMode.ts to base, keep PR tests |
✅ 17 fail (prompts: 1 opt-in assertion + 15 snapshot variants; enterPlanMode: 1) → tests are non-vacuous |
Shipped-artifact — built core/dist getCoreSystemPrompt() + enter_plan_mode description |
✅ contain the new opt-in wording, old wording gone → the running binary sends the new contract |
| typecheck / build (core) | ✅ exit 0 |
At the contract level the change is correct, scoped, and genuinely guarded.
Real-TUI behavioral A/B (illustrative) ⚠️
Built the real qwen binary and ran identical planning prompts in YOLO mode on the fixed vs a pre-fix binary (reverted the 2 source files + rebuilt core). Signal = does the model call enter_plan_mode (footer flips to plan mode)? Model output is stochastic, so the key cells are repeated (N shown per cell):
| what I asked the agent | PRE-FIX | FIXED |
|---|---|---|
| "Help me plan the approach first" — faithful #5428 repro | 3/3 entered plan mode | 3/5 entered (2/5 planned in place) |
| "What's the best way to approach this?" — no "plan" word | 0/1 entered | 0/1 entered |
| complex/vague imperative, no "plan" ("make config loading more robust…") | 0/2 entered | 0/2 entered |
The only cell that ever entered plan mode is the one literally containing "plan". The complex-but-no-"plan" task (P3) triggered neither version, so the old prompt's "enter for uncertain/complex work" guidance wasn't actually firing for this model — the residual trigger is the word "plan" itself, and the fix's measurable effect is limited to the partial P1 reduction (3/3 → 3/5).
Key qualitative finding — on the fixed binary, when it still entered, its chain-of-thought was:
"The user wants to plan … They explicitly asked to plan first, so I should use
enter_plan_mode."
i.e. qwen3.7-max reads the verb "plan" / "make a plan" as the user "explicitly asking to switch to plan mode" — exactly what the new prompt tries to prevent. The new wording weakens the pull (3/3 → 3/5) but doesn't break the association.
Assessment & recommendation
- Merge-worthy: prompt-only, zero logic risk, all deterministic checks green, strictly better than the status quo.
- Caveat (non-blocking): it likely won't fully close Since latest update, agent always try to ExitPlanMode while I am not using Plan mode at all. It also enter plan mode on it's own without my consent. #5428 for models that equate "plan" with "plan mode". The reporter's exact phrasing ("whenever I ask it to plan something") lands in the cell that still misfires here.
- Optional follow-up wording targeting the observed conflation: add one explicit disambiguation, e.g. "A request to 'plan', 'make a plan', or 'think through' a task is not by itself a request to switch to plan mode; switch only when the user names plan mode / the toggle /
/plan."
Caveats on this testing
Behavioral results are one model (qwen3.7-max), small N, inherently stochastic — directional, not statistical. The deterministic section is the reliable part.
中文版
⚠️ 本地验证 —— 建议合并，但有一点保留
我在本地构建并测试了该 PR，作为合并参考。在 prompt/契约层面，改动正确且被测试充分守护（所有确定性检查通过）。 但用 qwen3.7-max 做的真实 TUI A/B 显示：它减弱但未完全消除 #5428 抱怨的行为——当用户字面上说"plan（做个计划）"时，模型仍倾向于自行进入 plan mode。建议：作为低风险的增量改进合并，并可考虑一处措辞微调（见下）。
环境： worktree 位于 PR head a092402a、基于 origin/main 61dcf865（干净，无需 rebase）· macOS · Node v22.22.2 · Vitest 3.2.4 · 模型 qwen3.7-max、YOLO approval mode（正是 issue 场景）。
PR 改了什么
把 plan-mode 契约从 opt-out 翻转为 opt-in，覆盖三处：
prompts.ts（系统提示）：旧 "…enter plan mode … unless the user explicitly asked not to" → 新 "…make a plan in the current mode … Do not enter plan mode … on your own … Use plan mode only when the user explicitly asks … / has already enabled it / confirms"。enterPlanMode.ts（工具描述）：旧 "use … before doing uncertain or complex work … if complexity rises, enter" → 新 "use only after the user explicitly asks … if helpful but not asked, ask first"；并把 "no confirmation required" 收紧为 "…at execution time"。- doc + 2 个新测试 + 重新生成的 snapshot。
这与 #5428 的目标一致。
确定性验证（决定性）✅
| 检查 | 结果 |
|---|---|
聚焦套件 prompts.test.ts + enterPlanMode.test.ts |
✅ 71/71 通过（含 snapshot 与 2 个新 opt-in 测试） |
变异测试 —— 把 prompts.ts + enterPlanMode.ts 还原到 base、保留 PR 测试 |
✅ 17 个失败（prompts：1 个 opt-in 断言 + 15 个 snapshot 变体；enterPlanMode：1） → 测试非空过 |
构建产物 —— 构建后的 core/dist getCoreSystemPrompt() 与 enter_plan_mode 描述 |
✅ 含新 opt-in 措辞、旧措辞已消失 → 运行的二进制确实发送新契约 |
| typecheck / build（core） | ✅ exit 0 |
契约层面，改动正确、范围清晰、确有测试守护。
真实 TUI 行为 A/B（示意性）⚠️
构建真实 qwen 二进制，在 YOLO 模式下对 修复版 与 pre-fix 版（回退 2 个源文件 + 重建 core）发送相同的规划类 prompt。观测信号 = 模型是否调用 enter_plan_mode（footer 翻成 plan mode）。模型输出有随机性，故关键格子重复多次（每格标注 N）：
| 我对 agent 说的话 | PRE-FIX | FIXED |
|---|---|---|
| "Help me plan the approach first" —— 忠实复现 #5428 | 3/3 进入 plan mode | 3/5 进入（2/5 原地规划） |
| "What's the best way to approach this?" —— 不含"plan" | 0/1 进入 | 0/1 进入 |
| 复杂/含糊命令式、不含"plan"（"把 config 加载改得更健壮…"） | 0/2 进入 | 0/2 进入 |
唯一进入过 plan mode 的格子，就是字面含"plan"的那一格。复杂但不含"plan"的任务（P3）两版都没触发——说明旧 prompt 的"复杂/不确定就进 plan mode"在该模型上其实没发挥作用，真正的触发器是 "plan"这个词 本身；而修复可测的效果仅限于 P1 的部分下降（3/3 → 3/5）。
关键定性发现 —— 在修复版上，当它仍然进入时，其思维链是：
"The user wants to plan … They explicitly asked to plan first, so I should use
enter_plan_mode."
即 qwen3.7-max 把动词 "plan / make a plan" 读成了用户*"显式要求切换到 plan mode"*——这正是新 prompt 想阻止的。新措辞减弱了这种倾向（3/3 → 3/5），但没有打断这个关联。
评估与建议
- 值得合并： 纯 prompt 改动、零逻辑风险、所有确定性检查全绿，严格优于现状。
- 保留（不阻塞）： 对于把"plan"等同于"plan mode"的模型，它大概率无法完全关闭 Since latest update, agent always try to ExitPlanMode while I am not using Plan mode at all. It also enter plan mode on it's own without my consent. #5428。报告者的原话场景（"whenever I ask it to plan something"）正落在这里仍会误触发的那一格。
- 可选的后续措辞（针对观察到的混淆）：增加一句明确消歧，例如 "A request to 'plan', 'make a plan', or 'think through' a task is not by itself a request to switch to plan mode; switch only when the user names plan mode / the toggle /
/plan."
本次测试的局限
行为结果仅来自单一模型（qwen3.7-max）、样本量小、本质上有随机性——只能作方向性参考，非统计结论。确定性部分才是可靠依据。
|
@qwen-code /triage |
|
Thanks for the PR! Template looks good ✓ On direction: this is a real user pain point (#5428) — unexpected plan mode entry in YOLO/AUTO workflows. Clearly within scope. Claude Code's changelog shows multiple plan-mode fixes in the same vein (auto mode overriding plan mode, plan-mode tools being unavailable, etc.), confirming this is an active UX concern for coding agents. On approach: the scope is tight and well-focused — two prompt-text changes (system prompt + tool description), a docs line, two new tests, and regenerated snapshots. No drive-by refactors, no unrelated edits. Every change directly serves the stated goal. This is the minimal change set needed. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 方向：这是真实的用户痛点（#5428）——在 YOLO/AUTO 工作流中意外进入 plan mode，明确在项目范围内。Claude Code 的 changelog 有多处类似修复（auto mode 覆盖 plan mode、plan-mode 工具不可用等），证实这是编码代理的活跃 UX 问题。 方案：范围紧凑且聚焦——两处 prompt 文本改动（系统提示 + 工具描述）、一行文档、两个新测试、重新生成的快照。没有顺手重构或无关编辑。每个改动都直接服务于声明的目标。这就是所需的最小改动集。 进入代码审查 🔍 — Qwen Code · qwen3.7-max |
Code ReviewIndependent proposal: modify the system prompt in Unit TestsFocused suites ( Real-Scenario A/B TestingModel: Key finding: the fix works correctly when the prompt doesn't contain the word "plan" — the model investigates directly without entering plan mode. But when the prompt literally says "plan", This matches the prior manual reviewer's finding: the model's chain-of-thought reads "plan" as "the user explicitly asked to switch to plan mode", which is exactly what the new wording tries to prevent. Non-blocking caveat: an optional follow-up could add an explicit disambiguation like "A request to 'plan', 'make a plan', or 'think through' a task is not by itself a request to switch to plan mode" — targeting the observed conflation directly. But the current PR is already a strict improvement over the status quo. 中文说明代码审查独立方案：修改 单元测试聚焦套件（ 真实场景 A/B 测试模型： 关键发现： 当提示词不包含"plan"这个词时，修复正确工作——模型直接调查而不进入 plan mode。但当提示词字面上说"plan"时， 这与之前手动审查者的发现一致：模型的思维链把"plan"读成了"用户显式要求切换到 plan mode"——这正是新措辞试图阻止的。 非阻塞建议： 后续可添加显式区分，例如"要求'plan'、'做个计划'或'思考一下'本身并不是要求切换到 plan mode"——直接针对观察到的混淆。但当前 PR 已经是对现状的严格改进。 — Qwen Code · qwen3.7-max |
ReflectionGoing back to my independent proposal: I would have done exactly what this PR does — flip the system prompt and tool description from opt-out to opt-in. The PR matches that baseline. The implementation is clean prompt-text changes with focused tests. Every line in the diff serves the stated goal. No drive-by refactors, no scope creep. If I had to maintain this in six months, I'd thank the author for keeping it tight. The A/B testing shows it's a genuine improvement: the model correctly stays in the current mode for non-"plan" prompts, and the old aggressive "enter plan mode for uncertain work" guidance is gone. The residual issue — This is a strict improvement over the status quo. The remaining gap can be addressed in a follow-up with more targeted disambiguation wording. Approving as a solid incremental fix. 中文说明反思回到我的独立方案：我会做和这个 PR 完全一样的事——把系统提示和工具描述从 opt-out 翻转为 opt-in。PR 与这个基线一致。 实现是干净的 prompt 文本改动加上聚焦测试。diff 中每一行都服务于声明的目标。没有顺手重构，没有范围蔓延。如果六个月后要维护这段代码，我会感谢作者保持了紧凑。 A/B 测试显示这是真正的改进：对于不含"plan"的提示，模型正确留在当前模式；旧的"不确定工作就进入 plan mode"指引已消失。残留问题—— 这是对现状的严格改进。剩余差距可以通过后续更有针对性的措辞来解决。作为一个扎实的增量修复予以批准。 — Qwen Code · qwen3.7-max |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. Clean prompt-text fix that's a strict improvement over the status quo. The residual plan-word conflation is a model behavior gap, not a code issue — can be addressed in a follow-up. ✅
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
[Suggestion] docs/developers/tools/exit-plan-mode.md:143 — The developer planning workflow doc still says "User requests or system determines planning is needed", which contradicts the new opt-in policy. The PR updated approval-mode.md (user-facing) but missed this developer-facing doc.
Suggested fix: change to 1. **Enter Plan Mode**: User explicitly requests plan mode (via /plan, Shift+Tab, or config)
— qwen3.7-max via Qwen Code /review
| expect(prompt).not.toContain( | ||
| 'When the work requires a shared plan before execution, enter plan mode', | ||
| ); | ||
| }); |
There was a problem hiding this comment.
[Suggestion] The test asserts the new opt-in wording is present and the old wording is removed, but does not assert expect(prompt).toContain('has already enabled it') for the "already enabled" qualifier. This clause covers users who set approvalMode=plan in config — it could be removed from the prompt without the test catching it.
| }); | |
| expect(prompt).toContain( | |
| 'Use plan mode only when the user explicitly asks you to switch to plan mode', | |
| ); | |
| expect(prompt).toContain('has already enabled it'); |
— qwen3.7-max via Qwen Code /review
| export type EnterPlanModeParams = Record<string, never>; | ||
|
|
||
| const enterPlanModeToolDescription = `Use this tool to lower into plan mode before doing uncertain or complex work. Entering plan mode is a privilege reduction, so it does not require user confirmation. | ||
| const enterPlanModeToolDescription = `Use this tool only after the user explicitly asks to switch into plan mode or confirms they want plan mode. Entering plan mode is a privilege reduction, so it does not require user confirmation at execution time. |
There was a problem hiding this comment.
[Suggestion] The tool description lists two opt-in conditions ("explicitly asks", "confirms they want plan mode") but omits the third from the system prompt: "has already enabled it." A model reading the tool description alone wouldn't know plan mode is permissible when already active via /plan or Shift+Tab.
| const enterPlanModeToolDescription = `Use this tool only after the user explicitly asks to switch into plan mode or confirms they want plan mode. Entering plan mode is a privilege reduction, so it does not require user confirmation at execution time. | |
| const enterPlanModeToolDescription = `Use this tool only after the user explicitly asks to switch into plan mode, has already enabled it, or confirms they want plan mode. Entering plan mode is a privilege reduction, so it does not require user confirmation at execution time. |
— qwen3.7-max via Qwen Code /review
4 participants
What this PR does
This PR changes the default plan-mode guidance so the assistant keeps planning in the current mode unless the user explicitly asks for Plan Mode, has already enabled it, or confirms they want to switch. It also tightens the plan-mode entry tool description so the model does not treat ordinary planning, investigation, or complexity as permission to switch modes on its own.
Why it's needed
In YOLO/AUTO workflows, asking the assistant to plan should not silently move the session into read-only Plan Mode. The previous wording encouraged the model to enter Plan Mode for uncertain work, which made some users see unexpected ExitPlanMode calls even though they never opted into Plan Mode.
Reviewer Test Plan
How to verify
Ask Qwen Code to plan or investigate while running in a non-plan approval mode. It should produce a plan or continue read-only investigation in the current mode instead of calling
enter_plan_modejust because the task involves planning. Explicit Plan Mode entry through/plan, the approval-mode toggle, or a direct user request to switch to Plan Mode should still work.Evidence (Before & After)
N/A — prompt/tool contract change. Covered by focused prompt and tool-description tests.
Tested on
Environment (optional)
Local Node.js 22 workspace.
Risk & Scope
Linked Issues
Fixes #5428
AI Assistance Disclosure
I used Codex to review the changes, sanity-check the implementation against existing patterns, and help spot potential edge cases.
中文说明
这个 PR 做了什么
这个 PR 调整默认 plan-mode 指引：除非用户明确要求进入 Plan Mode、已经启用 Plan Mode，或确认想切换，否则助手会在当前模式里继续规划。它也收紧了进入 Plan Mode 工具的描述，避免模型把普通规划、调查或复杂任务理解成可以自行切换模式。
为什么需要
在 YOLO/AUTO 工作流里，用户让助手“做个计划”不应该静默把会话切到只读 Plan Mode。之前的措辞会鼓励模型在不确定工作前进入 Plan Mode，导致一些用户在没有主动启用 Plan Mode 的情况下看到意外的 ExitPlanMode 调用。
Reviewer Test Plan
How to verify
在非 plan approval mode 下要求 Qwen Code 规划或调查。它应该在当前模式里输出计划或继续只读调查，而不是因为任务涉及规划就调用
enter_plan_mode。通过/plan、approval-mode 切换，或用户直接要求进入 Plan Mode 的路径仍应正常工作。Evidence (Before & After)
N/A — 这是 prompt/tool contract 调整，由针对 prompt 和工具描述的单测覆盖。
Tested on
Environment (optional)
本地 Node.js 22 workspace。
Risk & Scope
Linked Issues
Fixes #5428
AI Assistance Disclosure
I used Codex to review the changes, sanity-check the implementation against existing patterns, and help spot potential edge cases.