fix(agent): inject OPENAI_MODEL_EXECUTION_GUIDANCE for GLM/Qwen/DeepSeek - #1
Closed
tauros1983 wants to merge 1 commit into
Closed
fix(agent): inject OPENAI_MODEL_EXECUTION_GUIDANCE for GLM/Qwen/DeepSeek#1tauros1983 wants to merge 1 commit into
tauros1983 wants to merge 1 commit into
Conversation
Telegram sessions on z-ai/glm-5.2 stalled after one assistant turn:
the model replied with a plain-text '[TOOL_CALL]...[/TOOL_CALL]'
marker instead of a structured tool_calls JSON block, so the runtime
saw tool_calls=None and finished with finish_reason=stop. Two
sessions on 2026-06-27 (20260627_144930_7531d0bd, 20260627_145132_
3c5abc52) reproduced this — 3 messages, 0 tool_calls each, then
silence.
Root cause: build_system_prompt_parts only appended
OPENAI_MODEL_EXECUTION_GUIDANCE when the model name contained
'gpt', 'codex', or 'grok'. GLM/Qwen/DeepSeek matched
TOOL_USE_ENFORCEMENT_MODELS (so they got the lighter
TOOL_USE_ENFORCEMENT_GUIDANCE block) but not the execution-discipline
block (tool persistence, mandatory_tool_use, anti-fabrication). On a
weaker tool_use format-trained model that lighter block alone is not
enough to stop the model from emitting its tool intent as natural-
language markdown.
The OPENAI_ prefix reflects origin, not exclusivity (see the comment
on prompt_builder.OPENAI_MODEL_EXECUTION_GUIDANCE). Expand the gate
to the non-Google slice of TOOL_USE_ENFORCEMENT_MODELS — Google keeps
its own more specific GOOGLE_MODEL_OPERATIONAL_GUIDANCE block, so we
skip gemini/gemma here to avoid a duplicate parallel-tool-call steer.
Tests: 8 regression tests in TestOpenAIExecutionGuidanceInjection cover
GLM/DeepSeek/Qwen (positive), GPT/Grok (no regression), Opus/Gemini
(no false-positive), and the off-switch. 169/169 pass in
test_prompt_builder.py + test_system_prompt.py; full suite minus the
unrelated pre-existing test_anthropic_adapter.py MagicMock failure
also clean.
Reproduction evidence (from ~/.hermes/state.db on 2026-06-27):
SELECT message_count, tool_call_count, end_reason FROM sessions
WHERE id IN ('20260627_144930_7531d0bd','20260627_145132_3c5abc52');
-> 3|0|| (both)
SELECT substr(content, 1, 200) FROM messages
WHERE session_id='20260627_144930_7531d0bd' AND role='assistant';
-> 'Ik ga die 3 commits lezen. Eerst kijken waar ik sta...
[TOOL_CALL]\nrun_command\ncommand=cd /Users/guidolassally/esmi-dashboard
2>/dev/null && git log --oneline -1 cc3cd6b92 && ...'
finish_reason was 'stop' and tool_calls was NULL on both sessions.
Live verification (after patch):
python -c 'from unittest.mock import patch; from types import
SimpleNamespace; ... build_system_prompt_parts(agent)["stable"]'
for z-ai/glm-5.2 now contains 'Execution discipline',
'<tool_persistence>', and '<mandatory_tool_use>' sections that
previously were absent.
Owner
Author
|
Superseded by upstream PR NousResearch#53847. Same branch, same commit, same diff. The fork PR is redundant now that the upstream PR is open. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Telegram sessions on weaker tool-use-format-trained models (GLM-5.2 in particular) stalled after a single assistant turn: the model emitted its tool intent as a plain-text
[TOOL_CALL]...[/TOOL_CALL]markdown block instead of a structuredtool_callsJSON array, so the runtime sawtool_calls=Noneand finished withfinish_reason=stopafter one turn.Two Telegram sessions from 2026-06-27 reproduced this (IDs
20260627_144930_7531d0bdand20260627_145132_3c5abc52) — 3 messages each, 0 tool calls each, then silence.Root cause
build_system_prompt_partsinjectedOPENAI_MODEL_EXECUTION_GUIDANCE(the strong tool-persistence + mandatory_tool_use + anti-fabrication block) only when the model name containedgpt,codex, orgrok. GLM/Qwen/DeepSeek matchedTOOL_USE_ENFORCEMENT_MODELSso they got the lighterTOOL_USE_ENFORCEMENT_GUIDANCEblock — but not the execution-discipline block that specifically steers the model away from emitting plans instead of executing. On a weaker tool_use format-trained model that lighter block alone is not enough to stop the model from emitting its tool intent as natural-language markdown.The block's own comment already noted "body is family-agnostic; the OPENAI_ prefix reflects origin, not exclusivity" — the gate just wasn't widened to match the comment.
Fix
agent/system_prompt.py:257— expand the gate to the non-Google slice ofTOOL_USE_ENFORCEMENT_MODELS(i.e.gpt,codex,grok,glm,qwen,deepseek). Google models are excluded to avoid duplicating the parallel-tool-call steer thatGOOGLE_MODEL_OPERATIONAL_GUIDANCEalready provides.19 lines added, 4 removed. Two files:
agent/system_prompt.py— gate expansion + comment updatetests/agent/test_system_prompt.py— 8 new regression testsTests
8 new regression tests in
TestOpenAIExecutionGuidanceInjection:test_glm_5_2_receives_openai_execution_guidance(the exact model from the stall)test_deepseek_receives_openai_execution_guidancetest_qwen_receives_openai_execution_guidancetest_gpt_still_receives_openai_execution_guidance(no regression)test_grok_still_receives_openai_execution_guidance(no regression)test_anthropic_opus_does_not_receive_openai_execution_guidance(no false positive)test_google_gemini_does_not_receive_openai_execution_guidance(no duplicate steer)test_disabled_when_enforcement_off169/169 tests pass in
tests/agent/test_prompt_builder.py+tests/agent/test_system_prompt.py. Pre-existingtest_anthropic_adapter.py::TestRunOauthSetupToken::test_returns_token_from_credential_filesfailure is unrelated (MagicMock JSONDecode error onmainHEAD without this patch; reproduced withgit stash).ruff+tyboth clean.Reproduction evidence
System prompt size also showed the cause: Telegram sessions received 19,600 chars of system prompt vs 38,666 chars for CLI sessions — the lighter guidance was the only one Telegram got.
Live verification (after patch)
Running
build_system_prompt_partswith a mocked agent for each model name now produces:E2E confirmation requires a fresh Telegram session with GLM-5.2 after merge — that's a downstream verification step for the user, not something the unit tests can cover.