Skip to content

fix(agent): inject OPENAI_MODEL_EXECUTION_GUIDANCE for GLM/Qwen/DeepSeek - #1

Closed
tauros1983 wants to merge 1 commit into
mainfrom
fix/openai-exec-guidance-for-glm
Closed

fix(agent): inject OPENAI_MODEL_EXECUTION_GUIDANCE for GLM/Qwen/DeepSeek#1
tauros1983 wants to merge 1 commit into
mainfrom
fix/openai-exec-guidance-for-glm

Conversation

@tauros1983

Copy link
Copy Markdown
Owner

Summary

Telegram sessions on weaker tool-use-format-trained models (GLM-5.2 in particular) stalled after a single assistant turn: the model emitted its tool intent as a plain-text [TOOL_CALL]...[/TOOL_CALL] markdown block instead of a structured tool_calls JSON array, so the runtime saw tool_calls=None and finished with finish_reason=stop after one turn.

Two Telegram sessions from 2026-06-27 reproduced this (IDs 20260627_144930_7531d0bd and 20260627_145132_3c5abc52) — 3 messages each, 0 tool calls each, then silence.

Root cause

build_system_prompt_parts injected OPENAI_MODEL_EXECUTION_GUIDANCE (the strong tool-persistence + mandatory_tool_use + anti-fabrication block) only when the model name contained gpt, codex, or grok. GLM/Qwen/DeepSeek matched TOOL_USE_ENFORCEMENT_MODELS so they got the lighter TOOL_USE_ENFORCEMENT_GUIDANCE block — but not the execution-discipline block that specifically steers the model away from emitting plans instead of executing. On a weaker tool_use format-trained model that lighter block alone is not enough to stop the model from emitting its tool intent as natural-language markdown.

The block's own comment already noted "body is family-agnostic; the OPENAI_ prefix reflects origin, not exclusivity" — the gate just wasn't widened to match the comment.

Fix

agent/system_prompt.py:257 — expand the gate to the non-Google slice of TOOL_USE_ENFORCEMENT_MODELS (i.e. gpt, codex, grok, glm, qwen, deepseek). Google models are excluded to avoid duplicating the parallel-tool-call steer that GOOGLE_MODEL_OPERATIONAL_GUIDANCE already provides.

19 lines added, 4 removed. Two files:

  • agent/system_prompt.py — gate expansion + comment update
  • tests/agent/test_system_prompt.py — 8 new regression tests

Tests

8 new regression tests in TestOpenAIExecutionGuidanceInjection:

  • test_glm_5_2_receives_openai_execution_guidance (the exact model from the stall)
  • test_deepseek_receives_openai_execution_guidance
  • test_qwen_receives_openai_execution_guidance
  • test_gpt_still_receives_openai_execution_guidance (no regression)
  • test_grok_still_receives_openai_execution_guidance (no regression)
  • test_anthropic_opus_does_not_receive_openai_execution_guidance (no false positive)
  • test_google_gemini_does_not_receive_openai_execution_guidance (no duplicate steer)
  • test_disabled_when_enforcement_off

169/169 tests pass in tests/agent/test_prompt_builder.py + tests/agent/test_system_prompt.py. Pre-existing test_anthropic_adapter.py::TestRunOauthSetupToken::test_returns_token_from_credential_files failure is unrelated (MagicMock JSONDecode error on main HEAD without this patch; reproduced with git stash). ruff + ty both clean.

Reproduction evidence

-- sqlite3 ~/.hermes/state.db
SELECT message_count, tool_call_count, end_reason FROM sessions
WHERE id IN ('20260627_144930_7531d0bd','20260627_145132_3c5abc52');
-- 3|0||   (both sessions)

SELECT substr(content, 1, 250) FROM messages
WHERE session_id='20260627_144930_7531d0bd' AND role='assistant';
-- 'Ik ga die 3 commits lezen. Eerst kijken waar ik sta en wat de context is.
--  [TOOL_CALL]
--  run_command
--  command=cd /Users/guidolassally/esmi-dashboard 2>/dev/null && ...'
-- finish_reason='stop', tool_calls=NULL

System prompt size also showed the cause: Telegram sessions received 19,600 chars of system prompt vs 38,666 chars for CLI sessions — the lighter guidance was the only one Telegram got.

Live verification (after patch)

Running build_system_prompt_parts with a mocked agent for each model name now produces:

model tool_enforce openai_exec (was) openai_exec (now) google_ops
z-ai/glm-5.2 yes no yes no
openai/gpt-5.5 yes yes yes no
xai/grok-4 yes yes yes no
deepseek/deepseek-v4-pro yes no yes no
google/gemini-2.5-pro yes no no yes
anthropic/claude-opus-4.8 yes no no no

E2E confirmation requires a fresh Telegram session with GLM-5.2 after merge — that's a downstream verification step for the user, not something the unit tests can cover.

Telegram sessions on z-ai/glm-5.2 stalled after one assistant turn:
the model replied with a plain-text '[TOOL_CALL]...[/TOOL_CALL]'
marker instead of a structured tool_calls JSON block, so the runtime
saw tool_calls=None and finished with finish_reason=stop. Two
sessions on 2026-06-27 (20260627_144930_7531d0bd, 20260627_145132_
3c5abc52) reproduced this — 3 messages, 0 tool_calls each, then
silence.

Root cause: build_system_prompt_parts only appended
OPENAI_MODEL_EXECUTION_GUIDANCE when the model name contained
'gpt', 'codex', or 'grok'. GLM/Qwen/DeepSeek matched
TOOL_USE_ENFORCEMENT_MODELS (so they got the lighter
TOOL_USE_ENFORCEMENT_GUIDANCE block) but not the execution-discipline
block (tool persistence, mandatory_tool_use, anti-fabrication). On a
weaker tool_use format-trained model that lighter block alone is not
enough to stop the model from emitting its tool intent as natural-
language markdown.

The OPENAI_ prefix reflects origin, not exclusivity (see the comment
on prompt_builder.OPENAI_MODEL_EXECUTION_GUIDANCE). Expand the gate
to the non-Google slice of TOOL_USE_ENFORCEMENT_MODELS — Google keeps
its own more specific GOOGLE_MODEL_OPERATIONAL_GUIDANCE block, so we
skip gemini/gemma here to avoid a duplicate parallel-tool-call steer.

Tests: 8 regression tests in TestOpenAIExecutionGuidanceInjection cover
GLM/DeepSeek/Qwen (positive), GPT/Grok (no regression), Opus/Gemini
(no false-positive), and the off-switch. 169/169 pass in
test_prompt_builder.py + test_system_prompt.py; full suite minus the
unrelated pre-existing test_anthropic_adapter.py MagicMock failure
also clean.

Reproduction evidence (from ~/.hermes/state.db on 2026-06-27):

  SELECT message_count, tool_call_count, end_reason FROM sessions
  WHERE id IN ('20260627_144930_7531d0bd','20260627_145132_3c5abc52');
  -> 3|0||  (both)

  SELECT substr(content, 1, 200) FROM messages
  WHERE session_id='20260627_144930_7531d0bd' AND role='assistant';
  -> 'Ik ga die 3 commits lezen. Eerst kijken waar ik sta...
      [TOOL_CALL]\nrun_command\ncommand=cd /Users/guidolassally/esmi-dashboard
      2>/dev/null && git log --oneline -1 cc3cd6b92 && ...'

  finish_reason was 'stop' and tool_calls was NULL on both sessions.

Live verification (after patch):

  python -c 'from unittest.mock import patch; from types import
  SimpleNamespace; ... build_system_prompt_parts(agent)["stable"]'
  for z-ai/glm-5.2 now contains 'Execution discipline',
  '<tool_persistence>', and '<mandatory_tool_use>' sections that
  previously were absent.
@tauros1983

Copy link
Copy Markdown
Owner Author

Superseded by upstream PR NousResearch#53847. Same branch, same commit, same diff. The fork PR is redundant now that the upstream PR is open.

@tauros1983 tauros1983 closed this Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant