feat: execution-discipline guidance now reaches DeepSeek/Kimi/Qwen-class models (Composio eval findings) - #90335
Merged
Merged
Conversation
…ls (config model.execution_guidance) Un-fences OPENAI_MODEL_EXECUTION_GUIDANCE from the gpt/codex/grok substring check and gives it its own injection gate, independent of tool_use_enforcement, controlled by config.yaml `agent.execution_guidance` (auto/true/false/list — same semantics as tool_use_enforcement). The "auto" list (EXECUTION_GUIDANCE_MODELS) now also covers deepseek, kimi, qwen, glm, minimax, mimo, and mistral. Composio agentic-eval traces showed Hermes+DeepSeek/Kimi failing where competitors passed: financial math done in prose, no read-back after external writes, malformed identifiers "repaired", completeness claimed despite count mismatches. The discipline block existed but those models never received it. The block is extended with compact clauses distilled from that analysis: - external-write read-back (tool-call success is not task success; internal file edits already confirmed by the tool are not re-verified) - count reconciliation (declared totals/has_more are hard assertions) - literal preservation (never normalize identifiers that fail a stated format; lookup success does not validate a malformed token) - retry-differently (empty/partial/suspiciously narrow results get a broader retry before concluding) - completion gated on verification (done = every named acceptance criterion verified, never a plausible subset) The todo tool description now encourages enumeration-as-checklist for "all N items" tasks and gates completed status on verified work, never intent. Guidance is chosen once at session start keyed on model name, so the system prompt stays byte-stable for the life of a conversation. Supersedes/absorbs prior contributor proposals: #20588, #35087, #41874 (MiMo), #53847 (GLM tool-calls-as-text stall). Co-authored-by: Mat-London <56627804+Mat-London@users.noreply.github.com> Co-authored-by: intelac <8803887+intelac@users.noreply.github.com> Co-authored-by: 6ylqq <51219463+6ylqq@users.noreply.github.com> Co-authored-by: tauros1983 <267660491+tauros1983@users.noreply.github.com>
alt-glitch
added
type/feature
Contributor
૮ >ﻌ< ა ci reviewran on 6cb5a86 — feat: execution-discipline guidance now reaches all tool-cap
|
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Execution-discipline guidance now reaches every major tool-capable model family — previously only models whose name contained gpt/codex/grok received
OPENAI_MODEL_EXECUTION_GUIDANCE, so DeepSeek/Kimi/Qwen/GLM/MiMo/MiniMax/Mistral sessions ran with zero verification/computation pressure in the system prompt.Root cause of a measurable share of Composio's published eval failures (Hard-30, DeepSeek V4 + Kimi K3 campaigns): trace analysis showed Hermes doing financial rule-math in prose, skipping read-back after external writes, "repairing" malformed identifiers, and finalizing despite count mismatches — exactly the failure modes this block addresses, on models that weren't getting it.
Changes
agent/prompt_builder.py:EXECUTION_GUIDANCE_MODELSauto-list (gpt, codex, grok + deepseek, kimi, qwen, glm, minimax, mimo, mistral); block extended with 5 clauses distilled from winning-harness research: external-write read-back (with a don't-re-verify-internal-edits counterweight), count reconciliation on declared totals/has_more, literal preservation (lookup success ≠ valid malformed token), retry-differently on suspiciously-narrow results, completion-gated-on-verification.agent/system_prompt.py: guidance gets its own injection gate (auto/true/false/list viaagent.execution_guidance), no longer nested under tool-use enforcement.tools/todo_tool.py: description adds enumeration-as-checklist for "all N items" tasks + "completed only after verification, never intent".configuration.md), tests (injection matrix, model-list contracts, byte-stability).Validation
build_system_prompt, temp HERMES_HOME)Targeted tests: 104 passed (1 pre-existing cross-file flake on main, unrelated —
test_build_system_prompt_records_stable_prefixfails identically on origin/main). E2E injection matrix verified with real AIAgent construction.Prior contributor work credited (same direction, narrower scope): #20588 @Mat-London, #35087 @intelac, #41874 @6ylqq, #53847 @tauros1983 — co-authored.
Infographic