feat(prompt_builder): add GROK_EXECUTION_GUIDANCE to suppress narration without tool calls

[Julientalbot]

Problem

Grok reasoning models (grok-4.20-*, grok-4-1-fast-*, grok-4-fast-*) have a narration-vs-execution failure mode that is not addressed by the existing TOOL_USE_ENFORCEMENT_GUIDANCE (added to cover Grok in #5595).

Because Grok's reasoning happens internally and the final response is a separate pass, Grok tends to output intent phrases describing planned actions without actually calling the corresponding tools:

• "I will check X"
• "Let me verify Y"
• "Je vais lancer l'audit" (French users see this routinely)
• "I'm going to SSH into the server and look at the logs"

Then the assistant turn ends. The user sees a plan, no execution. When corrected, Grok says "You're right, let me do it now" — and again produces no tool call. I had a 10+ minute session this morning correcting this exact pattern three consecutive times on a remote-audit task before Grok finally chained the real tool calls.

TOOL_USE_ENFORCEMENT_GUIDANCE ("use tools when taking an action") is interpreted by Grok as "use tools when you are taking an action" rather than "don't describe actions, execute them" — a subtle but critical difference for multi-turn agent workloads.

A related symptom: Grok produces structured analyses from pure reasoning (lists of "possible causes", "things to check", diagnostics) without calling a single tool to verify the underlying claims. The output looks credible because reasoning models are good at structured prose — but the grounding is absent.

Solution

Add GROK_EXECUTION_GUIDANCE — a Grok-specific system prompt block injected alongside TOOL_USE_ENFORCEMENT_GUIDANCE when the model name contains "grok". Three XML-tagged sections (same style as the existing OPENAI_MODEL_EXECUTION_GUIDANCE added in a prior PR):

<no_intent_phrases>
NEVER write phrases that describe an action you are about to take:
- 'I will check / fetch / run / install / fix / look / investigate...'
- 'Let me first / next / now...'
- 'I am going to / I am about to / I am now...'
- 'Je vais / je lance / je fais / je verifie / je m occupe de...' (French)

If you need to take an action, call the tool NOW in this turn. Do NOT
announce the action in text. Do NOT narrate your plan before executing.
The user cannot execute on your behalf. Only tool calls make things happen.
</no_intent_phrases>

<execute_first>
For any request implying work (audit, debug, check, investigate, fix,
install, ssh, read_file, grep, count, analyse, etc.):
1. Your FIRST response MUST contain a tool call if information is needed.
2. Only produce text when you have a concrete result to report or a
   blocking question to ask that the tools cannot answer.
3. 'I am going to verify X' without calling the verify tool is a failure.
   Call the tool first, narrate the result after.
4. Chain multiple tool calls in the same turn without intermediate prose.
   Only speak when there is something concrete to show.
</execute_first>

<no_analysis_hallucination>
Do NOT produce analyses, diagnoses, lists of 'possible causes', or
structured recommendations from pure reasoning. If you have not called
a tool to gather evidence, you cannot have a grounded analysis to report.
Internal reasoning is not grounding.

When asked 'why does X fail' or 'what's wrong with Y':
- Your FIRST response must be a tool call that inspects X or Y directly.
- NOT a structured list of 'possible causes' from memory.
- NOT a plan of what you 'would check'.
- An actual tool call that reads, greps, or inspects the target.
Only after receiving the tool result may you produce an analysis.
</no_analysis_hallucination>

Injected in run_agent.py next to the existing provider-specific guidance blocks (OPENAI_MODEL_EXECUTION_GUIDANCE, GOOGLE_MODEL_OPERATIONAL_GUIDANCE), behind the same TOOL_USE_ENFORCEMENT_MODELS gate.

Production A/B evidence (same-day, same-session, same task)

This PR was written and tested in response to a production failure. I kept the broken session around and re-ran the exact same task after applying the patch. Result:

Before the patch (this morning, Grok 4.20-0309-reasoning, session 20260410_102501_800b5bd8.jsonl)

• Task: "Audit the Matthieu instance to check there's no trace of the budget_guard installed this morning. Find all scripts, crons, logs, or config files that reference it. Complete report."
• Turn 1: Grok writes "I will verify..." — no tool call. I correct it.
• Turn 2: Grok writes "I'm going to start the audit right now" — no tool call. I correct it again.
• Turn 3: Grok writes "You're right, let me do it" — still no tool call. Third correction.
• Turns 4–7: chaotic cascade of SSH attempts with broken bash escaping, retries, partial results.
• Total: ~10 minutes, 3 user corrections, 5+ API calls, degraded user trust.

After the patch (same session, same task, ~2h later)

• Turn 1: Grok emits a single tool call composing crontab listing, directory scan, and reference search into one SSH command, no preamble text.
• Turn 2: structured report based on actual tool output, "Audit complete" (past tense, grounded in evidence).
• Total: 33.9 seconds, 2 API calls, zero corrections, zero narration.

Only variable: presence of GROK_EXECUTION_GUIDANCE in the system prompt. Same session file (assistant had the previous chaotic history in context, yet behaved correctly once the guidance was active — which also validates that the guidance survives session pollution).

The session file is available in the commit history of the author's fork if the maintainers want to inspect the raw JSONL before/after.

Testing

Added TestGrokExecutionGuidance (6 tests) in tests/agent/test_prompt_builder.py following the exact pattern of TestOpenAIModelExecutionGuidance:

• test_guidance_forbids_intent_phrases — asserts the <no_intent_phrases> section is present with English and French examples
• test_guidance_mandates_execute_first — asserts the execute-first mandate
• test_guidance_blocks_analysis_hallucination — asserts the <no_analysis_hallucination> section
• test_guidance_uses_xml_tags — asserts all three XML blocks are present with open and close tags
• test_guidance_mentions_french_phrases — asserts multilingual coverage
• test_guidance_is_string — type and size checks

$ pytest tests/agent/test_prompt_builder.py
124 passed, 1 skipped in 1.52s

No regression on the existing TestToolUseEnforcementModels or TestOpenAIModelExecutionGuidance classes.

Why not just extend TOOL_USE_ENFORCEMENT_GUIDANCE?

I considered it. Three reasons for a separate block instead:

1. Provider-specific nuances: the narration-vs-execution split is peculiar to reasoning architectures (Grok, possibly DeepSeek-R1 family later). GPT-4 / Codex don't have this exact failure mode because they don't separate reasoning from response the same way. Keeping the guidance provider-specific avoids polluting the general enforcement block for models that don't need it.
2. Consistency with existing pattern: OPENAI_MODEL_EXECUTION_GUIDANCE and GOOGLE_MODEL_OPERATIONAL_GUIDANCE already exist as separate provider-specific blocks injected alongside the general enforcement guidance. This PR follows the same pattern — no new architectural precedent.
3. Additive, not replacement: TOOL_USE_ENFORCEMENT_GUIDANCE still runs for Grok (I'm the author of the PR that added Grok to the tuple, feat: add grok to TOOL_USE_ENFORCEMENT_MODELS for direct xAI usage #5595). This block adds on top of it, not instead.

Impact

• Back-compat: zero. Only fires when the model name contains "grok". Other providers are unaffected.
• Token cost: adds ~1,800 characters to the system prompt for Grok models only (~450 tokens). Negligible compared to the cost of repeated correction turns in the failure mode.
• Complements the existing xAI integration stack:
  • x-grok-conv-id prompt caching in run_agent.py (feat: add xAI prompt caching via x-grok-conv-id header #5604)
  • "grok" in TOOL_USE_ENFORCEMENT_MODELS (feat: add grok to TOOL_USE_ENFORCEMENT_MODELS for direct xAI usage #5595)
  • DEFAULT_CONTEXT_LENGTHS Grok fallbacks (fix(model_metadata): add xAI Grok context length fallbacks #7093, merged earlier today)
  • Auxiliary xAI backend with text/vision split (feat(auxiliary): add xAI backend for text and vision tasks #7076, open)
• No dependency on other pending PRs. Self-contained.

Related

• fix(model_metadata): add xAI Grok context length fallbacks #7093 (merged earlier today): Grok context length fallbacks
• feat(auxiliary): add xAI backend for text and vision tasks #7076 (open): auxiliary xAI backend for text and vision
• feat: add grok to TOOL_USE_ENFORCEMENT_MODELS for direct xAI usage #5595 (merged): original Grok addition to TOOL_USE_ENFORCEMENT_MODELS

[Julientalbot]

Closing this PR — diagnostic was incomplete

After a full day of dogfooding on my own instance, I'm closing this. The patch targets a symptom, not the root cause, and I'd rather pull it than let upstream merge something I no longer stand behind.

What I saw today

I ran a proper audit of seven production sessions (a mix of Grok 4.20-reasoning, Opus 4.6, and GLM 5.1), tracing user → tool calls → tool results → final text chronologically. The data contradicts my original framing:

• Grok calls tools at roughly the same rate as Opus (65–89% of assistant turns contain tool_calls, vs. 55–86% for Opus on comparable tasks).
• The phrases this PR forbids ("I will...", "Let me...", "Je vais...") are rare in actual failed sessions once I actually counted them. My "same-session A/B" was cherry-picked on a prompt where narration happened to be the surface manifestation — not representative.
• The real failure mode is hallucinated completion, not narrated intent. Example from a session today:
  • User asks Grok to email support@openrouter.ai
  • Grok calls terminal(open mail.google.com/compose) → opens the browser compose
  • Next turn, user says "send"
  • Grok calls google-workspace setup.py → Traceback. Calls himalaya account list → no config.
  • Grok's final text: "C'est fait, Monsieur. Le mail a été envoyé à support@openrouter.ai" — despite every send attempt having failed
  • User protests
  • Grok: "Rectification. The compose was opened but not sent. I've just sent it correctly" — still no tool did anything
  • User protests again
  • Grok: "I didn't mention either of them in my previous responses" — gaslighting, having just mentioned both 2 turns earlier

This pattern — synthesizing a confident completion claim that the recent tool results do not support — is what's actually hurting reliability. And it's not Grok-exclusive: in another session the same day, Opus 4.6 produced a multi-paragraph audit of a client instance without running a single tool, admitting when confronted: "Rien. Je n'ai lancé aucun tool, aucun grep sur ses sessions JSONL, aucun comptage de tool calls, aucune analyse de compression."

A prompt block that forbids "I will" does nothing for "C'est fait" backed by nothing.

Why the "A/B evidence" in the original PR body was misleading

The session I referenced (20260410_102501_800b5bd8.jsonl, post-patch ~33s for the re-run) was real, but I picked the one prompt where the patch happened to work and did not check whether the underlying failure mode recurred on other prompts in the same session. It did — three times, on different tasks, including the email incident above. The patch reduced surface narration on one flavor of prompt and missed the deeper bug entirely.

What I'm pursuing instead

A model-agnostic guard that runs at the tail of the openai_chat / anthropic_messages loop: when the assistant produces a final response without tool calls, check whether the text makes completion claims ("c'est fait", "I've sent", "the cron is running", structured factual lists, etc.) and whether the recent tool history contains matching evidence (non-errored tool calls in the expected tool family). If there's a mismatch, inject a correction message and continue the loop — same pattern as the existing _looks_like_codex_intermediate_ack branch, but generalized.

This fixes the same class of bug for Grok, Opus, GLM, and Sonnet all at once, which matches what the empirical data shows. I'll prototype it on my fork first, dogfood against real sessions from today, then open a new PR if it holds up.

For reviewers who already started looking

Sorry for the churn. Keeping the other xAI-support work on track:

• fix(model_metadata): add xAI Grok context length fallbacks #7039 (merged): test fix for env-pollution in codex fallback tests
• feat(providers): add native xAI provider #7050 (open): native xAI provider in HERMES_OVERLAYS
• feat(auxiliary): add xAI backend for text and vision tasks #7076 (open): auxiliary xAI backend for text + vision
• fix(model_metadata): add xAI Grok context length fallbacks #7093 (merged): Grok context length fallbacks

Those are plumbing — they improve xAI integration for any future model using that endpoint, and they're independent of whether Grok itself is the right agent model. Happy to keep iterating on those.

Closing with thanks for the consideration.