Skip to content

feat(run_agent): runtime intermediate-ack nudge when tool_use_enforcement=required - #22061

Open
Julientalbot wants to merge 1 commit into
NousResearch:mainfrom
Julientalbot:jt/tool-use-enforcement-required-runtime
Open

feat(run_agent): runtime intermediate-ack nudge when tool_use_enforcement=required#22061
Julientalbot wants to merge 1 commit into
NousResearch:mainfrom
Julientalbot:jt/tool-use-enforcement-required-runtime

Conversation

@Julientalbot

Copy link
Copy Markdown
Contributor

Problem

agent.tool_use_enforcement: required is currently a silent no-op. The string "required" is not recognised by the parsing block in _build_system_prompt:

elif isinstance(_enforce, list):
    ...
else:
    # "auto" or any unrecognised value — use hardcoded defaults
    ...

It falls through to the default branch, so users who set required actually get auto. The prompt-level TOOL_USE_ENFORCEMENT_GUIDANCE is injected (via the model-substring fallback) but there is zero runtime guarantee that the model actually invokes a tool. A short narrative ack like "I'll check the directory." or "Je vais vérifier le dossier." ends the turn on a promise, and the user sees an Intention Without Execution (IWE) — the recurring failure mode reported on grok-4.x and other agentic models.

The post-response detector _looks_like_codex_intermediate_ack already implements the right shape of fix (append a system nudge, re-prompt) but is gated to api_mode == "codex_responses". Chat-completions paths (DeepSeek, OpenRouter grok-4, etc.) get nothing.

Fix

Make "required" a first-class runtime mode without any new public API surface:

  1. Init. Add self._tool_use_required_runtime: bool = False next to the existing self._tool_use_enforcement attribute.
  2. Parsing block. Add a branch that recognises the literal string "required" (case-insensitive). Sets _inject = True (the existing prompt-level guidance still applies) and self._tool_use_required_runtime = True (the new flag).
  3. Post-response gate. The existing intermediate-ack nudge gate is extended:
if (
    (
        self.api_mode == "codex_responses"
        or getattr(self, "_tool_use_required_runtime", False)
    )
    and self.valid_tool_names
    and codex_ack_continuations < 2
    and self._looks_like_codex_intermediate_ack(...)
):
    ...

Now the existing nudge mechanism fires for any api_mode/model combo when the user has explicitly opted into required.

What this PR does NOT do

It deliberately does not flip tool_choice to "required" on the API call. Forcing a tool_call on benign turns ("ok", "merci", simple acknowledgements that don't need any action) would create absurd outputs. The nudge is prompt-driven: if the model legitimately cannot act, it can say so on the next turn. The codex_ack_continuations < 2 cap caps retry storms.

Tests

5 new cases in TestToolUseEnforcementConfig:

  • test_required_injects_guidance_for_any_model — the prompt-level guidance still injects for non-default model families.
  • test_required_sets_runtime_flag — the flag is set after _build_system_prompt().
  • test_auto_does_not_set_runtime_flag — default config leaves the flag off (no behavioural drift for users who do not opt in).
  • test_required_is_case_insensitiveREQUIRED, Required, required all work.
  • test_required_runtime_flag_independent_from_api_mode — flag set even when the model is on chat_completions (e.g. deepseek/deepseek-v4-pro), proving the gate now covers them.

pytest tests/run_agent/test_run_agent.py -v -k ToolUseEnforcement17 passed.

Pairing

Third in a small series of PRs improving IWE detection on grok-4.x and adjacent agentic stacks:

The three are independent and can land in any order.

…ment=required

Currently agent.tool_use_enforcement: required is silently treated as
auto: the value isn't recognised by the parsing block in
_build_system_prompt(), falls through to the default branch, and only
the prompt-level TOOL_USE_ENFORCEMENT_GUIDANCE is injected. There is
no runtime guarantee that the model actually invokes a tool — it can
return a short narrative ack ("I'll check the directory", "Je vais
vérifier le dossier") and the turn ends on a promise.

This change makes "required" a first-class runtime mode:

1. `__init__` initialises `_tool_use_required_runtime: bool = False`.
2. The enforcement parsing block recognises the literal string
   "required" (case-insensitive) and sets:
     - `_inject = True` (existing prompt-level guidance still applies)
     - `self._tool_use_required_runtime = True` (new flag).
3. The post-response no-tool-calls path already runs the codex
   intermediate-ack nudge (`_looks_like_codex_intermediate_ack`) but
   gated on `api_mode == "codex_responses"`.  The gate is extended
   to also trigger when `_tool_use_required_runtime` is set, so the
   nudge fires for any api_mode/model combination — chat_completions
   paths (DeepSeek, OpenRouter grok-4, etc.) included.

The retry uses the existing nudge mechanism (a system-flagged user
message asking the model to execute the tool) and reuses the existing
`codex_ack_continuations < 2` cap. Crucially, the retry does not
toggle `tool_choice="required"` on the API call: that would force a
tool_call on benign turns ("ok", "merci"). Instead we keep
`tool_choice="auto"` and let the prompt nudge guide the model.

Tests (5 new) in TestToolUseEnforcementConfig:
- test_required_injects_guidance_for_any_model
- test_required_sets_runtime_flag
- test_auto_does_not_set_runtime_flag
- test_required_is_case_insensitive
- test_required_runtime_flag_independent_from_api_mode

Pairs naturally with NousResearch#22059 (FR i18n for the same intermediate-ack
detector) and NousResearch#22055 (xAI reasoning.effort transmission).
@alt-glitch alt-glitch added type/feature New feature or request P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 9, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating the intermediate-ack failure mode and reusing the existing bounded continuation mechanism.

Problems

  • Current main moved this runtime behavior out of run_agent.py: initialization is in agent/agent_init.py:1437-1441, and the continuation gate is in agent/conversation_loop.py:5072-5102. The submitted patch therefore needs a substantive port rather than a direct application.
  • Main now exposes the all-api-mode behavior through agent.intent_ack_continuation (hermes_cli/config.py:1028-1037). tool_use_enforcement: required remains an unsupported value in agent/system_prompt.py:264-276, so accepting it as an alias needs an explicit API decision.
  • The added tests assert flag resolution only; they do not run a chat-completions acknowledgement through the loop to prove that the continuation message is emitted.

Suggested changes

  • If maintainers want the alias, implement it in the current continuation-mode resolver and add a chat-completions loop regression alongside the existing intent-ack tests.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants