feat(run_agent): runtime intermediate-ack nudge when tool_use_enforcement=required - #22061
Open
Julientalbot wants to merge 1 commit into
Open
Conversation
…ment=required
Currently agent.tool_use_enforcement: required is silently treated as
auto: the value isn't recognised by the parsing block in
_build_system_prompt(), falls through to the default branch, and only
the prompt-level TOOL_USE_ENFORCEMENT_GUIDANCE is injected. There is
no runtime guarantee that the model actually invokes a tool — it can
return a short narrative ack ("I'll check the directory", "Je vais
vérifier le dossier") and the turn ends on a promise.
This change makes "required" a first-class runtime mode:
1. `__init__` initialises `_tool_use_required_runtime: bool = False`.
2. The enforcement parsing block recognises the literal string
"required" (case-insensitive) and sets:
- `_inject = True` (existing prompt-level guidance still applies)
- `self._tool_use_required_runtime = True` (new flag).
3. The post-response no-tool-calls path already runs the codex
intermediate-ack nudge (`_looks_like_codex_intermediate_ack`) but
gated on `api_mode == "codex_responses"`. The gate is extended
to also trigger when `_tool_use_required_runtime` is set, so the
nudge fires for any api_mode/model combination — chat_completions
paths (DeepSeek, OpenRouter grok-4, etc.) included.
The retry uses the existing nudge mechanism (a system-flagged user
message asking the model to execute the tool) and reuses the existing
`codex_ack_continuations < 2` cap. Crucially, the retry does not
toggle `tool_choice="required"` on the API call: that would force a
tool_call on benign turns ("ok", "merci"). Instead we keep
`tool_choice="auto"` and let the prompt nudge guide the model.
Tests (5 new) in TestToolUseEnforcementConfig:
- test_required_injects_guidance_for_any_model
- test_required_sets_runtime_flag
- test_auto_does_not_set_runtime_flag
- test_required_is_case_insensitive
- test_required_runtime_flag_independent_from_api_mode
Pairs naturally with NousResearch#22059 (FR i18n for the same intermediate-ack
detector) and NousResearch#22055 (xAI reasoning.effort transmission).
Contributor
|
Thanks for isolating the intermediate-ack failure mode and reusing the existing bounded continuation mechanism. Problems
Suggested changes
Automated hermes-sweeper review. |
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
agent.tool_use_enforcement: requiredis currently a silent no-op. The string"required"is not recognised by the parsing block in_build_system_prompt:It falls through to the default branch, so users who set
requiredactually getauto. The prompt-levelTOOL_USE_ENFORCEMENT_GUIDANCEis injected (via the model-substring fallback) but there is zero runtime guarantee that the model actually invokes a tool. A short narrative ack like"I'll check the directory."or"Je vais vérifier le dossier."ends the turn on a promise, and the user sees an Intention Without Execution (IWE) — the recurring failure mode reported on grok-4.x and other agentic models.The post-response detector
_looks_like_codex_intermediate_ackalready implements the right shape of fix (append a system nudge, re-prompt) but is gated toapi_mode == "codex_responses". Chat-completions paths (DeepSeek, OpenRouter grok-4, etc.) get nothing.Fix
Make
"required"a first-class runtime mode without any new public API surface:self._tool_use_required_runtime: bool = Falsenext to the existingself._tool_use_enforcementattribute."required"(case-insensitive). Sets_inject = True(the existing prompt-level guidance still applies) andself._tool_use_required_runtime = True(the new flag).Now the existing nudge mechanism fires for any
api_mode/model combo when the user has explicitly opted intorequired.What this PR does NOT do
It deliberately does not flip
tool_choiceto"required"on the API call. Forcing a tool_call on benign turns ("ok","merci", simple acknowledgements that don't need any action) would create absurd outputs. The nudge is prompt-driven: if the model legitimately cannot act, it can say so on the next turn. Thecodex_ack_continuations < 2cap caps retry storms.Tests
5 new cases in
TestToolUseEnforcementConfig:test_required_injects_guidance_for_any_model— the prompt-level guidance still injects for non-default model families.test_required_sets_runtime_flag— the flag is set after_build_system_prompt().test_auto_does_not_set_runtime_flag— default config leaves the flag off (no behavioural drift for users who do not opt in).test_required_is_case_insensitive—REQUIRED,Required,requiredall work.test_required_runtime_flag_independent_from_api_mode— flag set even when the model is on chat_completions (e.g.deepseek/deepseek-v4-pro), proving the gate now covers them.pytest tests/run_agent/test_run_agent.py -v -k ToolUseEnforcement→ 17 passed.Pairing
Third in a small series of PRs improving IWE detection on grok-4.x and adjacent agentic stacks:
reasoning.effortto xAI Responses API (otherwisereasoning_effortis silently dropped on the xAI direct path)._looks_like_codex_intermediate_ackto French phrases and relaxes the global bail-on-prior-tool to a current-turn-only check.tool_use_enforcement: requiredfor any api_mode/model.The three are independent and can land in any order.