Skip to content

feat(tools): validate required params before dispatch - #61550

Open
John-Lussier wants to merge 2 commits into
NousResearch:mainfrom
John-Lussier:feat/required-param-validation
Open

feat(tools): validate required params before dispatch#61550
John-Lussier wants to merge 2 commits into
NousResearch:mainfrom
John-Lussier:feat/required-param-validation

Conversation

@John-Lussier

Copy link
Copy Markdown
Contributor

What

Adds schema-driven required-parameter validation to the tool dispatch pipeline. When a model omits a required parameter from its tool call, the agent now receives an actionable error: "Missing required parameter(s): X. Please retry with all required parameters." instead of a confusing KeyError, silent None default, or partial result deep inside the tool handler.

Why

AIRTBench (Dreadnode, arXiv:2506.14682) found that ~21.7% of all agent execution failures are syntax/format errors — the single largest category. Failed runs cost 10× more than successful runs ($8.91 vs $0.89).

Hermes already has excellent type-level repair (trailing commas, Python None, unclosed braces via _repair_tool_call_arguments; string→int/bool/array coercion via coerce_tool_args; fuzzy tool-name matching via repair_tool_call). But there's no presence-level check — the schema's required array is never consulted before dispatch.

This fills that gap with a 15-line function (validate_required_params) that reads the tool's registered required array and returns any missing param names. It runs after coerce_tool_args and before the tool handler.

Where

model_tools.py:
  validate_required_params()   ← new function (after _coerce_boolean)
  dispatch_tool()              ← calls validate_required_params() right after coerce_tool_args

tests/model_tools/test_validate_required_params.py:  ← new test file (10 tests)

Design notes

  • Zero overhead for unaffected paths: returns [] immediately for tools with no schema, no required array, or unknown tools.
  • Runs after coercion: coerce_tool_args has already fixed type mismatches (string→int, etc.) before this check runs.
  • Key-existence only: a param set to None counts as "present" — this is presence validation, not nullability validation.
  • Error format matches existing patterns: returns {"error": "..."} JSON string, same as other pre-dispatch rejections.
  • Uses isolated ToolRegistry in tests to avoid polluting the global registry.

Verification

python -m pytest tests/model_tools/test_validate_required_params.py -v
# 10 passed

python -m pytest tests/run_agent/test_tool_arg_coercion.py tests/run_agent/test_repair_tool_call_arguments.py -v
# 98 passed (no regressions)

After coerce_tool_args normalizes types, add a lightweight check that
all JSON Schema 'required' parameters are actually present in the
arguments dict before dispatching to the tool handler.

Currently, when a model omits a required param (e.g. calling terminal
without 'command'), the call reaches the handler which either crashes
with a KeyError, silently uses a None default, or produces a confusing
partial result. This surfaces an actionable error to the model instead:
'Missing required parameter(s): X. Please retry with all required
parameters.'

The check is schema-driven (reads the tool's registered 'required'
array from the registry) and runs after coercion so type-mismatched
values are already normalized. Returns early for tools with no schema,
no required array, or unknown tools — zero overhead for unaffected paths.

10 unit tests covering: all-present, one-missing, multi-missing,
no-required-array, empty-required, unknown-tool, non-dict-args,
empty-dict, extra-args, null-value-present.
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jul 9, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting a real dispatch gap: current model_tools.py:1065-1068 has no generic check of schema required fields before registry execution.

Problems

  • The new check at model_tools.py:1103 runs before tool-request middleware. That middleware is explicitly allowed to replace effective arguments before execution (hermes_cli/middleware.py:125-155; current dispatcher applies it at model_tools.py:1146-1162). A middleware that supplies a required field will be rejected too early; later execution middleware can also alter the arguments passed to registry.dispatch().
  • The added tests call only validate_required_params() directly. They do not cover handle_function_call() or either middleware boundary.

Suggested changes

  • Validate the final arguments immediately before registry.dispatch() and add integration tests for request middleware adding a required field and execution middleware removing one.
  • Clarify whether agent-level dispatches are in scope: agent/tool_executor.py:1215-1248 bypasses handle_function_call() even though tools/memory_tool.py:1127 has a root required field.

Automated hermes-sweeper review.

Comment thread model_tools.py Outdated
# coerce_tool_args handles *type* mismatches; this catches genuinely
# *missing* required params that would otherwise surface as confusing
# KeyError or None-default behaviour deep inside the tool handler.
_missing = validate_required_params(function_name, function_args)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this validates before tool_request middleware, but that middleware may replace effective arguments before execution. Validate at the final dispatch boundary instead, so middleware can supply required fields and later execution middleware cannot remove them undetected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e039765. Required-param validation now runs at the final execution boundary immediately before registry dispatch, after tool-request middleware and inside the tool-execution middleware chain. Request middleware can supply a required field; execution middleware removing one now returns the actionable validation error without calling the handler. I also included agent-level tools in scope for both sequential and concurrent paths (including memory) and added integration coverage for all four boundaries. Verification: 155 focused tests passed; git diff --check passed.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 11, 2026
swissly added a commit to swissly/hermes-agent that referenced this pull request Jul 11, 2026
Structured stats collection for the existing tool-call repair pipeline.
Records RepairEvent (pattern, tool, model, timestamp) at each repair
pass in message_sanitization.py and model_tools.py coerce_tool_args.

New module: agent/tool_repair_stats.py
- RepairPattern enum (20 known failure patterns)
- ToolRepairStats singleton: thread-safe, ring-buffer (10k events)
- record_repair() convenience function
- summary() for CLI display

Hooks added (1-2 lines each, zero-overhead when unused):
- message_sanitization.py: 6 hooks in _repair_tool_call_arguments
  (empty_args, none_literal, control_char_escape, trailing_comma,
   unrepairable)
- model_tools.py: 2 hooks in coerce_tool_args
  (bare_string_wrap, bare_object_wrap)

Design constraints:
- No new model tools (zero API cost impact)
- No prompt caching impact
- No new config keys
- Import failure → no-op (never breaks repair pipeline)
- Thread-safe with threading.Lock
- Bounded memory (ring buffer caps at 10k events)

Tests: 19 new tests (stats, thread-safety, ring-buffer, resilience)
Regression: 82 existing repair/coercion tests still pass

Complementary to existing repair PRs (NousResearch#62578, NousResearch#56399, NousResearch#61550, NousResearch#59267,
NousResearch#52747, NousResearch#55620, NousResearch#56557, NousResearch#21696) — adds observability, not repairs.
@John-Lussier

Copy link
Copy Markdown
Contributor Author

Review feedback is addressed in the current head e039765c78 (fix(tools): validate final middleware arguments):

  • Validation now runs on the final effective arguments immediately before registry.dispatch(), after request and execution middleware have run.
  • Added integration tests proving request middleware can add a required param and execution middleware cannot remove one.
  • Extended validation to agent-level sequential and concurrent dispatch paths (agent/tool_executor.py and agent/agent_runtime_helpers.py).

Tests: test_validate_required_params.py expanded; test_model_tools.py updated; compile and git diff --check clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants