Skip to content

fix(agent): coerce tool-result content to string for OpenAI wire format - #31770

Closed
hclsys wants to merge 2 commits into
NousResearch:mainfrom
hclsys:fix/tool-result-content-stringify-31435
Closed

fix(agent): coerce tool-result content to string for OpenAI wire format#31770
hclsys wants to merge 2 commits into
NousResearch:mainfrom
hclsys:fix/tool-result-content-stringify-31435

Conversation

@hclsys

@hclsys hclsys commented May 25, 2026

Copy link
Copy Markdown

Fixes #31435.

Summary

Plugin tool handlers that return Dict[str, Any] were persisted into the chat history with a raw dict in the role: "tool" message content field. The OpenAI Chat Completions spec requires tool message content to be a string. Strict upstreams reject the dict with HTTP 400 (Z.ai 1210, Manifest fallback_exhausted); permissive providers silently coerce, masking the bug until a provider switch.

The fix normalizes content in one place — make_tool_result_message() — through a small defensive helper:

  • str → passes through unchanged
  • multimodal envelope (_multimodal=True + content: list) → preserved as-is, so multipart-capable providers still get the list
  • anything else (plain dict, list, int, …) → json.dumps(content, default=str), falling back to str() if unserializable

This reuses the exact json.dumps(default=str) idiom already present in _multimodal_text_summary rather than inventing a new serializer.

Pre-implement audit

  • Existing-helper check: reused the in-module json.dumps(..., default=str) idiom and the existing _is_multimodal_tool_result predicate; no new serializer introduced.
  • Shared-helper caller check: make_tool_result_message is imported by agent_runtime_helpers.py, tool_executor.py, transports/chat_completions.py, mini_swe_runner.py. The signature is unchanged; the only behavior change is that a non-string, non-multimodal content is now stringified — which is what every OpenAI-spec caller already required. Multimodal dicts (the one legitimate non-string case) are explicitly preserved, so no caller contract is broken.
  • Broader-fix rival scan: no competing PR for tool message content must be string: plugin tools returning dict cause upstream 400 (Z.ai error 1210, OpenAI/Manifest fallback_exhausted) #31435.

Real-behavior proof

$ python3 -c "from agent.tool_dispatch_helpers import make_tool_result_message; \
m=make_tool_result_message('list_workflows', {'definitions':[{'name':'wf1'}],'count':1,'ok':True}, 'call_abc'); \
print(type(m['content']).__name__, '::', m['content'])"
str :: {"definitions": [{"name": "wf1"}], "count": 1, "ok": true}

Before: content was the raw dict → strict upstream returns 400. After: valid JSON string. Multimodal results verified still passed through as a list.

Test plan

python3 -m pytest tests/test_tool_result_content_coercion.py tests/run_agent/test_tool_name_db_persistence.py -q -o 'addopts='
# 7 passed in 1.66s

New tests/test_tool_result_content_coercion.py covers: dict→JSON string, string passthrough, multimodal preserved, bare-list stringified, non-serializable fallback, and that the other message fields (role/name/tool_name/tool_call_id) are unaffected.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 25, 2026

@Tranquil-Flow Tranquil-Flow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix — the approach of normalizing at make_tool_result_message() is exactly the right place. Clean and minimal.

One edge case worth adding: None tool content → empty string. Tool handlers can return None (no output, silent success), and strict OpenAI-compatible providers reject null content the same way they reject dicts. A two-line addition to _coerce_tool_result_content:

if content is None:
    return ""

(before the isinstance(str) check, since None isn't a str)

Also worth a quick test: test_none_content_becomes_empty_string.

Otherwise LGTM.

@hclsys

hclsys commented May 25, 2026

Copy link
Copy Markdown
Author

Good catch — you're right, None was hitting the json.dumps path and producing the literal string "null", which is exactly the kind of misleading content a strict provider would also choke on. Pushed ca40aed36: None -> "" mapped before the other checks, plus the test_none_content_becomes_empty_string test you suggested. Confirmed locally (None -> '', 7 passed).

hclsys added 2 commits May 26, 2026 06:16
Plugin tool handlers returning Dict[str, Any] were persisted into the
chat history with a raw dict as the role:"tool" message content field.
The OpenAI Chat Completions spec requires tool message content to be a
string; strict upstreams reject the dict with HTTP 400 (Z.ai 1210,
Manifest fallback_exhausted), while permissive providers silently coerce
and mask the bug.

make_tool_result_message now normalizes content via a single defensive
helper: strings and multimodal content-part envelopes pass through
unchanged; any other value is JSON-encoded with the json.dumps(default=str)
idiom already used in this module.

Fixes NousResearch#31435
Per review feedback on NousResearch#31770: a handler returning None (silent success)
hit the json.dumps path and produced the literal string "null", which
strict providers reject the same way as a dict. Map None -> "" before the
other checks, with a regression test.
@hclsys
hclsys force-pushed the fix/tool-result-content-stringify-31435 branch from ca40aed to cb837cf Compare May 25, 2026 22:18
@hclsys

hclsys commented May 25, 2026

Copy link
Copy Markdown
Author

Rebased onto current main to resolve the conflict with #32269 (promptware defense), which reworked make_tool_result_message in the same spot.

The two changes are complementary, not redundant — #31435 (non-string tool content) is still open and main still doesn't stringify dict/None tool results. I composed them so both fixes apply: _maybe_wrap_untrusted(name, _coerce_tool_result_content(content)) — coerce first (dict→json.dumps, None""), then your untrusted-content wrapping runs on the resulting string.

While rebasing I caught and fixed an interaction: _coerce_tool_result_content now passes through OpenAI content-part lists ([{"type": "text", ...}]) unchanged instead of JSON-stringifying them, so #32269's multimodal-passthrough contract (test_high_risk_message_with_multimodal_content_unwrapped) stays green. Non-content-part lists (e.g. [1, 2, 3]) still stringify. Added test_content_part_list_passes_through to lock that in.

All of tests/test_tool_result_content_coercion.py + tests/agent/test_tool_dispatch_helpers.py pass (35).

Interstellar-code added a commit to Interstellar-code/hermes-agent that referenced this pull request May 27, 2026
… sessions

Constraint: current main no longer has make_tool_result_message, so the fix had to land on the shared outbound sanitizer and tool-content boundary that still exist.
Rejected: Reapply NousResearch#31770 verbatim | target helper is absent on current main
Rejected: Reapply NousResearch#29920 verbatim | would stringify valid multimodal content-part lists
Confidence: high
Scope-risk: narrow
Directive: Keep role=tool content coercion centralized and preserve wire-valid content-part lists when tightening future sanitizers.
Tested: scripts/run_tests.sh tests/run_agent/test_agent_guardrails.py tests/tools/test_computer_use.py -q
Not-tested: Full suite; live provider round-trip against strict upstreams
@Bartok9

Bartok9 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Re-verified on current origin/main (860cf5133): the bug this PR fixes is still live, and #48244 (filed today, P2, labeled duplicate) is the same root cause — a dict tool result reaches make_tool_result_message() and is emitted as a JSON object in role: "tool" content, which OpenAI-compatible providers reject (GLM 1210, minimax/deepseek HTTP 400) and the fallback chain exhausts.

Runtime proof on main:

>>> make_tool_result_message('some_plugin_tool', {'source':'capability.md','capability':'x'}, 'call_1')['content']
{'source': 'capability.md', 'capability': 'x'}   # dict, not a string → invalid wire format

This PR is the right place to fix it: _coerce_tool_result_content sits at the central chokepoint before _maybe_wrap_untrusted, so it covers every core + plugin tool and composes correctly with the promptware defense — exactly the approach #48244's author independently recommended. It also correctly preserves str and multimodal content-part lists, mirroring the precedent already in bedrock_adapter.py (json.dumps(content)) and codex_event_projector.py.

Cross-ref: the competing #31963 coerces in tool_executor.py at the two dispatch sites instead of the builder — narrower (misses any other caller of make_tool_result_message). Recommend consolidating on this PR (#31770) and closing #48244 as a duplicate of #31435.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused reproduction and multimodal analysis. This is now covered on main, so this PR is redundant.

  • Automated hermes-sweeper review verified f8361d29c8e2a2be6ba9ada32f1d694bf47a4b6a (fix(tools): enforce registry result contract, fix(tools): enforce registry result contract #61787).
  • tools/registry.py:575-603 converts plain dict, None, bytes, and scalar handler returns into a JSON string contract error, while preserving the explicit _multimodal envelope.
  • tools/registry.py:617-623 enforces that result contract at every registry dispatch; plugin tools register through hermes_cli/plugins.py:426-439, and agent calls dispatch through model_tools.py:1273-1278.
  • tests/tools/test_registry.py:66-112 covers unsupported results and the handle_function_call pipeline.

No release tag containing the fix was available locally.

@teknium1 teknium1 closed this Jul 13, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tool message content must be string: plugin tools returning dict cause upstream 400 (Z.ai error 1210, OpenAI/Manifest fallback_exhausted)

5 participants