Skip to content

fix(agent): call scan_for_threats on untrusted tool results to log promptware hits - #32793

Closed
sprmn24 wants to merge 1 commit into
NousResearch:mainfrom
sprmn24:fix/threat-scan-tool-result-eksik
Closed

fix(agent): call scan_for_threats on untrusted tool results to log promptware hits#32793
sprmn24 wants to merge 1 commit into
NousResearch:mainfrom
sprmn24:fix/threat-scan-tool-result-eksik

Conversation

@sprmn24

@sprmn24 sprmn24 commented May 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

tools/threat_patterns.py was introduced with documentation stating it is
"shared with the tool-result delimiter system", but tool_dispatch_helpers.py
never imported or called scan_for_threats(). As a result, tool results
from web_extract, web_search, and browser_* were wrapped in
<untrusted_tool_result> delimiters but never scanned — known C2 heartbeat,
role-hijack, and promptware patterns passed through silently with no log entry.

This PR wires the missing call: after confirming the content is long enough to
wrap, scan_for_threats(content, scope="context") is called. Any matches
are logged at WARNING level so operators can detect injection attempts in
production logs. The import is guarded with a try/except ImportError so
the module remains functional even if tools.threat_patterns is unavailable.

Note: logging does not block the request — the <untrusted_tool_result>
wrapper is the primary architectural defence; threat scanning adds observability.

Related Issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Security fix
  • Documentation update
  • Tests (adding or improving test coverage)
  • Refactor (no behavior change)

Changes Made

  • agent/tool_dispatch_helpers.py: added guarded import of
    tools.threat_patterns.scan_for_threats; in _maybe_wrap_untrusted()
    call _scan_threats(content, scope="context") and log a WARNING when
    threat patterns are detected; updated docstring.

How to Test

  1. Pass content containing a known threat pattern (e.g. ignore previous instructions)
    as the result of a web_extract tool call
  2. Verify a WARNING log line appears: Untrusted tool 'web_extract' returned content matching threat patterns: [...]
  3. Verify the content is still wrapped in <untrusted_tool_result> as before
  4. Run: pytest tests/ -q

Checklist

Code

Documentation & Housekeeping

  • I've updated the docstring to document the new threat-scan behaviour
  • I've considered cross-platform impact — or N/A

Screenshots / Logs

@alt-glitch alt-glitch added type/security Security vulnerability or hardening P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 26, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for identifying the observability gap. This is already implemented on current main by a later, broader security change.

  • Automated hermes-sweeper review verified that agent/tool_dispatch_helpers.py:439 scans textual output from every untrusted source, including multimodal text parts, via scan_for_threats(..., scope="context") at agent/tool_dispatch_helpers.py:465.
  • agent/tool_executor.py:979 and agent/tool_executor.py:1658 propagate non-low-risk findings as tool.output_risk events across concurrent and sequential execution.
  • Commit b9b463f3bd6517b76687d9b3c9dea1e62f01f9e1 (feat(security): expose deterministic tool output risk, feat(security): expose deterministic tool output risk #61793) added this behavior and its coverage in tests/agent/test_tool_dispatch_helpers.py:288.

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 13, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/security Security vulnerability or hardening

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants