Skip to content

fix(threat_patterns): stop flagging legitimate ZWJ emoji as invisible-unicode injection - #59710

Closed
Ahmett101 wants to merge 1 commit into
NousResearch:mainfrom
Ahmett101:fix/59492-zwj-emoji-false-positive
Closed

fix(threat_patterns): stop flagging legitimate ZWJ emoji as invisible-unicode injection#59710
Ahmett101 wants to merge 1 commit into
NousResearch:mainfrom
Ahmett101:fix/59492-zwj-emoji-false-positive

Conversation

@Ahmett101

Copy link
Copy Markdown
Contributor

Summary

U+200D (zero-width joiner) was in the INVISIBLE_CHARS frozenset,
so scan_for_threats() flagged every occurrence — including the benign
ZWJ that joins emoji like 🐈‍⬛, 👨‍💻, 👩‍👧 — as invisible_unicode_U+200D
and prompt_builder._scan_context_content() silently dropped the entire
SOUL.md / AGENTS.md / .cursorrules file.

The original motivation for flagging ZWJ was text-hiding: foo\u200dbar
renders as 'foobar' visually but preserves the ZWJ for injection-based
logic splitting. We keep that detection but exempt sequences where ZWJ's
immediate neighbours are both emoji code points.

Changes

  • tools/threat_patterns.py
    • U+200D removed from INVISIBLE_CHARS, kept as module-level _ZWJ
      for the custom scanner.
    • New _is_likely_emoji_codepoint(ch) helper — checks Unicode category
      (So/Sk) and code-point blocks (U+1F000–U+1FFFF, U+2600–U+27BF,
      U+2B00–U+2BFF, U+2300–U+23FF) as a high-confidence fallback for the
      Extended_Pictographic property (not in stdlib unicodedata).
    • scan_for_threats now performs an indexed neighbour-check on U+200D:
      flagged only when both immediate neighbours are NOT both emoji code
      points. Emoji-ZWJ-emoji passes through; "foo\u200dbar" /
      "a\u200d🔥" (mixed with non-emoji) are still blocked.
  • tests/test_threat_patterns_zwj.py — new regression suite, 8/8 PASS
    (standalone runner):
    • CAT ZWJ (🐈‍⬛) not flagged
    • Technologist ZWJ (👨‍💻) not flagged
    • Family ZWJ (👩‍👧) not flagged
    • Text-hiding foo\u200dbar still flagged
    • Mixed non-emoji left + emoji right still flagged
    • Clean text stays clean
    • first_threat_message returns None for 🐈‍⬛
    • _is_likely_emoji_codepoint unit coverage

How to Test

cd ~/.hermes/hermes-agent
python3 tests/test_threat_patterns_zwj.py
# Results: 8/8 passed

Manual repro from the issue body — put 🐈‍⬛ in SOUL.md and start the
agent. Previously: WARNING agent.prompt_builder: Context file SOUL.md blocked: invisible_unicode_U+200D. Now: file loads normally, black cat emoji passes
through, any other invisible chars (U+202E, etc.) are still blocked.

Checklist

  • Tests pass — 8/8 (standalone runner)
  • Follows Conventional Commits (fix(threat_patterns): ...)
  • Changes scoped to this fix only (tools/threat_patterns.py + test)
  • Cross-platform impact: unicodedata + ord + set()& — pure stdlib,
    no signals/subprocess/file I/O. No .env/path/signal changes.
  • profile-safe paths used — no path handling
  • .env not used

Risk & Impact

Minimal. The change removes exactly one character from INVISIBLE_CHARS
and replaces its detection with a neighbour-aware scanner that ignores
emoji-bound ZWJ only. All other invisible characters — U+200B, U+200C,
U+202A–U+202E, directional isolates — remain unchanged. The ZWJ text-hiding
case (foo\u200dbar, \u200dbar) is still caught. No existing scan_for_threats
behaviour changes for any other code path.

Type: Bug fix
Closes #59492

…-unicode injection (NousResearch#59492)

U+200D (zero-width joiner) was in the INVISIBLE_CHARS frozenset, so
scan_for_threats() flagged every occurrence — including the benign
ZWJ that joins emoji like 🐈‍⬛, 👨‍💻, 👩‍👧 — as 'invisible_unicode_U+200D'
and prompt_builder._scan_context_content() silently dropped the entire
SOUL.md / AGENTS.md / .cursorrules file.

The original motivation for flagging ZWJ was text-hiding: a string
like 'foo\u200dbar' renders as 'foobar' visually but preserves the
ZWJ for injection-based logic splitting. We keep that detection but
exempt sequences where ZWJ's immediate neighbours are both emoji code
points — joining two Extended_Pictographic characters together is the
only purpose of a ZWJ in a legitimate file.

Changes in tools/threat_patterns.py:

- `U+200D removed from INVISIBLE_CHARS frozenset`. The constant
  _ZWJ = '\u200d' is a named reference for the scanner.
- New helper `_is_likely_emoji_codepoint(ch)` that checks category
  ('So' / 'Sk') and code-point blocks (U+1F000–U+1FFFF, U+2600–U+27BF,
  U+2B00–U+2BFF, U+2300–U+23FF) — a high-confidence fallback for the
  Unicode Extended_Pictographic property, which stdlib unicodedata
  does not expose.
- `scan_for_threats` now performs a separate indexed ZWJ check: when
  U+200D is present, walk the content and flag it only when both
  immediate neighbours are NOT both emoji code points. Emoji-ZWJ-emoji
  passes through without a finding; 'a\u200db' / 'foo\u200dbar' /
  'a\u200d🔥' (mixed with non-emoji) are still blocked.

Tests: tests/test_threat_patterns_zwj.py — standalone runner, 8/8 PASS:
  cat ZWJ-emoji unblocked, technologist unblocked, family unblocked,
  text-hiding 'foo+ZWJ+bar' still flagged, mixed non-emoji+ZWJ+emoji
  still flagged, clean text stays clean, first_threat_message returns
  None for 🐈‍⬛, and _is_likely_emoji_codepoint unit coverage.

Refs NousResearch#59492
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jul 6, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #59503 — same mechanism (make U+200D emoji-neighbour-aware in scan_for_threats in tools/threat_patterns.py) and same code site, both fixing the ZWJ-emoji context-file false positive (#59492). #59503 was opened first (08:56Z) and is the canonical variant; the extra regression-test file here does not change that. Also related to the broader #12673 (ZWJ across all scanners). Marking duplicate; maintainer picks the canonical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

False positive: ZWJ emoji (U+200D) in context files (SOUL.md/AGENTS.md) is flagged as injection and drops the whole file

2 participants