fix(threat_patterns): stop flagging legitimate ZWJ emoji as invisible-unicode injection - #59710
Closed
Ahmett101 wants to merge 1 commit into
Closed
fix(threat_patterns): stop flagging legitimate ZWJ emoji as invisible-unicode injection#59710Ahmett101 wants to merge 1 commit into
Ahmett101 wants to merge 1 commit into
Conversation
…-unicode injection (NousResearch#59492) U+200D (zero-width joiner) was in the INVISIBLE_CHARS frozenset, so scan_for_threats() flagged every occurrence — including the benign ZWJ that joins emoji like 🐈⬛, 👨💻, 👩👧 — as 'invisible_unicode_U+200D' and prompt_builder._scan_context_content() silently dropped the entire SOUL.md / AGENTS.md / .cursorrules file. The original motivation for flagging ZWJ was text-hiding: a string like 'foo\u200dbar' renders as 'foobar' visually but preserves the ZWJ for injection-based logic splitting. We keep that detection but exempt sequences where ZWJ's immediate neighbours are both emoji code points — joining two Extended_Pictographic characters together is the only purpose of a ZWJ in a legitimate file. Changes in tools/threat_patterns.py: - `U+200D removed from INVISIBLE_CHARS frozenset`. The constant _ZWJ = '\u200d' is a named reference for the scanner. - New helper `_is_likely_emoji_codepoint(ch)` that checks category ('So' / 'Sk') and code-point blocks (U+1F000–U+1FFFF, U+2600–U+27BF, U+2B00–U+2BFF, U+2300–U+23FF) — a high-confidence fallback for the Unicode Extended_Pictographic property, which stdlib unicodedata does not expose. - `scan_for_threats` now performs a separate indexed ZWJ check: when U+200D is present, walk the content and flag it only when both immediate neighbours are NOT both emoji code points. Emoji-ZWJ-emoji passes through without a finding; 'a\u200db' / 'foo\u200dbar' / 'a\u200d🔥' (mixed with non-emoji) are still blocked. Tests: tests/test_threat_patterns_zwj.py — standalone runner, 8/8 PASS: cat ZWJ-emoji unblocked, technologist unblocked, family unblocked, text-hiding 'foo+ZWJ+bar' still flagged, mixed non-emoji+ZWJ+emoji still flagged, clean text stays clean, first_threat_message returns None for 🐈⬛, and _is_likely_emoji_codepoint unit coverage. Refs NousResearch#59492
Collaborator
Duplicate of #59503 — same mechanism (make U+200D emoji-neighbour-aware in |
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
U+200D (zero-width joiner) was in the
INVISIBLE_CHARSfrozenset,so
scan_for_threats()flagged every occurrence — including the benignZWJ that joins emoji like 🐈⬛, 👨💻, 👩👧 — as
invisible_unicode_U+200Dand
prompt_builder._scan_context_content()silently dropped the entireSOUL.md/AGENTS.md/.cursorrulesfile.The original motivation for flagging ZWJ was text-hiding:
foo\u200dbarrenders as 'foobar' visually but preserves the ZWJ for injection-based
logic splitting. We keep that detection but exempt sequences where ZWJ's
immediate neighbours are both emoji code points.
Changes
tools/threat_patterns.pyU+200Dremoved fromINVISIBLE_CHARS, kept as module-level_ZWJfor the custom scanner.
_is_likely_emoji_codepoint(ch)helper — checks Unicode category(
So/Sk) and code-point blocks (U+1F000–U+1FFFF, U+2600–U+27BF,U+2B00–U+2BFF, U+2300–U+23FF) as a high-confidence fallback for the
Extended_Pictographicproperty (not in stdlibunicodedata).scan_for_threatsnow performs an indexed neighbour-check on U+200D:flagged only when both immediate neighbours are NOT both emoji code
points. Emoji-ZWJ-emoji passes through;
"foo\u200dbar"/"a\u200d🔥"(mixed with non-emoji) are still blocked.tests/test_threat_patterns_zwj.py— new regression suite, 8/8 PASS(standalone runner):
foo\u200dbarstill flaggedfirst_threat_messagereturns None for 🐈⬛_is_likely_emoji_codepointunit coverageHow to Test
Manual repro from the issue body — put
🐈⬛inSOUL.mdand start theagent. Previously:
WARNING agent.prompt_builder: Context file SOUL.md blocked: invisible_unicode_U+200D. Now: file loads normally, black cat emoji passesthrough, any other invisible chars (U+202E, etc.) are still blocked.
Checklist
fix(threat_patterns): ...)tools/threat_patterns.py+ test)unicodedata+ord+set()&— pure stdlib,no signals/subprocess/file I/O. No
.env/path/signal changes..envnot usedRisk & Impact
Minimal. The change removes exactly one character from
INVISIBLE_CHARSand replaces its detection with a neighbour-aware scanner that ignores
emoji-bound ZWJ only. All other invisible characters — U+200B, U+200C,
U+202A–U+202E, directional isolates — remain unchanged. The ZWJ text-hiding
case (
foo\u200dbar,\u200dbar) is still caught. No existingscan_for_threatsbehaviour changes for any other code path.
Type: Bug fix
Closes #59492