fix(tools): allow zero-width joiner inside emoji ZWJ sequences

[oscarello]

Problem

The invisible-unicode scanner (tools/threat_patterns.py) flags any U+200D (zero-width joiner) via set intersection, with no context awareness. But U+200D is a required, legitimate part of standard emoji ZWJ sequences — surfer 🏄♂️, family 👨‍👩‍👧‍👦, heart-on-fire ❤️🔥 — which are structured as [emoji][U+FE0F]?[U+200D][emoji][U+FE0F]?....

Real-world impact: a SOUL.md persona containing a single 🏄♂️ emoji is blocked wholesale as invisible_unicode_U+200D and never reaches the system prompt — the entire identity file silently fails to load. The same false positive blocks any SKILL.md (via tools/skills_guard.py, which has the same duplicated check) that contains such an emoji.

Fix

Allow U+200D when every occurrence sits inside an emoji ZWJ sequence (adjacent to an emoji-ish codepoint or a variation selector). Bare ZWJs glued between plain characters — the classic text-hiding trick — are still flagged.

• tools/threat_patterns.py: shared helper is_zwj_in_emoji_sequence() + emoji-neighbor range check; scan skips U+200D only when all occurrences are emoji-context.
• tools/skills_guard.py: same rule in the per-line invisible-char check (imports the shared helper so the two scanners can't drift).
• The exception is scoped to U+200D only — all other invisible/bidi codepoints are flagged exactly as before.

Tests

• test_threat_patterns.py: emoji ZWJ allowed (surfer + family sequences), bare ZWJ between ASCII still detected, mixed emoji+bare still detected, zero-width space next to an emoji still detected.
• test_skills_guard.py: SKILL.md with emoji ZWJ is clean; SKILL.md with a bare ZWJ still yields invisible_unicode.

All 60 tests in the two affected files pass; ruff check clean.

[teknium1]

Thanks for addressing a real false positive: current main 26e0b1c flags U+200D unconditionally in tools/threat_patterns.py:234-237 and tools/skills_guard.py:616-630.

Problems

• tools/threat_patterns.py:200-203 accepts a ZWJ if either immediate neighbor is emoji-ish or a variation selector. That lets A\u200d🏄 and 🏄\u200dA bypass the invisible-Unicode guard even though neither has an emoji base on both sides. The same predicate is used by tools/skills_guard.py:623-630.

Suggested changes

• Require emoji bases on both sides after skipping optional VS15/VS16. tools/cronjob_tools.py:145-157 already uses that two-sided shape for its existing emoji-ZWJ exception.
• Add one-sided-neighbor and variation-selector-only regression cases to both affected test files.

Automated hermes-sweeper review.

[alt-glitch]

This was generated by AI during triage.

Duplicate of #59503: both implement the same emoji-context U+200D false-positive repair for the threat scanner. This patch also updates the paired skills scanner, but the shared mechanism belongs with the existing open canonical fix.

[oscarello]

Thanks for the review @teknium1 — the either-side predicate was a real hole. Fixed in e7b33ac8f:

• Two-sided check. is_zwj_in_emoji_sequence() now mirrors tools/cronjob_tools.py exactly: skip optional VS16 (U+FE0F) on either side, then require an emoji base on both sides. A\u200d🏄 and 🏄\u200dA are flagged again (regression tests added in both test_threat_patterns.py and test_skills_guard.py).
• Dedupe. Since cronjob_tools.py already had this two-sided shape, I consolidated: it now imports is_zwj_in_emoji_sequence from tools/threat_patterns.py instead of keeping its own copy — one implementation for the cron tripwire, the context-file scanner, and the skills install scanner. 503 tests green, ruff clean.

On the duplicate note from triage (re: #59503): this PR covers the sibling path too — tools/skills_guard.py had the same unconditional U+200D flag (SKILL.md files with emoji were blocked on install), plus tests for both scanners, whereas #59503 touches only threat_patterns.py with no test coverage. Happy to consolidate either direction — fold the skills_guard half into #59503, or keep this one as the broader fix — whatever the maintainers prefer.

CI: the workflow run is parked on the first-time-contributor approval gate (action_required) — needs a maintainer to approve it at https://github.com/NousResearch/hermes-agent/actions/runs/30753528924 before it executes.

[GottZ]

This was generated by AI during triage.

Summary

Twenty-two PRs address or reference this scanner complex: the main clusters are emoji-aware U+200D handling, user-visible context-block warnings, Mythic/C2 false positives, exfiltration-pattern false positives, and adjacent scanner hardening. The diffs range from narrow one-line exemptions to shared two-sided emoji validation across threat, skill, and cron scanners; the notification and pattern-policy changes remain separable from the ZWJ fix.

Related pull requests

• fix: allow ZWJ inside emoji grapheme clusters in context/memory/skills scanners #12673 [closed] best fix — (+137/-19) — closed precursor/reference: introduces a shared emoji-aware invisible-character helper and applies it to the threat and skill scanners, with prompt-builder regressions preserving detection of non-emoji ZWJ. It remains relevant because Verify recorded it as a best existing fix for 🤸‍♀️ SOUL.md blocked by ZWJ emoji — cartwheel gymnast triggers prompt injection filter #18581 and False positive: ZWJ emoji (U+200D) in context files (SOUL.md/AGENTS.md) is flagged as injection and drops the whole file #59492 and it established the broader cross-scanner design later pursued by fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857.
• fix: remove ZWJ from invisible char blocklist to allow compound emoji in context files #18644 [closed] fixes — (+1/-1) — closed duplicate of fix(prompt): allow ZWJ emoji in context files #18616: removes U+200D from only the prompt-builder blocklist, allowing all ZWJ rather than distinguishing emoji sequences from text-hiding uses. It remains relevant as the minimal but security-weaker variant and was explicitly closed in favor of fix(prompt): allow ZWJ emoji in context files #18616, with overlap noted against fix: allow ZWJ inside emoji grapheme clusters in context/memory/skills scanners #12673.
• fix: allow ZWJ inside emoji grapheme clusters in context/memory/cron scanners #24339 [closed] partial — (+246/-24) — closed duplicate of fix: allow ZWJ inside emoji grapheme clusters in context/memory/skills scanners #12673: adds a shared Unicode scanner and updates context, memory, and cron paths, but uses broad category/range heuristics and includes unrelated scanner aliases. It remains relevant as evidence for cross-scanner consolidation, while its author explicitly closed it as duplicating fix: allow ZWJ inside emoji grapheme clusters in context/memory/skills scanners #12673.
• fix: allow ZWJ (U+200D) in legitimate emoji sequences #35808 [closed] best fix — (+180/-2) — closed reference implementation: changes the shared threat scanner to strip only ZWJs with emoji bases on both sides while retaining ASCII, CJK, mixed-content, and edge detection, with focused tests. Verify recorded it as a best existing fix for 🤸‍♀️ SOUL.md blocked by ZWJ emoji — cartwheel gymnast triggers prompt injection filter #18581, and its two approved reviews support the core algorithm even though the PR is closed and does not cover the skill scanner.
• fix(security): normalize context-file content before injection scanning #41594 related — (+120/-5) — keep open with a salvage path: preserve the raw-plus-normalized matching concept for punctuation and compatibility-character splice evasions, but port it onto the current 64 KiB cap and bounded patterns and add a prompt-builder blocking regression. This follows the maintainer-bot keep_open review, which cites current tools/threat_patterns.py:245-253 and commit 060779b.
• feat(security): vendor-anchored secret-format patterns for strict scope #43302 related — (+71/-1) — author action: split or extend the vendor-format secret detector so the intended skill-install path uses the same checks as strict memory scanning. This follows the contributor keep_open review: the diff changes tools/threat_patterns.py, while skill installation uses the independent tools/skills_guard.py table and therefore does not receive the claimed coverage.
• fix(tools): require C2 context for 'mythic' in threat pattern scanner #44638 best fix — (+24/-2) — keep open with a salvage path: retain removal of the bare Mythic match and the creative-persona regression, but replace generic qualifiers such as “mythic agent” and “mythic server” with independently C2-specific context and add negative tests. Verify identified fix(tools): require C2 context for 'mythic' in threat pattern scanner #44638 as the best existing fix for [Bug] SOUL.md false positive: "known_c2_framework" blocks legitimate persona file #44631, and this directly addresses the maintainer-bot keep_open review rather than bypassing its blocking concern.
• fix: narrow Mythic threat pattern context #44665 [closed] partial — (+18/-2) — closed duplicate of fix(tools): require C2 context for 'mythic' in threat pattern scanner #44638: makes the same bare-Mythic-to-qualified-Mythic change in the same scanner and adds a persona regression. It remains relevant as corroborating implementation evidence, but contributor discussion explicitly consolidated it into the earlier fix(tools): require C2 context for 'mythic' in threat pattern scanner #44638.
• feat: support configured external context files #48809 related — (+243/-14) — author action: rebase onto the current context schema and split out the external-file loading feature after restoring underscore-delimited hidden-comment detection and complete .cursorrules de-duplication. This follows the contributor keep_open review, which also requires preserving the existing context.engine setting.
• fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503 duplicate — (+60/-5) — author action: rebase onto main, or split out the part that can merge after replacing the immediate-neighbour heuristic with two-sided emoji-base validation that skips extensions and adding direct scanner plus prompt-builder tests. Despite its keep_open review, the diff currently accepts malformed selector-adjacent input and is narrower than fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 because it does not repair the skill scanner.
• fix(agent): surface threat-scanner block warnings to users #59622 best fix — (+15/-0) — keep open with a salvage path: retain the focused warning-pipeline hook and add an integration regression proving that a blocked startup context reaches agent._emit_status(), not merely drain_truncation_warnings(). Verify recorded fix(agent): surface threat-scanner block warnings to users #59622 as a best existing fix for AGENTS.md Silent Threat-Scanner Block — No User Notification #59612, and the requested test directly follows its high-salvageability keep_open review.
• fix(agent): surface blocked context file warnings #59625 fixes — (+26/-3) — close as duplicate of fix(agent): surface threat-scanner block warnings to users #59622: both diffs add the same _record_truncation_warning() call at the same blocking boundary for AGENTS.md Silent Threat-Scanner Block — No User Notification #59612. Despite the keep_open review on fix(agent): surface blocked context file warnings #59625, its requested _emit_status() regression is equally applicable to the earlier canonical fix(agent): surface threat-scanner block warnings to users #59622 and the diff supplies no distinct production behavior.
• fix(prompt-builder): surface threat-scanner context blocks to the user #59652 best fix — (+110/-3) — author action: split the notification change toward fix(agent): surface threat-scanner block warnings to users #59622 and salvage the Mythic policy work through fix(tools): require C2 context for 'mythic' in threat pattern scanner #44638 with a C2-specific positive signature. Verify recorded fix(prompt-builder): surface threat-scanner context blocks to the user #59652 as a best existing fix for AGENTS.md Silent Threat-Scanner Block — No User Notification #59612, but its contributor keep_open review identifies three diff-level blockers: “Mythic C2” is no longer detected, the warning advertises a nonexistent exception mechanism, and tests stop before _emit_status().
• fix(agent): preserve ZWJ (U+200D) in context files (#59492) #59668 fixes — (+422/-1) — close as duplicate of fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 after preserving any useful negative-range cases: both make the shared threat scanner exempt only emoji-context U+200D, while fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 also covers skills and consolidates cron behavior. Despite the keep_open review on fix(agent): preserve ZWJ (U+200D) in context files (#59492) #59668, its diff accepts entire U+2300–U+23FF and U+2B00–U+2BFF blocks as emoji neighbours, including documented non-emoji members, whereas fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 already has the stronger two-sided structure to refine.
• fix(agent): allow ZWJ emoji sequences in threat scanner (#59492) #59701 [closed] fixes — (+92/-3) — closed duplicate of fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503: adds positional ZWJ handling and mixed-content tests, but its unicodedata category heuristic treats broad Symbol characters and variation selectors as pictographic neighbours. It remains relevant for its focused regression cases; contributor discussion and the closing comment explicitly identify fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503 as the earlier equivalent.
• fix(agent): notify user when AGENTS.md/context files are blocked by threat scanner (#59612) #59708 [closed] fixes — (+11/-0) — closed duplicate of fix(agent): surface threat-scanner block warnings to users #59622: routes blocked context findings through the existing warning accumulator but adds no tests. It remains relevant as a minimal confirmation of the notification mechanism, while contributor triage and the closing comment explicitly consolidate it into fix(agent): surface threat-scanner block warnings to users #59622.
• fix(threat_patterns): stop flagging legitimate ZWJ emoji as invisible-unicode injection #59710 [closed] fixes — (+237/-3) — closed duplicate of fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503: adds an emoji-neighbour ZWJ scan and standalone tests, but uses broad Unicode blocks and an ad hoc test runner rather than the project suite. It remains relevant as another reproduction of False positive: ZWJ emoji (U+200D) in context files (SOUL.md/AGENTS.md) is flagged as injection and drops the whole file #59492, while contributor triage explicitly identifies the same code site and mechanism as fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503.
• fix(agent): surface blocked AGENTS.md warning to user and remove mythic false positive #59918 [closed] fixes — (+6/-1) — closed duplicate of fix(prompt-builder): surface threat-scanner context blocks to the user #59652: combines the same warning-accumulator hook with unconditional removal of bare Mythic and supplies no regression tests. It remains relevant as a minimal expression of those two changes, but contributor triage and the closing comment explicitly consolidate it into the earlier, tested fix(prompt-builder): surface threat-scanner context blocks to the user #59652.
• fix(threat-scanner): exempt ZWJ emoji sequences from invisible unicode flag (#59492) #59925 [closed] fixes — (+62/-0) — closed duplicate of fix(security): skip ZWJ in legitimate emoji sequences during injection scan #59503: exempts ZWJ through a broad “emoji-like” heuristic and also carries unrelated Tencent model changes. It remains relevant mainly as negative evidence for isolation and conservative membership checks; contributor triage explicitly marks the intended fix as duplicate and calls out the unrelated branch content.
• fix(tools): reduce false positives in exfil_curl/exfil_wget patterns #63994 partial — (+40/-2) — author action: rebase onto main and split out a policy-complete exfiltration fix with tests using the actual $TRILLIUM_ETAPI_URL reproduction and explicit coverage for $CREDENTIAL/$API. This follows its keep_open review: the current diff keeps scope="all", drops existing keyword coverage, and its purported regression inputs contain no triggering environment variable.
• fix(tools): narrow exfil_curl/exfil_wget scope and add \b boundary #64053 [closed] best fix — (+79/-4) — closed competing reference: moves curl/wget secret patterns from all/context scans to strict scope and adds a trailing boundary, directly addressing the whole-file false positive in exfil_curl context-scan pattern blocks legitimate API recipes in SOUL.md — whole identity file silently replaced, agent runs on stock persona #63977. Verify recorded it as the best existing fix for exfil_curl context-scan pattern blocks legitimate API recipes in SOUL.md — whole identity file silently replaced, agent runs on stock persona #63977, so it remains the root-cause reference despite closure; its contributor review still requires prompt-builder integration coverage and an explicit decision about malicious curl content in context files.
• fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 best fix — (+164/-43) — keep open with a salvage path: retain the shared two-sided ZWJ helper, threat-scanner and skill-scanner regressions, and cron de-duplication, then tighten emoji-base membership and skip both VS15 and VS16 before validating each side. Verify identified fix(tools): allow zero-width joiner inside emoji ZWJ sequences #76857 as the best existing fix for False positive: ZWJ emoji (U+200D) in context files (SOUL.md/AGENTS.md) is flagged as injection and drops the whole file #59492; the updated diff addresses the visible keep_open review’s one-sided-neighbour blocker, but the broad U+2300–U+23FF ranges still need conservative validation.

Duplicates

ZWJ chain: #18644 duplicates #18616 and overlaps #12673; #24339 duplicates #12673; #59701, #59710, and #59925 were closed as duplicates of #59503; #59668 and #59503 implement the same threat-scanner mechanism now covered more broadly by #76857; #35808 is a focused closed reference for that algorithm. Notification chain: #59625 and closed #59708 duplicate #59622; closed #59918 duplicates #59652, whose notification half overlaps #59622. Mythic chain: closed #44665 duplicates #44638, while the Mythic half of #59652/#59918 overlaps that fix. Exfiltration chain: #63994 and closed #64053 compete on regex boundary versus scope policy rather than being byte-for-byte duplicates.

Suggested consolidation

Keep #76857 open with a salvage path: use it as the current ZWJ consolidation point because it covers both tools/threat_patterns.py and tools/skills_guard.py, preserves the cron implementation through a shared helper, and is the recorded best existing fix for #59492. Before acceptance, require conservative emoji-base membership on both sides after skipping VS15/VS16 and retain one-sided, malformed-selector, mixed-content, and non-emoji-block regressions; this explicitly follows the contributor and maintainer-bot keep_open reviews. Close open #59503 and #59668 as duplicates of #76857 only after carrying over any stronger tests; despite their keep_open reviews, their diffs have concrete malformed-selector or overbroad-range holes and cover less of the affected scanner surface. Separately, keep #41594, #44638, and #59622 open on their documented salvage paths; author action is needed on #43302, #48809, #59652, and #63994 to split or rebase their independently useful parts. Close #59625 as duplicate of #59622, while retaining closed #35808, #44665, #59701, #59708, #59710, #59918, #59925, and #64053 only as reference implementations or duplicate-chain evidence.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I59492(["issue #59492 (open)"])
    subgraph Dup12673 ["PRs duplicating each other"]
        P12673["PR #12673 (closed)"]
        P18644["PR #18644 (closed)"]
        P24339["PR #24339 (closed)"]
        P35808["PR #35808 (closed)"]
        P59503["PR #59503 (open)"]
        P59668["PR #59668 (open)"]
        P59701["PR #59701 (closed)"]
        P59710["PR #59710 (closed)"]
        P59925["PR #59925 (closed)"]
        P76857["PR #76857 (open)"]
    end
    P76857 -->|best fix| I59492
    class I59492 open
    class P12673 closed
    class P18644 closed
    class P24339 closed
    class P35808 closed
    class P59503 open
    class P59668 open
    class P59701 closed
    class P59710 closed
    class P59925 closed
    class P76857 open
    class P12673 best
    class P12673 best
    class P35808 best
    class P76857 best
    class P76857 target
    click I59492 "https://github.com/NousResearch/hermes-agent/issues/59492"
    click P12673 "https://github.com/NousResearch/hermes-agent/pull/12673"
    click P18644 "https://github.com/NousResearch/hermes-agent/pull/18644"
    click P24339 "https://github.com/NousResearch/hermes-agent/pull/24339"
    click P35808 "https://github.com/NousResearch/hermes-agent/pull/35808"
    click P59503 "https://github.com/NousResearch/hermes-agent/pull/59503"
    click P59668 "https://github.com/NousResearch/hermes-agent/pull/59668"
    click P59701 "https://github.com/NousResearch/hermes-agent/pull/59701"
    click P59710 "https://github.com/NousResearch/hermes-agent/pull/59710"
    click P59925 "https://github.com/NousResearch/hermes-agent/pull/59925"
    click P76857 "https://github.com/NousResearch/hermes-agent/pull/76857"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 22 pull requests and 5 issues in this complex. Each diff was read against this issue; Assessment working set: 151 kB of PR diffs, 71 kB of issue/PR text, 38 kB of discussion (60 comments), 52 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[alt-glitch]

This was generated by AI during triage.

Related to #59503, not a duplicate: this patch extends the same two-sided emoji-ZWJ rule to skills_guard and centralizes the helper across the threat and cron scanners.

[oscarello]

Quick nudge for the maintainers: the workflow runs on this PR (CI + both label reruns) are all parked at the first-time-contributor approval gate (action_required). The two-sided ZWJ fix from @teknium1's review is in and tests pass locally (503 green, ruff clean). Could a maintainer approve the workflow run so CI can go green? Thanks!

[oscarello]

Friendly follow-up: the CI run on this PR (pushed Aug 5, 22f0db934) has been parked at the first-time-contributor approval gate for ~2 weeks now, along with the label-rerun runs. Could a maintainer approve the workflow run so checks can actually execute? Local validation is green (509 tests, ruff clean) — just need the gate cleared.

@teknium1 — flagging for a re-review when you have a moment: your either-side predicate point was addressed in e7b33ac8f (two-sided emoji-base check + dedupe), and 22f0db934 additionally skips VS15/VS16 variation selectors on either side of the ZWJ. Happy to adjust if anything still looks off. Thanks!