fix(security): normalize context-file content before injection scanning

[petrichor-op]

What does this PR do?

The context-file scanner in agent/prompt_builder.py (_scan_context_content)
is the only barrier that blocks, rather than merely warns, before a
discovered context file (AGENTS.md, CLAUDE.md, .cursorrules,
.hermes.md/HERMES.md, SOUL.md) is injected verbatim into the trusted
system-prompt channel. It relies entirely on
scan_for_threats(scope="context") from tools/threat_patterns.py, and the
user has no opportunity to intervene at that layer.

The threat patterns use a (?:\w+\s+)* filler segment between anchor tokens
to defeat word-insertion bypasses, but that segment only spans word
characters and whitespace. As a result the anchors are trivially defeated by
splicing non-word characters between the tokens: punctuation
("ignore, all previous instructions"), hyphens ("ignore all-prior
instructions"), markup, or zero-width characters — none of which \w
matches and, in the zero-width case, \s does not match either. A poisoned
context file in a cloned repository (auto-loaded merely by changing into the
directory) could therefore plant durable instructions in the system prompt
while passing the scan.

This change normalizes the content before matching. scan_for_threats now
evaluates every pattern against both the raw content and a normalized copy
(NFKC, with runs of non-word/non-space characters collapsed to a single
space) and unions the results. The raw pass preserves patterns that
legitimately depend on punctuation (curl $VAR exfiltration, <!-- -->
comments, .env reads); the normalized pass restores the canonical token
sequence so the anchors fire on spliced payloads. Invisible-unicode
detection continues to run on the raw content so the offending codepoint is
still surfaced.

Related Issue

N/A

Type of Change

• 🔒 Security fix

Changes Made

• tools/threat_patterns.py: added _normalize_for_matching() (NFKC plus
  collapsing non-word/non-space runs to a single space) and updated
  scan_for_threats() to match each pattern against both the raw and the
  normalized content, deduplicating findings. Documented the rationale in
  the module docstring.
• tests/tools/test_threat_patterns.py: added a TestSpliceEvasion class
  covering comma, hyphen, punctuation, zero-width, and NFKC full-width
  splices, plus regressions confirming punctuation-dependent patterns still
  match and that normalization introduces no new false positives.

How to Test

1. Before the change, confirm the bypass:
   python -c "from tools.threat_patterns import scan_for_threats as s; print(s('ignore, all previous instructions', scope='context'))"
   prints [] (the payload is not detected).
2. After the change, the same command prints ['prompt_injection'], and the
   hyphen and zero-width variants are likewise detected.
3. Run the suite: scripts/run_tests.sh tests/tools/test_threat_patterns.py
   (46 tests pass, including the 9 new splice-evasion cases) and
   scripts/run_tests.sh tests/tools/test_memory_tool.py tests/agent/test_tool_dispatch_helpers.py tests/agent/test_prompt_builder.py
   to confirm the shared callers are unaffected.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run the test suite for the affected areas and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS (Darwin 25.5.0)

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — module docstring updated
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

[liuhao1024]

✅ Verified — Splice-evasion normalization in threat pattern scanner

Reviewed the diff for tools/threat_patterns.py dual-pass matching.

1. Correct dual-pass design: candidates = [content] if normalized == content else [content, normalized] ensures the raw pass preserves punctuation-dependent patterns (curl $VAR exfil, <!-- --> comment injection, cat ~/.env) while the normalized pass catches token-sequence splices. The optimization to skip the second pass when normalization is a no-op avoids unnecessary regex work on clean input.

2. NFKC normalization scope: _normalize_for_matching correctly folds full-width look-alikes (ｉｇｎｏｒｅ → ignore) and collapses non-word/non-space runs to single spaces. The regex r"[^\w\s]+" also collapses zero-width characters that Python's \s doesn't classify as whitespace — confirmed by the test_zero_width_splice_caught test which catches both the invisible char and the injection pattern.

3. Deduplication: The seen set in scan_for_threats prevents double-reporting when both passes match the same pattern ID. Correct — a pattern shouldn't appear twice in findings.

4. False-positive guard: test_no_new_false_positive_from_normalization verifies that benign prose ("You are obligated to comply with the data-retention policy") doesn't match after normalization. test_punctuation_patterns_unaffected confirms existing punctuation-anchored patterns (exfil_curl, html_comment_injection, read_secrets) still work via the raw pass.

No issues found. The implementation is clean, well-tested, and addresses a real evasion vector.

[teknium1]

Thanks for identifying a real scanner-evasion path. The premise remains valid on current main: tools/threat_patterns.py:245-253 applies NFKC but does not convert non-word splice characters to whitespace, while the injection patterns still require \s+ separators at tools/threat_patterns.py:65-68.

Problems

• The patch predates current scanner hardening. Commit 060779bb762a68524e13758b1b8cd08129417803 added the 64KiB scan cap and bounded filler patterns now present at tools/threat_patterns.py:53-59; preserve those protections when salvaging the dual-pass matcher.
• The new tests cover scan_for_threats directly, but not the claimed blocking boundary in agent/prompt_builder.py:61-64. Add a spliced payload regression through _scan_context_content in tests/agent/test_prompt_builder.py.

Suggested changes

• Layer raw-plus-normalized matching onto current tools/threat_patterns.py:229-253, retaining raw invisible-character detection and the current cap.

Automated hermes-sweeper review.