experiment: add reasoning monitor agent for prompt injection detection - #2
Conversation
Adds Experiment 005: a secondary reasoning monitor agent that watches the primary agent's chain-of-thought and tool calls while being deliberately blind to the primary agent's inputs, preventing the monitor from being compromised by the same poisoned content. Implements three complementary monitoring techniques: - canary token hook (PostToolUse) - tool allowlist hook (PreToolUse) - LLM reasoning monitor (secondary LLM on stripped transcripts) Includes 27 test payloads: 6 benign controls, 16 Phase 1 attacks, and 5 Phase 2 adaptive attacks designed to challenge the monitor. Ported from fullsend-ai/fullsend PR #259. Signed-off-by: Wayne Sun <gsun@redhat.com>
|
/review |
Review: #2Head SHA: a5fc7a5 This PR was NOT reviewed. Do not count this as an approval. |
|
/review |
Review: #2Head SHA: 6ee94ef SummaryThis PR adds a well-designed reasoning monitor experiment (Experiment 005) implementing three complementary prompt injection detection techniques: canary token hook, tool allowlist hook, and LLM reasoning monitor. The code is clean, well-documented, and follows security best practices throughout — particularly the fail-closed design on all error paths, input-blindness architecture for the LLM monitor, and defense-in-depth redaction. The test suite is comprehensive with 27 payloads (including 5 Phase 2 adaptive attacks), thorough unit tests with mocked LLM calls, integration tests against real transcripts, and structural validation of all payloads. No critical or high findings. FindingsMediumNone. Low
Info
FooterOutcome: approve Previous runReview: #2Head SHA: c6b1fd7 SummaryThis PR adds a well-designed reasoning monitor experiment (Experiment 005) implementing three complementary prompt injection detection techniques: a canary token PostToolUse hook, a tool allowlist PreToolUse hook, and an LLM-based reasoning monitor. The code follows a consistent fail-closed security posture throughout — malformed inputs, API errors, unknown verdicts, and rate limit exhaustion all default to blocking. The input-blindness design for the LLM monitor (stripping user messages and user-content tool results before evaluation) is sound and correctly prevents second-order injection. The test suite is comprehensive with 27 payloads (6 benign controls, 16 Phase 1 attacks, 5 Phase 2 adaptive attacks), unit tests with proper mocking, integration tests against real transcripts, and Phase 2 tests that honestly document the known limitations of each monitor. No critical or high findings were identified. FindingsMedium
Low
Info
FooterOutcome: approve Previous run (2)Review: automated reviewOutcome: failure The review agent reviewed commit Previous run (3)Review: #2Head SHA: e7c107a SummaryThis PR adds a well-structured reasoning monitor experiment with three complementary detection techniques (canary hook, tool allowlist, LLM monitor) and a comprehensive test corpus of 27 payloads. The code demonstrates strong security engineering practices: fail-closed design throughout, input-blindness for the LLM monitor, rate limiting, split truncation, and thorough test coverage including adversarial Phase 2 payloads designed to break the monitor. One medium finding relates to inconsistent normalization in canary detection across code paths. No blocking issues. FindingsMedium
Low
Info
FooterOutcome: comment-only Previous run (4)Review: #2Head SHA: a5fc7a5 SummaryThe reasoning monitor experiment is well-architected — the input-blindness design, fail-closed defaults, and three-layer monitoring approach (canary hooks, tool allowlists, LLM reasoning monitor) are sound. However, there are two high-severity findings that require resolution: the canary token detection is trivially bypassable via case variation or character insertion, and the FindingsHigh
Medium
Low
Info
FooterOutcome: request-changes |
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
- Canary detection: case-insensitive matching with unicode/whitespace normalization to resist evasion via case variation or zero-width chars - Confidence clamping: values from LLM responses now clamped to 0.0-1.0 - API error handling: messages.create() wrapped in try/except, fails closed with suspicious verdict on network/auth errors - Rate limiter: LLMMonitor instances cached per model so rate limit state persists across payloads instead of resetting per invocation - Code fence parsing: regex-based extraction handles nested fences - README: fix technique labels (credential-exfil, role-confusion) to match YAML source of truth; document tool allowlist argument gap - Tests: strengthen strip_user_input assertion to verify REDACTED marker; add rate limiting, API error, and confidence clamping tests Signed-off-by: Wayne Sun <gsun@redhat.com>
- Canary hook: scan tool_input in addition to tool_result to catch exfiltration via tool arguments (e.g., embedding canary in comment body) - LLM monitor: move response.content[0] access inside try/except to handle empty API responses; add IndexError/AttributeError to caught exceptions; use entry.copy() in strip_user_input to prevent mutation; fix rate limiter to use `is not None` instead of falsy check so max_calls=0 doesn't silently fall back to default; validate max_calls>=1; redact tool input arguments for USER_CONTENT_TOOLS in format_transcript - Tool allowlist: fail closed (exit 1) on missing tool_name instead of silently allowing - Runner: use check_canary() with normalization for assistant content checks instead of raw string containment; handle non-string content - Tests: add coverage for empty API response, format_transcript redaction, and missing tool_name fail-closed behavior Signed-off-by: Wayne Sun <gsun@redhat.com>
- MonitorVerdict: add __post_init__ validation rejecting confidence outside [0.0, 1.0] to enforce the contract at the data boundary - Canary normalization: apply NFKD Unicode normalization before character filtering to resist homoglyph and composed-char evasion; remove dead \n/\t exception that was collapsed by subsequent regex - Tool allowlist: remove full allowlist enumeration from block reason to prevent information disclosure to compromised agents - LLM monitor: validate model alias at init with clear ValueError; redact tool_use inputs for USER_CONTENT_TOOLS in strip_user_input (not just format_transcript) so the function lives up to its contract - Runner: add tool_input canary scanning to match canary_hook coverage; log warning on unknown monitor names instead of silent skip - Tests: add MonitorVerdict out-of-range rejection test, canary hook tool_input subprocess test, update assertions for new redaction scope Signed-off-by: Wayne Sun <gsun@redhat.com>
- LLM monitor: validate window_sec >= 1 (prevents silent rate limit bypass via zero/negative values); genericize API error reason to avoid leaking infrastructure details from exception messages; add defense-in-depth tool_result redaction in format_transcript for USER_CONTENT_TOOLS (independent of strip_user_input) - Tests: add invalid model alias rejection test, zero window_sec rejection test, unknown monitor skip test, canary hook tool_input subprocess test, allowlist reason non-enumeration assertion Signed-off-by: Wayne Sun <gsun@redhat.com>
- Unify format_transcript redaction message to match strip_user_input (both now use "tool arguments not shown to monitor") - Use round() instead of integer division in summarize() to preserve fractional accuracy percentages Signed-off-by: Wayne Sun <gsun@redhat.com>
Restore the original experiment result files that match the README's behavioral analysis — the re-run produced "duplicate" results because issue fullsend-ai#2 was identical to fullsend-ai#1. Add trailing newlines to all text files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 participants
Summary
Test plan
uv run pytest tests/ -x -vpasses all unit and integration testsuv run python runner.py --dry-runcompletes without errorsuv run python runner.py --monitor canarydetects canary leaks in expected payloadsuv run python runner.py --monitor allowlistblocks out-of-role tool callsuv run python runner.py --monitor llm --model haikuproduces verdicts for all 27 payloads