-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Update prompt injection detection metrics (due to errors exporting to datadog) #5692
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR attempts to fix Datadog metrics export errors by removing problematic elements from log messages: format string placeholders with {}, gauge.goose.* metric keys, newlines in explanation strings, and emojis.
Key changes:
- Converted
gauge.goose.*metrics tocounter.goose.* - Sanitized explanation strings to replace newlines with " | " for metrics
- Simplified log messages to remove format placeholders in some locations
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| crates/goose/src/security/mod.rs | Changed gauge metrics to counters, added explanation sanitization for metrics, and partially simplified log messages (but emojis and some format placeholders remain) |
| crates/goose/src/agents/tool_execution.rs | Removed format placeholders from user decision logging, converted to structured logging with fields |
Comments suppressed due to low confidence (1)
crates/goose/src/security/mod.rs:102
- The log message still uses a format placeholder with
"{}"on line 97 and contains emojis (🔒) on lines 99 and 101. According to the PR description, both format placeholders and emojis may be causing Datadog export errors and should be removed.
"{}",
if above_threshold {
"🔒 Current tool call flagged as malicious after security analysis (above threshold)"
} else {
"🔒 Security finding below threshold - logged but not blocking execution"
}
* origin/main: (51 commits) Compaction resiliency improvements (#5618) docs: ask goose button (#5730) Update prompt injection detection metrics (due to errors exporting to datadog) (#5692) Spence/icon2 tokyo drift (#5728) docs: logs rotation and misc updates (#5727) docs: automatic ollama model detection (#5725) Fix context progress bar not resetting after /clear command (#5652) docs: removing double announcements (#5714) docs: mcp sampling support (#5708) hackathon banner (#5710) Fix documentation-only change detection for push events (#5712) Added transaction commits to multi sql functions in session_manager (#5693) fix: improve and simplify tool call chain rendering (#5704) Fix: Always show autocompact threshold ui (#5701) chore: Update governance to include Discord (#5690) Ollama improvements (#5609) feat: add Supabase MCP server to registry (#5629) Unlist VS Code extension tutorials from MCP and experimental sections (#5677) fix: make image processing work in github copilot provider (#5687) fix: do not take into account gitignore in developer mcp (#5688) ...
* main: (65 commits) Fix: Recipes respect the quiet flag (#5743) docs: update cli commands (#5744) Run smoke tests on a free runner (#5745) faster, cheaper (pick two): improve CI workflow and switch to free github runner (#5702) Compaction resiliency improvements (#5618) docs: ask goose button (#5730) Update prompt injection detection metrics (due to errors exporting to datadog) (#5692) Spence/icon2 tokyo drift (#5728) docs: logs rotation and misc updates (#5727) docs: automatic ollama model detection (#5725) Fix context progress bar not resetting after /clear command (#5652) docs: removing double announcements (#5714) docs: mcp sampling support (#5708) hackathon banner (#5710) Fix documentation-only change detection for push events (#5712) Added transaction commits to multi sql functions in session_manager (#5693) fix: improve and simplify tool call chain rendering (#5704) Fix: Always show autocompact threshold ui (#5701) chore: Update governance to include Discord (#5690) Ollama improvements (#5609) ...
* main: scan recipe for security when saving recipe (#5747) feat: trying grok for live test (#5732) Platform Extension MOIM (Minus One Info Message) (#5027) docs: remove hackathon banner (#5756) Fix: Recipes respect the quiet flag (#5743) docs: update cli commands (#5744) Run smoke tests on a free runner (#5745) faster, cheaper (pick two): improve CI workflow and switch to free github runner (#5702) Compaction resiliency improvements (#5618) docs: ask goose button (#5730) Update prompt injection detection metrics (due to errors exporting to datadog) (#5692) Spence/icon2 tokyo drift (#5728) docs: logs rotation and misc updates (#5727) docs: automatic ollama model detection (#5725) Fix context progress bar not resetting after /clear command (#5652) docs: removing double announcements (#5714) docs: mcp sampling support (#5708)
… datadog) (block#5692) Signed-off-by: Blair Allan <[email protected]>
Summary
There were errors exporting metrics to Datadog, although local testing didn't cause any issues which makes it harder to troubleshoot. Suspicion is that the issues may be due to formatted string messages with
{}placeholders,gauge.goose.*as attribute keys, newlines in explanation strings, or emojis in message strings.Part of error log (not super descriptive):
Type of Change
AI Assistance
No AI assistance on this PR
Testing
Manual testing - I set up a local Docker OTLP collector to capture and inspect the raw metric payloads from goose, which allowed me to verify that the prompt injection metrics were emitted correctly (along with looking at the already existing metrics).