Skip to content

Logging uplift for model training purposes (command injection model) [Small change] #6330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dorien-koelemeijer merged 1 commit into from
Jan 27, 2026
Merged

Logging uplift for model training purposes (command injection model) [Small change] #6330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion crates/goose/src/agents/tool_execution.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,8 @@ impl Agent {
counter.goose.prompt_injection_user_decisions = 1,
decision = ?confirmation.permission,
finding_id = %finding_id,
"User security decision"
tool_request_id = %request.id,
"Prompt injection detection: user decision on command injection finding"
);
}

Expand Down
11 changes: 8 additions & 3 deletions crates/goose/src/security/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,20 +114,25 @@ impl SecurityManager {
let above_threshold = analysis_result.confidence > config_threshold;
let finding_id = format!("SEC-{}", Uuid::new_v4().simple());

let tool_call_json =
serde_json::to_string(&tool_call).unwrap_or_else(|_| "{}".to_string());

tracing::warn!(
counter.goose.prompt_injection_finding = 1,
threat_type = "command_injection",
above_threshold = above_threshold,
tool_name = %tool_call.name,
tool_request_id = %tool_request.id,
tool_call_json = %tool_call_json,
Comment on lines +117 to +126

Copilot AI Jan 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging the complete tool_call JSON may expose sensitive data. Tool arguments can contain credentials, API keys, file paths, or other sensitive information that could be logged to Snowflake. Consider sanitizing or redacting sensitive fields from tool_call.arguments before serialization, or implement a privacy filter for the logged data.

Suggested change
let tool_call_json =
serde_json::to_string(&tool_call).unwrap_or_else(|_| "{}".to_string());
tracing::warn!(
counter.goose.prompt_injection_finding = 1,
threat_type = "command_injection",
above_threshold = above_threshold,
tool_name = %tool_call.name,
tool_request_id = %tool_request.id,
tool_call_json = %tool_call_json,
tracing::warn!(
counter.goose.prompt_injection_finding = 1,
threat_type = "command_injection",
above_threshold = above_threshold,
tool_name = %tool_call.name,
tool_request_id = %tool_request.id,

Copilot uses AI. Check for mistakes.

@dorien-koelemeijer dorien-koelemeijer Jan 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if any sanitisation happens (i.e. in vector fx), otherwise think of another way + check if these logs go over the public internet

@DOsinga DOsinga Jan 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think the robot makes a good point. can we log something here that allows us to get the actual LLM request back from databricks? that would make sure only people who have access to that can see these potentially PII data

@michaelneale michaelneale Jan 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah this is problematic, could possibly do name only I would think.

I wonder if this isn't the right level to log things at.

@dorien-koelemeijer dorien-koelemeijer Jan 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have set up a sensitive data filter in the OTLP logs here: https://github.com/squareup/ccd-goose-otlp-proxy/pull/12

confidence = analysis_result.confidence,
explanation = %sanitized_explanation,
finding_id = %finding_id,
threshold = config_threshold,
"{}",
if above_threshold {
"Current tool call flagged as malicious after security analysis (above threshold)"
"Prompt injection detection: Current tool call flagged as malicious after security analysis (above threshold)"
} else {
"Security finding below threshold - logged but not blocking execution"
"Prompt injection detection: Security finding below threshold (logged but not blocking execution)"
}
);
if above_threshold {
Expand Down Expand Up @@ -155,7 +160,7 @@ impl SecurityManager {
tracing::info!(
counter.goose.prompt_injection_analysis_performed = 1,
security_issues_found = results.len(),
"Security analysis complete"
"Prompt injection detection: Security analysis complete"
);
Ok(results)
}
Expand Down
Loading