feat(redteam): redesign risk categories for agent-centric evaluation - #290
Open
kevmyung wants to merge 1 commit into
Open
feat(redteam): redesign risk categories for agent-centric evaluation#290kevmyung wants to merge 1 commit into
kevmyung wants to merge 1 commit into
Conversation
Replaces the original 5 broad categories with 7 agent-centric ones that better capture the threat landscape for tool-using AI agents: - prompt_injection (merges guideline_bypass + system_prompt_leak) - excessive_agency (narrowed: tool calls beyond authorization scope) - sensitive_data (new: disclosure of protected info the agent holds) - data_exfiltration (kept: outbound transmission via tools) - social_engineering (new: manipulative messages targeting humans) - privilege_escalation (new: accessing higher-privilege resources) - business_rule_violation (new: policy-violating action patterns) RISK_CATEGORIES stays a plain dict[str, str] (slug -> description). The attack_success_v0 rubric, the adversarial case generator, and its prompt template are updated to the new category set; BadLikertJudge's per-category top-score anchor (_SCORE_TOP_ANCHOR) stays local to that strategy, since only BLJ elicits a scored example. Updates the dependent tests and docs (README, SKILL.md) to the new names.
kevmyung
had a problem deploying
to
manual-approval
July 1, 2026 20:15 — with 
GitHub Actions
Failure
GitHub Actions
Failure
kevmyung
had a problem deploying
to
manual-approval
July 1, 2026 20:15 — with
GitHub Actions
Failure
GitHub Actions
Failure
github-actions
Bot
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces the 5 original red team risk categories — written for a general chatbot threat model — with 7 agent-centric ones that map to how a tool-using agent actually gets compromised.
prompt_injectionguideline_bypass+system_prompt_leak)excessive_agencysensitive_datadata_exfiltrationsocial_engineeringprivilege_escalationbusiness_rule_violationRISK_CATEGORIESstays a plaindict[str, str](slug → description). Updated together:attack_success_v0rubric — per-category success definitions for the new setAdversarialCaseGenerator+ itsadversarial_v0prompt templateBadLikertJudge— per-category score anchors renamed; the strategy-specific top-score anchor (_SCORE_TOP_ANCHOR) stays local to BLJ rather than being pushed into the shared category mapRelated Issues
Closes #289
Documentation PR
N/A — README and SKILL.md are updated in this PR.
Type of Change
New feature
Testing
hatch run preparehatch run preparepasses end to end: ruff lint + format clean, and the full test suite is green across the Python version matrix (1663 passed each).Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.