Skip to content

docs(redteam): add module README - #271

Merged
poshinchen merged 2 commits into
strands-agents:mainfrom
yeomjiwonyeom:redteam/readme
Jun 16, 2026
Merged

docs(redteam): add module README#271
poshinchen merged 2 commits into
strands-agents:mainfrom
yeomjiwonyeom:redteam/readme

Conversation

@yeomjiwonyeom

@yeomjiwonyeom yeomjiwonyeom commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Adds a README for the experimental red-team module (src/strands_evals/experimental/redteam/).

Covers:

  • Overview + an ASCII flow diagram (case → experiment → strategy → judge → report)
  • Quick start running several strategies against an agent
  • The five attack strategies with mechanism + paper links (Crescendo, GOAT, PAIR, Bad Likert Judge, SequentialBreak)
  • The five risk categories
  • Hand-authoring custom cases
  • A worked report.display() sample and how to read the breach matrix

The full user guide lives in the Strands docs (separate harness-sdk PR); this README is the in-repo entry point for contributors. Every code sample and API reference was verified against the current package.

Add a README for the experimental red-team module: overview + diagram, quick
start (multi-strategy), the five attack strategies with paper links, risk
categories, custom-case authoring, and a worked report sample with how to read
it. Points to the hosted user guide for the full reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation area-community Repo health, governance, contributor process, release process, and CI dependency bumps enhancement New feature or request strands-running labels Jun 16, 2026
Comment thread src/strands_evals/experimental/redteam/README.md
Comment thread src/strands_evals/experimental/redteam/README.md Outdated
Comment thread src/strands_evals/experimental/redteam/README.md
Comment thread src/strands_evals/experimental/redteam/README.md
Comment thread src/strands_evals/experimental/redteam/README.md
Comment thread src/strands_evals/experimental/redteam/README.md Outdated
@github-actions

Copy link
Copy Markdown

Assessment: Request Changes

Thanks for putting together an entry-point README for the red-team module — the structure is clear and the prose is genuinely good. However, the PR description states "Every code sample and API reference was verified against the current package," and that doesn't hold: large parts of the README document an API that isn't in the merged code. A contributor following the Quick Start today would hit ImportError on the first block.

Review themes
  • Non-existent exports: CrescendoStrategy, GoatStrategy, PairStrategy, BadLikertJudgeStrategy, SequentialBreakStrategy and TargetSession are documented but not implemented/exported. __all__ only exposes AttackStrategy and PromptStrategy; the lone built-in strategy is gradual_escalation.
  • Wrong signatures: generate_cases(agent=...) → actual is target=; RedTeamExperiment(agent=..., attack_strategies=[...]) → actual is (cases, *, target, evaluators, max_turns, model) with no attack_strategies; the run_attack(...) strategy contract doesn't exist (base class uses enhance()/reset()).
  • Module layout drift: task.py and strategies/ descriptions reference TargetSession/MultiAgentBase that aren't in the code.
  • Accurate sections ✓: Risk categories, the custom-cases example (RedTeamCase/RedTeamConfig/AttackGoal), the report API (attack_results/passed/score/by_risk_category), and the pass_threshold 0.3 / partial-anchor explanation all check out.

Recommend a quick python -c "from strands_evals.experimental.redteam import *" plus actually running the Quick Start before merge. If the five strategies are planned-but-unmerged, label them clearly as "planned" rather than documenting them as available. The bones of this doc are solid — it mainly needs to be re-synced to the code that's actually shipping.

@poshinchen

Copy link
Copy Markdown
Contributor

/strands check your comments, they are all incorrect right?

@github-actions

Copy link
Copy Markdown

You're right — I re-verified every point against the code on this branch (5c9ed85), installed the package, and ran the import/signature checks. All six of my review comments were incorrect. They were generated against stale pre-#265 code; this PR correctly documents the current main API.

Verified against source + runtime introspection:

  • TargetSession ✅ exists — exported from __init__.py, implemented in strategies/target_session.py (with StrandsAgentSession / StrandsMultiAgentSession).
  • Five strategies ✅ all exported and importable: CrescendoStrategy, GoatStrategy, PairStrategy, BadLikertJudgeStrategy, SequentialBreakStrategy.
  • generate_cases ✅ signature is generate_cases(self, *, agent: Agent | TargetSpec, ...)agent= is correct, not target=.
  • RedTeamExperiment.__init__(self, cases=None, *, agent=None, attack_strategies=None, evaluators=None, model=None) — both agent= and attack_strategies= exist.
  • run_attack ✅ contract is run_attack(self, case, target_session, *, max_turns, model=None, **kwargs)enhance() is the pre-refactor(redteam): per-call judge lifecycle + per-risk-category judge rubric + minor strategy improvements #265 API.
  • MultiAgentBase ✅ supported — task.py wraps it in StrandsMultiAgentSession.

python -c "from strands_evals.experimental.redteam import *" and all the documented imports succeed (ALL IMPORTS OK). The README's code samples and API references match the shipping package. Apologies for the noise — withdrawing the "Request Changes" review; no documentation changes are needed.

@github-actions

Copy link
Copy Markdown

Assessment: Approve

Re-reviewed against the actual PR branch HEAD (redteam/readme, ac73b18) and confirming: the README is accurate, and my earlier "Request Changes" review was wrong.

Root cause of the bad review: my first pass resolved main to the fork's stale branch (pre-#265) and I never checked out the PR HEAD, so I compared the doc against an older API surface. That's on me.

Re-verified against the code on this branch + runtime introspection (pip install -e . then importing every documented symbol → ALL IMPORTS OK):

  • TargetSession ✅ exported from __init__.py, implemented in strategies/target_session.py (StrandsAgentSession / StrandsMultiAgentSession)
  • All five strategies ✅ exported and importable: CrescendoStrategy, GoatStrategy, PairStrategy, BadLikertJudgeStrategy, SequentialBreakStrategy
  • RedTeamExperiment(cases=None, *, agent=None, attack_strategies=None, evaluators=None, model=None) ✅ matches the Quick Start
  • generate_cases(*, agent, risk_categories=None, num_cases=5) ✅ — agent= is correct
  • run_attack(case, target_session, *, max_turns, model=None, ...) ✅ matches the documented contract
  • MultiAgentBase ✅ supported via StrandsMultiAgentSession

The code samples and API references match the shipping package. No documentation changes needed — withdrawing the prior Request Changes. Nice, well-structured README. 👍

@poshinchen
poshinchen merged commit 2584799 into strands-agents:main Jun 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-community Repo health, governance, contributor process, release process, and CI dependency bumps area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants