Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,8 @@ strands-evals/
│ │ ├── run.py
│ │ ├── validate.py
│ │ ├── report.py
│ │ └── diagnose.py
│ │ ├── diagnose.py
│ │ └── generate.py
│ │
│ ├── __main__.py # `python -m strands_evals` shim → cli.main
│ │
Expand Down Expand Up @@ -191,7 +192,7 @@ strands-evals/
### Directory Purposes

- **`src/strands_evals/`**: All production code
- **`src/strands_evals/cli/`**: `strands-evals` console script — four subcommands (`run`, `validate`, `report`, `diagnose`) plus the `module:attr` resolver and synthesized task wrapper used by `--agent`
- **`src/strands_evals/cli/`**: `strands-evals` console script — five subcommands (`run`, `validate`, `report`, `diagnose`, `generate`) plus the `module:attr` resolver and synthesized task wrapper used by `--agent`
- **`tests/strands_evals/`**: Unit tests mirroring src/ structure
- **`tests/strands_evals/cli/`**: CLI tests; fixtures live under `cli/fixtures/` and are referenced by tests via `tests.strands_evals.cli.fixtures.*:attr` specs
- **`tests_integ/`**: Integration tests using real trace providers and model endpoints
Expand Down
37 changes: 37 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,43 @@ report = experiment.run_evaluations(get_response)
report.run_display()
```

## Command-Line Interface

Installing `strands-agents-evals` also installs the `strands-evals` console script, a thin wrapper over the Python API for CI and one-off use. It has five subcommands:

| Command | Purpose |
| --- | --- |
| `strands-evals run` | Execute an Experiment against an `--agent` factory or `--task` callable, or run a single ad-hoc case via `--input` + `--evaluator`/`--expected-output`/`--rubric`. |
| `strands-evals validate` | Schema-check a serialized Experiment JSON file (CI gate before `run`). |
| `strands-evals report` | Render an existing `EvaluationReport` JSON via Rich, or dump it as JSON. |
| `strands-evals diagnose` | Run `detect_failures`, `analyze_root_cause`, or the full `diagnose_session` pipeline on a Session JSON file. |
| `strands-evals generate` | Synthesize an Experiment via `ExperimentGenerator` from a free-form `--context` or an existing `--experiment` file. |

Common flows:

```bash
# Schema-check, then run an experiment file against an agent factory
strands-evals validate experiments/customer_service.json
strands-evals run experiments/customer_service.json \
--agent my_pkg.agents:build_agent \
--display

# One-off ad-hoc run with no experiment file
strands-evals run \
--input "What is the capital of France?" \
--expected-output "Paris" \
--agent my_pkg.agents:build_agent

# Diagnose a failing session captured to JSON
strands-evals diagnose session.json --confidence medium

# Generate a starter experiment from a tools description
strands-evals generate --context "$(cat tools.txt)" --num-cases 10 \
--evaluator TrajectoryEvaluator -o experiments/generated.json
```

Run any subcommand with `--help` for the full flag set (custom evaluators via `MODULE:CLASS`, trace attributes, `--fail-on` exit-code rules, output formats, etc.).

## Installation

Ensure you have Python 3.10+ installed, then:
Expand Down
1 change: 1 addition & 0 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,7 @@ Only import from `typing` for symbols without a built-in equivalent: `Any`, `Cal
## Pointers

- Repo conventions, contribution rules, prompt versioning, review checklist: `AGENTS.md`
- CLI workflow (`strands-evals run` / `validate` / `diagnose` / `report` / `generate`): see the README "Command-Line Interface" section and `--help` on each subcommand
- Logging style: `STYLE_GUIDE.md`
- Human contributor guide: `CONTRIBUTING.md`
- User docs: https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/quickstart/
Loading