diff --git a/AGENTS.md b/AGENTS.md index ef645816..18773922 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -150,7 +150,8 @@ strands-evals/ │ │ ├── run.py │ │ ├── validate.py │ │ ├── report.py -│ │ └── diagnose.py +│ │ ├── diagnose.py +│ │ └── generate.py │ │ │ ├── __main__.py # `python -m strands_evals` shim → cli.main │ │ @@ -191,7 +192,7 @@ strands-evals/ ### Directory Purposes - **`src/strands_evals/`**: All production code -- **`src/strands_evals/cli/`**: `strands-evals` console script — four subcommands (`run`, `validate`, `report`, `diagnose`) plus the `module:attr` resolver and synthesized task wrapper used by `--agent` +- **`src/strands_evals/cli/`**: `strands-evals` console script — five subcommands (`run`, `validate`, `report`, `diagnose`, `generate`) plus the `module:attr` resolver and synthesized task wrapper used by `--agent` - **`tests/strands_evals/`**: Unit tests mirroring src/ structure - **`tests/strands_evals/cli/`**: CLI tests; fixtures live under `cli/fixtures/` and are referenced by tests via `tests.strands_evals.cli.fixtures.*:attr` specs - **`tests_integ/`**: Integration tests using real trace providers and model endpoints diff --git a/README.md b/README.md index 8114c99b..a5b95505 100644 --- a/README.md +++ b/README.md @@ -97,6 +97,43 @@ report = experiment.run_evaluations(get_response) report.run_display() ``` +## Command-Line Interface + +Installing `strands-agents-evals` also installs the `strands-evals` console script, a thin wrapper over the Python API for CI and one-off use. It has five subcommands: + +| Command | Purpose | +| --- | --- | +| `strands-evals run` | Execute an Experiment against an `--agent` factory or `--task` callable, or run a single ad-hoc case via `--input` + `--evaluator`/`--expected-output`/`--rubric`. | +| `strands-evals validate` | Schema-check a serialized Experiment JSON file (CI gate before `run`). | +| `strands-evals report` | Render an existing `EvaluationReport` JSON via Rich, or dump it as JSON. | +| `strands-evals diagnose` | Run `detect_failures`, `analyze_root_cause`, or the full `diagnose_session` pipeline on a Session JSON file. | +| `strands-evals generate` | Synthesize an Experiment via `ExperimentGenerator` from a free-form `--context` or an existing `--experiment` file. | + +Common flows: + +```bash +# Schema-check, then run an experiment file against an agent factory +strands-evals validate experiments/customer_service.json +strands-evals run experiments/customer_service.json \ + --agent my_pkg.agents:build_agent \ + --display + +# One-off ad-hoc run with no experiment file +strands-evals run \ + --input "What is the capital of France?" \ + --expected-output "Paris" \ + --agent my_pkg.agents:build_agent + +# Diagnose a failing session captured to JSON +strands-evals diagnose session.json --confidence medium + +# Generate a starter experiment from a tools description +strands-evals generate --context "$(cat tools.txt)" --num-cases 10 \ + --evaluator TrajectoryEvaluator -o experiments/generated.json +``` + +Run any subcommand with `--help` for the full flag set (custom evaluators via `MODULE:CLASS`, trace attributes, `--fail-on` exit-code rules, output formats, etc.). + ## Installation Ensure you have Python 3.10+ installed, then: diff --git a/SKILL.md b/SKILL.md index db24d0a8..20b2b672 100644 --- a/SKILL.md +++ b/SKILL.md @@ -323,6 +323,7 @@ Only import from `typing` for symbols without a built-in equivalent: `Any`, `Cal ## Pointers - Repo conventions, contribution rules, prompt versioning, review checklist: `AGENTS.md` +- CLI workflow (`strands-evals run` / `validate` / `diagnose` / `report` / `generate`): see the README "Command-Line Interface" section and `--help` on each subcommand - Logging style: `STYLE_GUIDE.md` - Human contributor guide: `CONTRIBUTING.md` - User docs: https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/quickstart/