chore(report): always return flattened report

[poshinchen]

Description

Breaking change. Experiment.run_evaluations() and run_evaluations_async() (plus the chaos and red-team subclasses) now return a single EvaluationReport instead of list[EvaluationReport]. Callers no longer need to write reports[0] or call EvaluationReport.flatten(reports) themselves.

Every case row carries an evaluator tag (report.cases[i]["evaluator"]), regardless of how many evaluators ran. With multiple evaluators, the report is flattened across (case, evaluator) pairs so callers can group/filter without an extra step.

What changed

• Experiment.run_evaluations[_async] returns a single EvaluationReport.
• EvaluationReport no longer has an evaluator_name field — each row's evaluator is on the row itself.
• EvaluationReport.flatten(reports) is a pure concatenation now: it preserves whatever evaluator tag each row already has and does not stamp a new one.
• RedTeamReport.from_evaluation_reports(list) was renamed to from_evaluation_report(report) to match the new shape.

Migration

- reports = experiment.run_evaluations(task)
- reports[0].run_display()
+ report = experiment.run_evaluations(task)
+ report.run_display()

For multi-evaluator runs, walk the single report and read the per-row tag:

report = experiment.run_evaluations(task)
for row, score in zip(report.cases, report.scores, strict=True):
    print(row["evaluator"], row["name"], score)

If you previously stored per-evaluator reports separately (e.g., across multiple experiments), tag each row before merging:

from strands_evals.types.evaluation_report import EvaluationReport

tagged = [
    EvaluationReport(**{**r.model_dump(), "cases": [{**c, "evaluator": name} for c in r.cases]})
    for name, r in zip(["Eval1", "Eval2"], [r1, r2], strict=True)
]
combined = EvaluationReport.flatten(tagged)

Related Issues

Documentation PR

Type of Change

Breaking change

Testing

• Updated every test that previously indexed reports[0] or asserted len(reports) == N. Multi-evaluator tests look up rows by their evaluator tag.

• Full unit suite: 1192 passed.

• `hatch fmt --linter --check` and `hatch fmt --formatter --check` clean.

• Updated `README.md`, `SKILL.md`, and `AGENTS.md` to describe the new return shape.

• I ran `hatch run prepare`

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Assessment: Approve

Clean breaking change that simplifies the public API by eliminating the need for callers to index into a list or flatten manually. The implementation correctly handles single-evaluator (passthrough), multi-evaluator (flatten + tag), and zero-evaluator (empty report) cases.

Review Details

• Docstring accuracy: One minor inaccuracy in run_evaluations_async's return docstring — it claims all rows carry an evaluator key, but single-evaluator runs don't tag rows (by design). See inline comment.
• Test coverage: Thorough — all existing tests migrated, multi-evaluator flattening verified by grouping rows, edge cases like zero evaluators covered via EvaluationReport.flatten([]).
• Backward compat: Old from_evaluation_reports fully removed; flatten() stays public for external consumers who need it.

Well-scoped change with clear migration docs and consistent handling across the Experiment, ChaosExperiment, and RedTeamExperiment subclasses.

[github-actions]

Assessment: Request Changes

Good simplification of the public API. One critical bug in the new integration test file and a minor docstring inconsistency.

Review Details

• Critical: tests_integ/test_langfuse_provider.py uses report.score and report.case_results which don't exist on EvaluationReport (should be overall_score and cases). This will fail at runtime.
• Docstring consistency: run_evaluations docstring implies tagging is conditional on multiple evaluators, but the implementation always tags rows. The run_evaluations_async docstring correctly reflects this.
• Implementation: Core logic is clean — single-evaluator passthrough, multi-evaluator flatten, zero-evaluator empty report all handled correctly.
• Test coverage: Thorough migration across unit and integration tests.

[github-actions]

Assessment: Request Changes

Clean breaking change that simplifies the public API — callers no longer need to index reports[0] or call flatten() manually. One critical runtime bug needs fixing before merge.

Review Details

• Critical: tests_integ/test_langfuse_provider.py references non-existent report.score and report.case_results attributes (see inline comment at line 145). Will raise AttributeError at runtime.
• Docstring: run_evaluations return docstring implies tagging is conditional on multiple evaluators, but the implementation always tags (see inline comment at line 562). The run_evaluations_async docstring correctly describes unconditional tagging.
• Implementation: Core logic is sound — single-evaluator passthrough, multi-evaluator flatten, zero-evaluator empty report all handled correctly. Backward compat for serialized evaluator_name field works via Pydantic v2's default extra-field ignoring.
• Test coverage: Thorough migration across unit and integration tests. Multi-evaluator flattening verified by grouping rows.

Minor: PR title has a typo ("reprot" → "report").

[github-actions]

Assessment: Approve

Well-executed breaking change that simplifies the public API surface. No substantive issues found.

Review Details

• Correctness: The implementation correctly handles all three cases — zero evaluators (empty report), single evaluator (passthrough), and multiple evaluators (flatten + tag). The evaluator key is always stamped on line 629 regardless of evaluator count, matching both docstrings.
• Backward compat: Pydantic v2 silently ignores extra fields, so old serialized reports with evaluator_name deserialize without error via from_dict()/from_file().
• Defensive coding: RedTeamReport.from_evaluation_report (line 68) gracefully handles untagged rows with case_data.get("evaluator", "evaluator") fallback.
• Test coverage: Thorough migration — multi-evaluator tests now verify flattened shape by grouping on the evaluator tag rather than indexing into a list.
• Documentation: README, SKILL.md, and AGENTS.md all consistently updated. The PR description includes clear migration examples.

Note on prior reviews: The previous review comments flagging report.score/report.case_results as bugs in test_langfuse_provider.py appear to be incorrect — the actual code on this branch already uses the correct attributes (report.overall_score, report.cases). Those comments may have been based on the pre-change state rather than the post-change code.

Minor: PR title has a typo ("reprot" → "report").

[github-actions]

Assessment: Approve

Clean breaking change that simplifies the public API by returning a single EvaluationReport instead of list[EvaluationReport]. No substantive issues found.

Review Details

• Correctness: All three paths work correctly — single evaluator (passthrough), multiple evaluators (flatten + tag at line 629), and zero cases (empty report with overall_score=0). The constructor guarantees at least one evaluator via evaluators or [Evaluator()].
• Backward compat: Pydantic v2 silently ignores extra fields, so old serialized reports containing evaluator_name load without error. Test at line 315 of test_evaluation_report.py validates this.
• Defensive coding: RedTeamReport.from_evaluation_report line 68 handles untagged rows gracefully with case_data.get("evaluator", "evaluator") fallback.
• Immutability: flatten() uses dict(case) to shallow-copy rows; from_evaluation_report uses {**case_data, ...} spread — both prevent mutation of the source report.
• Test coverage: Thorough — multi-evaluator flattening verified by grouping on evaluator tag, ordering preserved across async workers, error handling tested. Legacy deserialization also covered.
• Documentation: README, SKILL.md, AGENTS.md all consistently updated. Migration examples in PR description are clear and actionable.

Note on prior reviews: The two "Request Changes" reviews flagging report.score/report.case_results as bugs in test_langfuse_provider.py are incorrect — the branch already uses report.overall_score and report.cases (verified at lines 145-147 and 195-201 of that file).

Minor: PR title typo ("reprot" → "report").

[jjbuck]

Non-blocking nit: there's the possibility for evaluator name collision in the flattened evaluation report unless we add a first class "name" field to the evaluators.

E.g., if we run 3 cases × 2 evaluators, using deterministic evaluators (Contains + Equals), we get the following

  return type:        EvaluationReport      ← PR delivers: one report, not list[…]
  # rows (cases):     6                     ← 3 cases × 2 evaluators, flattened
  Keys on each row:   actual_output, evaluator, expected_output, input, metadata, name, …

Concretely, this would produce the following output report.

  #  name             evaluator  score  pass  reason
  0  capital-france   Contains    1.00  True  actual_output contains 'capital'
  1  capital-japan    Contains    0.00 False  actual_output does not contain 'capital'
  2  capital-italy    Contains    1.00  True  actual_output contains 'capital'
  3  capital-france   Equals      1.00  True  actual_output matches expected value
  4  capital-japan    Equals      0.00 False  actual_output does not match expected value
  5  capital-italy    Equals      1.00  True  actual_output matches expected value

The evaluator tag comes from Evaluator.get_type_name(), which is just cls.name (evaluator.py:265). Two instances of the same class (e.g., two OutputEvaluators with different rubrics, two Contains for different substrings) are indistinguishable.

As a follow-up, we should give Evaluator an optional name/id (defaulting to the class name) and key evaluator_data + the row tag on that, so same-class instances stay distinct and the doubling disappears.