Skip to content

fix: CorrectnessEvaluator honors expected_output for reference-mode grading - #356

Open
AmirF194 wants to merge 1 commit into
strands-agents:mainfrom
AmirF194:fix/334-correctness-evaluator-expected-output
Open

fix: CorrectnessEvaluator honors expected_output for reference-mode grading#356
AmirF194 wants to merge 1 commit into
strands-agents:mainfrom
AmirF194:fix/334-correctness-evaluator-expected-output

Conversation

@AmirF194

@AmirF194 AmirF194 commented Aug 8, 2026

Copy link
Copy Markdown

Description

CorrectnessEvaluator only enters reference-mode grading when expected_assertion is set on the evaluation case. expected_output is a separate, documented field on EvaluationData ("the expected response given the input"), but nothing in correctness_evaluator.py ever reads it: _has_reference() checks expected_assertion alone, and _format_reference_prompt() interpolates only expected_assertion into the judge prompt. A case that sets expected_output without also setting expected_assertion silently falls through to _evaluate_basic()'s trajectory-only grading, which can score a response CORRECT even when it contradicts the expected output, with no warning to the caller.

Fix: _has_reference() now treats either field as a reference, and _format_reference_prompt() prefers expected_assertion when both are set (unchanged behavior for existing callers) and falls back to expected_output when only that is present.

Related Issues

Fixes #334

Documentation PR

None; docstring updated in place, no separate docs page describes this behavior.

Type of Change

Bug fix

Testing

  • tests/strands_evals/evaluators/test_correctness_evaluator.py::test_has_reference_true_with_expected_output_only, test_format_reference_prompt_uses_expected_output_when_no_assertion, and test_evaluate_with_expected_output_only_uses_reference_mode fail on main and pass with this change.

  • Full suite passes: hatch test tests --cover (1760 passed).

  • hatch run test-lint (ruff check + mypy) is clean.

  • Not checked: behavior against a live model call in reference mode; the new tests mock the judge model the same way the existing reference-mode tests do.

  • I ran hatch run prepare (ran its constituent checks separately: hatch run test-lint and hatch test tests --cover, not the formatter step, to avoid reformatting unrelated pre-existing files)

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…rading

A case that sets only expected_output (documented as "the expected response
given the input") silently fell back to _evaluate_basic()'s trajectory
grading, since _has_reference() gated reference mode on expected_assertion
alone. That path can score a response CORRECT even when it contradicts the
expected output, with no warning.

_has_reference() now treats either field as a reference; _format_reference_prompt()
prefers expected_assertion when both are set (unchanged behavior) and falls
back to expected_output otherwise.

Fixes strands-agents#334
@AmirF194
AmirF194 requested a review from a team as a code owner August 8, 2026 12:32
@AmirF194
AmirF194 requested a review from mkmeral August 8, 2026 12:32
@github-actions github-actions Bot added bug Something isn't working area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics labels Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] CorrectnessEvaluator ignores Case.expected_output and silently falls back to trajectory grading

1 participant