fix: CorrectnessEvaluator honors expected_output for reference-mode grading - #356
Open
AmirF194 wants to merge 1 commit into
Open
Conversation
…rading A case that sets only expected_output (documented as "the expected response given the input") silently fell back to _evaluate_basic()'s trajectory grading, since _has_reference() gated reference mode on expected_assertion alone. That path can score a response CORRECT even when it contradicts the expected output, with no warning. _has_reference() now treats either field as a reference; _format_reference_prompt() prefers expected_assertion when both are set (unchanged behavior) and falls back to expected_output otherwise. Fixes strands-agents#334
AmirF194
requested a deployment
to
manual-approval
August 8, 2026 12:32 — with
GitHub Actions
Waiting
AmirF194
requested a deployment
to
manual-approval
August 8, 2026 12:32 — with
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CorrectnessEvaluatoronly enters reference-mode grading whenexpected_assertionis set on the evaluation case.expected_outputis a separate, documented field onEvaluationData("the expected response given the input"), but nothing incorrectness_evaluator.pyever reads it:_has_reference()checksexpected_assertionalone, and_format_reference_prompt()interpolates onlyexpected_assertioninto the judge prompt. A case that setsexpected_outputwithout also settingexpected_assertionsilently falls through to_evaluate_basic()'s trajectory-only grading, which can score a response CORRECT even when it contradicts the expected output, with no warning to the caller.Fix:
_has_reference()now treats either field as a reference, and_format_reference_prompt()prefersexpected_assertionwhen both are set (unchanged behavior for existing callers) and falls back toexpected_outputwhen only that is present.Related Issues
Fixes #334
Documentation PR
None; docstring updated in place, no separate docs page describes this behavior.
Type of Change
Bug fix
Testing
tests/strands_evals/evaluators/test_correctness_evaluator.py::test_has_reference_true_with_expected_output_only,test_format_reference_prompt_uses_expected_output_when_no_assertion, andtest_evaluate_with_expected_output_only_uses_reference_modefail on main and pass with this change.Full suite passes:
hatch test tests --cover(1760 passed).hatch run test-lint(ruff check + mypy) is clean.Not checked: behavior against a live model call in reference mode; the new tests mock the judge model the same way the existing reference-mode tests do.
I ran
hatch run prepare(ran its constituent checks separately:hatch run test-lintandhatch test tests --cover, not the formatter step, to avoid reformatting unrelated pre-existing files)Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.