feat(evaluators): allow custom tools on judge-based evaluators (Trajectory, Output, Multimodal)

[pdebjyot]

Closes #322

Description

Adds an optional tools= constructor parameter so the judge agent can call domain-specific verification functions during evaluation. Follows the same opt-in pattern accepted for ToolSimulator in #208 / #209.

Covered evaluators: TrajectoryEvaluator, OutputEvaluator, MultimodalOutputEvaluator, and its specialized subclasses (MultimodalCorrectnessEvaluator, MultimodalFaithfulnessEvaluator, MultimodalInstructionFollowingEvaluator, MultimodalOverallQualityEvaluator).

• TrajectoryEvaluator: user tools are appended after the default trajectory scoring tools (exact/in-order/any-order match), which are always retained.
• OutputEvaluator: tools defaults to None, preserving the current no-tools judge (Agent accepts tools=None).
• Multimodal evaluators: MultimodalOutputEvaluator accepts tools and forwards it to OutputEvaluator; the four specialized subclasses accept and forward it likewise, so tool-augmented judging works for multimodal rubrics too.

Non-breaking: tools is appended after name in every signature, so existing positional and keyword callers are unaffected; omitting it keeps current behavior exactly.

Example

from strands import tool
from strands_evals.evaluators import TrajectoryEvaluator

@tool
def verify_ticket_id(ticket_id: str) -> str:
    """Check whether a ticket ID matches the support system's format."""
    return "valid" if ticket_id.startswith("TKT-") else "invalid format"

evaluator = TrajectoryEvaluator(
    rubric="Verify the agent referenced only valid ticket IDs.",
    tools=[verify_ticket_id],
)

Testing

• New unit tests: constructor defaults preserved, merged tools list asserted explicitly (trajectory), tools passed through to the judge agent (output, sync + async), and tools=None default verified.
• Full evaluator + experiment unit suites pass; ruff check and ruff format clean.

Checklist

• Non-breaking change
• Unit tests added
• Conventional commit
• Lint/format clean

[github-actions]

Assessment: Comment

Clean, well-tested, opt-in addition that follows the established ToolSimulator pattern — tests pass locally (36) and ruff is clean. One correctness concern about parameter ordering worth addressing before merge; the rest are minor.

Review Categories

• API / Compatibility: New tools param is inserted before the existing name param in both evaluators, which shifts name's positional index and can silently break positional callers — this makes the change technically breaking despite the "Non-breaking" note. Appending it after name or making trailing params keyword-only resolves it.
• Consistency: OutputEvaluator extracted a _create_evaluator_agent() helper, but TrajectoryEvaluator still duplicates the Agent(...) construction across sync/async paths.
• Testing: Good coverage of the new behavior; one redundant/circular assertion could be tightened to an explicit expected list.
• Scope: The multimodal subclasses inherit the helper but don't expose/forward tools, so the description slightly overstates coverage there.

Nice use of the existing opt-in convention and thorough backward-compatibility tests.

[poshinchen]

In the description:

via a new _create_evaluator_agent() helper (inherited by MultimodalOutputEvaluator subclasses).

This is incorrect.

[github-actions]

Assessment: Approve (pending a doc tidy-up)

Re-reviewed at 36a4cee. All feedback from the previous round is addressed cleanly:

• ✅ tools now appended after name in OutputEvaluator, TrajectoryEvaluator, and MultimodalOutputEvaluator — no positional-ordering break.
• ✅ Dropped the _create_evaluator_agent() helper in favor of passing tools=self.tools directly (relying on Agent(tools=None) being valid) — consistent across sync/async.
• ✅ MultimodalOutputEvaluator now accepts and forwards tools.
• ✅ Test tightened to assert the explicit merged tools list instead of the redundant/circular membership check.

Verified locally: 52 evaluator tests pass and ruff check is clean.

Two minor, non-blocking follow-ups:

1. Stale PR description: it still says user tools are "merged ahead of the default trajectory scoring tools" and references the now-removed _create_evaluator_agent() helper. The code now appends user tools after the defaults ([*defaults, *(tools or [])]), so the description should be updated to match.
2. Attribute naming consistency (optional): OutputEvaluator exposes self.tools (public) while TrajectoryEvaluator keeps self._tools (private). Reasonable given _tools also holds the internal default scorers, but worth a quick thought on whether the two should align.

Nice, responsive iteration — this is in good shape.

[poshinchen]

I see that you've updated other evaluators too, could you update the description / title?

[pdebjyot]

Updated the title and description to cover all touched evaluators — the multimodal family (MultimodalOutputEvaluator and its four specialized subclasses) now accepts and forwards tools as well, which their to_dict/from_dict round-trip tests require once the attribute is public. Thanks for the catch.