Skip to content

feat(experiment): add opt-in model routing for evaluators - #325

Open
pdebjyot wants to merge 1 commit into
strands-agents:mainfrom
pdebjyot:feat/model-router
Open

feat(experiment): add opt-in model routing for evaluators#325
pdebjyot wants to merge 1 commit into
strands-agents:mainfrom
pdebjyot:feat/model-router

Conversation

@pdebjyot

Copy link
Copy Markdown
Contributor

Closes #323
Related: #88

Description

Adds ModelRouter and RoutingRule (new module strands_evals/model_router.py), plus an optional model_router= parameter on Experiment, so evaluation cost/accuracy can be matched to evaluator complexity: fast models for structural/rubric checks, strong models for nuanced judgment — including case-dependent escalation (e.g., long traces get a stronger judge).

Semantics:

  • Opt-in: no model_router → behavior unchanged.
  • Explicit wins: an evaluator constructed with model=... is never re-routed.
  • First matching rule wins; rules match on evaluator type (class or name) plus an optional per-case condition predicate. No match falls through to an optional default_model.
  • Routing returns a shallow copy of the evaluator carrying the routed model, so shared instances are never mutated across concurrent workers. Evaluators without a model attribute (deterministic) pass through untouched.
  • A raising condition logs a warning and skips the rule instead of failing the evaluation.
  • model_router is intentionally excluded from Experiment.to_dict()/from_dict() since routing conditions are arbitrary callables.

Example

from strands_evals import Experiment, ModelRouter, RoutingRule

router = ModelRouter(rules=[
    RoutingRule(
        evaluator_types=[ToolSelectionAccuracyEvaluator, ToolParameterAccuracyEvaluator],
        model="us.anthropic.claude-haiku-4-5-20251001-v1:0",
    ),
    RoutingRule(
        evaluator_types=[GoalSuccessRateEvaluator],
        model="us.anthropic.claude-opus-4-1-20250805-v1:0",
        condition=lambda case: len(str(case.actual_trajectory or "")) > 10_000,
    ),
])

experiment = Experiment(cases=cases, evaluators=evaluators, model_router=router)

Testing

  • 12 new tests: rule matching by class/name, condition gating, condition-exception safety, first-match-wins, default-model fallback, explicit-model preservation, no-mutation of shared instances, deterministic-evaluator passthrough, and two end-to-end Experiment integration tests asserting the judge agent is constructed with the routed model (and unchanged without a router).
  • Full evaluator + experiment unit suites pass (454 tests); ruff check and ruff format clean.

Checklist

  • Non-breaking change (opt-in)
  • Unit + integration tests added
  • Conventional commit
  • Lint/format clean

Adds ModelRouter and RoutingRule, plus an optional model_router parameter
on Experiment, so evaluation cost/accuracy can be matched to evaluator
complexity: fast models for structural/rubric checks, strong models for
nuanced judgment.

Semantics:
- Opt-in: no model_router means behavior is unchanged.
- Explicit wins: an evaluator constructed with model=... is never re-routed.
- First matching RoutingRule wins; rules match on evaluator type (class or
  name) and an optional per-case condition (e.g., trace length) enabling
  complexity-based routing.
- Routing returns a shallow copy of the evaluator with the routed model so
  shared instances are never mutated across concurrent workers; evaluators
  without a model attribute (deterministic) pass through untouched.
- A raising condition logs a warning and skips the rule instead of failing
  the evaluation.

model_router is intentionally excluded from Experiment.to_dict()/from_dict()
since routing conditions are arbitrary callables.
@pdebjyot
pdebjyot requested a review from a team as a code owner July 23, 2026 16:04
@pdebjyot
pdebjyot requested a review from arielnabavian July 23, 2026 16:04
@github-actions github-actions Bot added area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Model routing for evaluators — match model tier to evaluation complexity

1 participant