feat(tools): TraceIndex — progressive trace disclosure for judge-based evaluators

[pdebjyot]

feat(tools): TraceIndex — progressive trace disclosure for judge-based evaluators

Branch: feat/trace-index
Builds on: #324 (custom tools= on OutputEvaluator / TrajectoryEvaluator)
Addresses: #342

Problem

When the trajectory handed to a judge-based evaluator is larger than the judge
model's context window, the judge call raises ContextWindowOverflowException.
Experiment._run_evaluator catches it under error isolation and records the case
as score: 0, test_pass: False — indistinguishable from a genuine quality
failure. A correct agent response gets a false-negative failing score purely
because its trace was too big for the judge to read.

This isn't a theoretical edge. We observe it on real production agent traces,
where routine multi-step sessions serialize past a 200K-token judge window and
the largest run into the millions of tokens — so under a default Sonnet-class
judge, correct agents are being silently scored as failures today. The problem
is stack-agnostic: we reproduced identical overflow behavior across
strands-evals, DeepEval, and a Langfuse-style managed judge on a shared Bedrock
judge model (three distinct error signatures, same trace-size cliff).

The measured evidence below is from a synthetic, deterministic, offline
benchmark — no real data — so it's independently reproducible.

What this PR adds

TraceIndex — an in-memory index over a Session that lets a judge read a large
trace without inlining it. It is an established pattern: MLflow's
Agent-as-a-Judge trace scorers hand the judge ListSpans / GetSpan /
SearchTraceRegex, and Zhuge et al.'s Agent-as-a-Judge
(arXiv:2410.10934) uses retrieve/read/
locate modules to pull only the relevant segments.

from strands_evals.evaluators import OutputEvaluator
from strands_evals.tools.trace_index import TraceIndex

index = TraceIndex(session)  # session: a Session from any provider/mapper

evaluator = OutputEvaluator(
    rubric="Every factual claim must be supported by tool-result evidence in the trace.",
    tools=index.tools,  # list_spans, get_span, search_spans
)

# Compact overview in the prompt instead of the full trajectory:
output = f"{agent_answer}\n\n<TraceOverview>\n{index.overview()}\n</TraceOverview>"
evaluator.evaluate(EvaluationData(input="(from trace)", actual_output=output))

• overview() — one compact line per span (index, type, tool name, sizes,
  preview). Always fits the judge context; substituted for the full trajectory.
• list_spans / get_span / search_spans — discovery tools the judge
  calls on demand (mirrors the MLflow triad). get_span pages oversized spans
  via max_read_chars (default 8000) + offset, so no single tool return can
  itself overflow the judge.
• Backend-agnostic — consumes any Session a provider/mapper produces.

Two disclosure strategies compose from these pieces: index (substitute
overview() into the prompt — portable, works with any judge) and explore
(index + discovery tools via #324's tools= — Strands-native, best accuracy).

Evidence

Cross-framework matrix (1344 cells). 200-trace labeled corpus × 3 frameworks
(strands-evals, DeepEval, Langfuse-style) × 4 metrics (groundedness, accuracy,
trajectory, tool_use) × inline/index/explore, all bound to one shared Bedrock
judge. Each metric scored as a binary classifier against planted ground truth
(overflow ⇒ wrong prediction):

• The overflow cliff is stack-agnostic and lands at the same trace size for
  all three frameworks.
• index and explore hold accuracy on traces that fit and recover it on
  traces that overflow (inline is unusable in the overflow bucket; index/explore
  score correctly).
• The judge-side explore tools beat a bare index when the decisive fact is
  buried in a large tool result: on wrong_tool traces the deciding refund
  amount sits inside a large search result that overview() elides, so the index
  judge false-fails groundedness (0.83); get_span/search_spans let the judge
  retrieve it → groundedness 1.00.
• No degradation on traces that fit: index/explore ≈ inline everywhere inline
  still works.

Ground-truth A/B (grounded / fabricated claims, evidence buried mid-trace):
the index-equipped judge separates grounded from fabricated where the inline
judge overflows — captured as the integ test test_judge_reliability_inline_vs_explore.

Known limitation (called out honestly)

TrajectoryEvaluator inlines actual_trajectory unconditionally
(case_prompt_template.py:51), so the trajectory metric still overflows on large
traces even with the index — the substitution only reaches evaluators that route
through the caller-controlled actual_output (the output-family metrics, fixed
today). Making the trajectory metric disclosure-aware needs a template change and
is proposed as a follow-up; this PR does not change that path.

Separately, this PR adds the capability but does not change
_run_evaluator's error handling — distinguishing harness overflow from a
quality score: 0 is tracked in #342 as an independent change.

Testing

• 34 unit tests (overview formatting; list/get/search behavior; offset paging on
  oversized spans; cross-pattern; end-to-end through OutputEvaluator with
  tools=index.tools) — all pass.
• 1 integ A/B test (skips without live Bedrock credentials).
• ruff check / ruff format clean.

Checklist

• Backend-agnostic, additive (no change to existing evaluator behavior)
• Unit + integration tests added
• Conventional commits
• Lint/format clean
• Builds on merged feat(evaluators): allow custom tools on judge-based evaluators (Trajectory, Output, Multimodal) #324; known limitation documented, not hidden

[mehtarac]

/strands review