docs: add chaos testing and resilience evaluator doc pages and example scripts

[ybdarrenwang]

Description

Adds documentation pages and example scripts for the chaos testing module and three new resilience evaluators in Strands Evals.

Doc pages:

• Chaos Testing page (chaos_testing.mdx): Overview and guide for the chaos testing framework, covering ChaosPlugin, ChaosCase, ChaosExperiment, chaos effects (pre-hook and post-hook), ChaosCase.expand() for Cartesian product generation, integration with ToolSimulator, resilience evaluators, advanced patterns (agent comparison, degradation sweep, multi-turn with user simulator), and best practices
• FailureCommunicationEvaluator page: Documents the evaluator that assesses how well an agent communicates tool failures to users, using a five-level scoring rubric (No Communication → Excellent Communication)
• PartialCompletionEvaluator page: Documents the evaluator that scores what fraction of a user's goal was achieved despite failures, returning a continuous 0.0–1.0 score
• RecoveryStrategyEvaluator page: Documents the evaluator that scores the quality of an agent's recovery actions (retries, fallbacks, tool switching) when tools fail

Example scripts:

• chaos_testing.py: Demonstrates end-to-end chaos testing with named effect maps, ChaosCase.expand(), ChaosPlugin, ToolSimulator, and GoalSuccessRateEvaluator
• chaos_failure_communication_evaluator.py: Evaluating agent failure communication under timeout and network errors
• chaos_partial_completion_evaluator.py: Measuring partial task completion when tools return corrupted or failed responses
• chaos_recovery_strategy_evaluator.py: Assessing agent recovery strategies (tool pivoting, retry discipline)

Also updates:

• Evaluators index (index.mdx): Adds a new "Resilience Evaluators" subsection
• navigation.yml: Wires the new chaos testing and evaluator pages into the evals-sdk sidebar
• Fixes inconsistent import paths: all effect classes now import from strands_evals.chaos (the canonical re-export path) consistently across all four scripts

Related Issues

strands-agents/evals#114
strands-agents/docs#836 (original PR, repo archived)
#2724 (PR for doc pages only; combined)

Documentation PR

Type of Change

Documentation update

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

• I ran npm run dev

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Issue: These four example scripts aren't referenced by any documentation page or navigation entry. Every existing example under site/docs/examples/evals-sdk/ (e.g. goal_success_rate_evaluator.py, trajectory_evaluator.py) has a companion .mdx page in src/content/docs/user-guide/evals-sdk/evaluators/ that links to it with a "A complete example can be found [here]" reference. As-is, the chaos examples are undiscoverable from the docs site. The PR checklist marks "I have updated the documentation accordingly," but no doc page or nav entry was added.

Suggestion: Add at least one .mdx page (e.g. a "Chaos Testing" / resilience-evaluators page) that introduces the chaos module and links to these scripts, and wire it into the evals-sdk navigation. Note that the existing example links point to github.com/strands-agents/docs/blob/main/... (the archived repo) — any new page should reference the harness-sdk paths instead.

[github-actions]

Assessment: Comment

Solid set of chaos-testing examples — clear scenarios, good system prompts modeling realistic failure handling, and all four scripts compile cleanly. The earlier feedback on root-level imports and EvaluationReport.flatten has been addressed. Two items remain worth resolving before merge.

Review Categories

• Documentation integration: The new examples are not linked from any .mdx page or nav entry, unlike every other example in this directory — they're currently undiscoverable from the docs site.
• Consistency: Effect-class import paths vary within and across the four files (root strands_evals.chaos vs. strands_evals.chaos.effects); worth standardizing since examples model canonical usage.

Nice work demonstrating pre-hook, post-hook, and compound chaos with the ChaosCase.expand Cartesian-product pattern — the baseline-vs-injected comparison is a great teaching example.

[github-actions]

Assessment: Approve

Reviewed the latest commit (422c752, "address doc mismatch with implementation"). These are all accuracy fixes that bring the docs in line with the actual implementation, and they're correct:

What changed (6 files, +38/-39)

• Evaluation level corrected: SESSION_LEVEL → TRACE_LEVEL across all three resilience evaluator pages and the evaluators index. Verified the remaining SESSION_LEVEL references belong to pre-existing evaluators (Trajectory, Interactions, GoalSuccessRate) and are correctly untouched.
• Rubric labels aligned: shortened to Failure/Poor/Acceptable/Good/Excellent in the scoring tables, the "Evaluation:" example lines, and the output label examples.
• New constraint documented: clear note that each tool may have only one effect per ChaosCase (multiple effects raise ValueError), with helpful context on pre-hook mutual exclusivity and a possible future relaxation.
• Import consolidation and small example fixes (search_timeout → weather_timeout to match the get_weather sample; corrected the partial-completion case-2 comment).

Minor non-blocking nit: the scenario section headings (e.g., ### Scenario 1: Excellent Communication) still carry the old long-form label suffixes. They read fine as titles, so this is purely optional — no need to change.

Everything checks out and the docs now match the implementation. Still good to ship. 🚀