feat: add model-output corruption effects by venkatkrish543re · Pull Request #284 · strands-agents/evals

venkatkrish543re · 2026-06-24T21:57:54Z

Description

Adds model-output corruption to the existing ChaosPlugin via a guarded MessageAddedEvent hook, alongside the tool-chaos hooks.

Effects: FormatCorruption (malformed JSON, EmptyResponse), Hallucination (confabulation), Refusal (full refusal), plus composable success-framing.

The MessageAddedEvent callback is guarded to corrupt only final assistant responses (role==assistant, no toolUse blocks present) so destructive effects can't delete toolUse blocks mid-turn and break the agent loop. Includes 12 tests covering each effect and all guard conditions.

Related Issues

Builds on feat: add chaos testing module for fault injection #224 (tool-level chaos by @ybdarrenwang)

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

I ran hatch run prepare

Checklist

I have read the CONTRIBUTING document
I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
My change is focused and reasonably small; I have split unrelated work into separate PRs
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Adds model-output corruption to the existing ChaosPlugin via a guarded MessageAddedEvent hook, alongside the P0 tool-chaos hooks. Effects: FormatCorruption (malformed JSON, truncation, schema violation, empty, garbage), Hallucination (confabulation), Refusal (full refusal), plus optional success-framing. The MessageAddedEvent callback is guarded to corrupt only final assistant responses (role==assistant, no toolUse blocks present) so destructive effects can't delete toolUse blocks mid-turn and break the agent loop. Includes 13 tests covering each effect and all guard conditions.

…ntions Folded model effects into effects.py with ModelEffect base parallel to ToolEffect Replaced ModelOutputCorruptionType enum with class-per-effect + ModelEffectUnion Removed apply_rate; chaos always applies when configured Dropped speculative untested effects (TOXIC_CONTENT, truncation, schema violation, etc.) Moved perturbation data onto owning effect classes Deleted model_effects.py / model_types.py / model_utils.py

Resolve Darren's 5 review comments on PR strands-agents#284: - FullRefusal becomes a true pre-model effect using BeforeModelCallEvent.cancel (strands-agents >= 1.45.0) - All other model effects (EmptyResponse, Confabulation, MalformedJson, SuccessFraming) remain post-model via MessageAddedEvent - Move model_effects from ChaosPlugin to ChaosCase as a sibling field (flat list, no tool-name dimension) - Add before_model_invocation hook; rename message_added to after_model_invocation - Add pre-effects guard in post hook to prevent double-corruption in mixed pre+post cases - Delete _map_text_in_blocks, replace with _apply_text_to_blocks - Rewrite tests for case-based construction via ContextVar - Add pre-hook integration test and mixed-case test 14 tests pass. Lint, format, and mypy clean.

- FullRefusal becomes a true pre-model effect using BeforeModelCallEvent.cancel (strands-agents >= 1.45.0) - All other model effects (EmptyResponse, Confabulation, MalformedJson, SuccessFraming) remain post-model via MessageAddedEvent - Move model_effects from ChaosPlugin to ChaosCase as a sibling field (flat list, no tool-name dimension) - Add before_model_invocation hook; rename message_added to after_model_invocation - Add pre-effects guard in post hook to prevent double-corruption in mixed pre+post cases - Delete _map_text_in_blocks, replace with _apply_text_to_blocks - Rewrite tests for case-based construction via ContextVar - Add pre-hook integration test and mixed-case test 14 tests pass.

toolUse blocks (corrupts tool input JSON). Other post effects still skip toolUse messages. Guard 2 checks event.agent.tool_registry.dynamic_tools to identify structured-output toolUse blocks. Only those are corrupted by MalformedJson; ordinary mid-turn toolUse is left untouched. - Change 2: Move model_effects into keyed effects dict (effects={'model_effects': {'*': [...]}}) mirroring tool_effects. Remove flat sibling field. '*' wildcard resolves to all models. - Change 3: Rename _apply_to_blocks -> _apply_to_tool_blocks. Add _apply_to_model_blocks for model post effects. Delete the short _apply_text_to_blocks helper (folded into model blocks method). Added _apply_malformed_json_selective helper that skips toolUse blocks whose name is not in dynamic_tools. - Change 4: EmptyResponse becomes pre-hook using cancel=' ' (single space, truthy). Now two pre effects: FullRefusal and EmptyResponse. Test calibration: propagation assertion used (structured_output_model runs through normal event loop, fires MessageAddedEvent). Tests: plain-named toolUse asserted UNCHANGED; only SO toolUse corrupted; mixed message (regular + SO) selectively corrupts.

…l keys, semantic SO-tool detection

…ngle malform impl

…ed operations, trim public exports

venkatkrish543re had a problem deploying to manual-approval June 24, 2026 21:58 — with GitHub Actions Failure

github-actions Bot added area-chaos Chaos/fault injection: experiments, recovery strategy, partial completion, failure communication enhancement New feature or request labels Jun 24, 2026

ybdarrenwang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/strands_evals/chaos/model_types.py Outdated

ybdarrenwang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/strands_evals/chaos/model_effects.py Outdated

ybdarrenwang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/strands_evals/chaos/model_types.py Outdated

ybdarrenwang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/strands_evals/chaos/model_types.py Outdated

ybdarrenwang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/strands_evals/chaos/model_utils.py Outdated

venkatkrish543re had a problem deploying to manual-approval June 26, 2026 00:39 — with GitHub Actions Failure