-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Simmy docs #1883
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Simmy docs #1883
Changes from 35 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
664e4bb
expose simmy apis
vany0114 3dd5703
add missing docs
vany0114 928d250
Merge branch 'main' into simmy-expose-apis
vany0114 6c1eb6f
WIP:
vany0114 202bddc
WIP:
vany0114 5331538
Merge branch 'main' into simmy-docs
vany0114 6db206f
add latency monkey docs
vany0114 a60a1e0
add latency docs
vany0114 4b0a1ac
add outcome docs
vany0114 6bcbd52
Merge branch 'main' into simmy-docs
vany0114 d3ddc4b
fixes docs styling issues
vany0114 11b69e1
more docs styling issues
vany0114 1aaf08a
Merge branch 'App-vNext:main' into simmy-docs
peter-csala ceb9992
Move code samples to snippets project
peter-csala 966bb95
Fix spellcheck issues
peter-csala ebf3fa5
Format markdown tables
peter-csala d8adb49
Apply suggested changes
peter-csala ae445be
Fix a couple of docfx issues
peter-csala 722bca1
Use unicode emojis to describe scenarios
peter-csala 38b7ce8
Fix docfx link issue
peter-csala 59b4cca
Fix build by using another overload of AddChaosXYZ
peter-csala f15cb64
Merge branch 'App-vNext:main' into simmy-docs
peter-csala f33a29f
Apply suggested changes
peter-csala fe9a973
Rename logo
peter-csala 46b4fe5
Apply suggestions from code review
peter-csala edf0d9a
Update docs/chaos/index.md
peter-csala 7269a66
Fix chaos strategy registration order
peter-csala fa7805d
Apply suggested changes to Result chaos
peter-csala c34305f
Use InternalServerError for status code
peter-csala e00a12d
Remove default options where it does not make sense
peter-csala 93eed65
Fix spelling
peter-csala 1bc64c3
Merge branch 'main' into simmy-docs
peter-csala 1dcc052
Delete simmy logos
peter-csala 5d78a62
Apply suggested changes
peter-csala cfbca1a
Replace Fault to FaultGenerator
peter-csala 3239718
Update docs/chaos/index.md
peter-csala 58b4eaf
Prefer chaos over monkey wherever possible
peter-csala a2f0986
Rearrange Table of Contents
peter-csala File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| # Behavior monkey strategy | ||
|
|
||
| ## About | ||
|
|
||
| - **Options**: [`BehaviorStrategyOptions`](xref:Polly.Simmy.Behavior.BehaviorStrategyOptions) | ||
| - **Extensions**: `AddChaosBehavior` | ||
| - **Strategy Type**: Proactive | ||
|
|
||
| --- | ||
|
|
||
| The behavior chaos strategy is designed to inject custom behaviors into system operations right before such an operation is invoked. This strategy is flexible, allowing users to define specific behaviors such as altering the input, simulating resource exhaustion, putting the system in a given state before the actual operation is called, or other operational variations to simulate real-world scenarios. | ||
|
|
||
| ## Usage | ||
|
|
||
| <!-- snippet: chaos-behavior-usage --> | ||
| ```cs | ||
| // To use a custom function to generate the behavior to inject. | ||
| var optionsWithBehaviorGenerator = new BehaviorStrategyOptions | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.05 | ||
| }; | ||
|
|
||
| // To get notifications when a behavior is injected | ||
| var optionsOnBehaviorInjected = new BehaviorStrategyOptions | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.05, | ||
| OnBehaviorInjected = static args => | ||
| { | ||
| Console.WriteLine("OnBehaviorInjected, Operation: {0}.", args.Context.OperationKey); | ||
| return default; | ||
| } | ||
| }; | ||
|
|
||
| // Add a behavior strategy with a BehaviorStrategyOptions instance to the pipeline | ||
| new ResiliencePipelineBuilder().AddChaosBehavior(optionsWithBehaviorGenerator); | ||
| new ResiliencePipelineBuilder<HttpResponseMessage>().AddChaosBehavior(optionsOnBehaviorInjected); | ||
|
|
||
| // There are also a handy overload to inject the chaos easily. | ||
| new ResiliencePipelineBuilder().AddChaosBehavior(0.05, RestartRedisVM); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| Example execution: | ||
|
|
||
| <!-- snippet: chaos-behavior-execution --> | ||
| ```cs | ||
| var pipeline = new ResiliencePipelineBuilder() | ||
| .AddRetry(new RetryStrategyOptions | ||
| { | ||
| ShouldHandle = new PredicateBuilder().Handle<RedisConnectionException>(), | ||
| BackoffType = DelayBackoffType.Exponential, | ||
| UseJitter = true, // Adds a random factor to the delay | ||
| MaxRetryAttempts = 4, | ||
| Delay = TimeSpan.FromSeconds(3), | ||
| }) | ||
| .AddChaosBehavior(new BehaviorStrategyOptions // Monkey strategies are usually placed as the last ones in the pipeline | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.05 | ||
| }) | ||
| .Build(); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| ## Defaults | ||
|
|
||
| | Property | Default Value | Description | | ||
| |----------------------|---------------|------------------------------------------------| | ||
| | `OnBehaviorInjected` | `null` | Action executed when the behavior is injected. | | ||
| | `BehaviorAction` | `null` | Custom behavior to be injected. | | ||
|
|
||
| ## Diagrams | ||
|
|
||
| ### Normal 🐵 sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant B as Behavior | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>B: Calls ExecuteCore | ||
| activate B | ||
| B-->>B: Determines Injection<br/>Decision: 🐵 | ||
| deactivate B | ||
| B->>+D: Invokes | ||
| D->>-B: Returns result | ||
| B->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ### Chaos 🙈 sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant B as Behavior | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>B: Calls ExecuteCore | ||
| activate B | ||
| B-->>B: Determines Injection<br/>Decision: 🙈 | ||
| B-->>B: Injects Behavior | ||
| deactivate B | ||
| B->>+D: Invokes | ||
| D->>-B: Returns result | ||
| B->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ## Anti-patterns | ||
|
|
||
| ### Injecting delay | ||
|
|
||
| ❌ DON'T | ||
|
|
||
| Use behavior strategies to inject delays. | ||
|
|
||
| ✅ DO | ||
|
|
||
| Use the latency monkey instead as the [`LatencyChaosStrategy`](latency.md) already correctly handles synchronous/asynchronous delay executions, cancellations, etc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| # Fault monkey strategy | ||
|
|
||
| ## About | ||
|
|
||
| - **Options**: [`FaultStrategyOptions`](xref:Polly.Simmy.Fault.FaultStrategyOptions) | ||
| - **Extensions**: `AddChaosFault` | ||
| - **Strategy Type**: Proactive | ||
|
|
||
| --- | ||
|
|
||
| The fault chaos strategy is designed to introduce faults (exceptions) into the system, simulating real-world scenarios where operations might fail unexpectedly. It is configurable to inject specific types of exceptions or use custom logic to generate faults dynamically. | ||
|
|
||
| ## Usage | ||
|
|
||
| <!-- snippet: chaos-fault-usage --> | ||
| ```cs | ||
| // 10% of invocations will be randomly affected. | ||
| var optionsBasic = new FaultStrategyOptions | ||
| { | ||
| FaultGenerator = static args => new ValueTask<Exception?>(new InvalidOperationException("Dummy exception")), | ||
| Enabled = true, | ||
| InjectionRate = 0.1 | ||
| }; | ||
|
|
||
| // To use a custom function to generate the fault to inject. | ||
| var optionsWithFaultGenerator = new FaultStrategyOptions | ||
| { | ||
| FaultGenerator = static args => | ||
| { | ||
| Exception? exception = args.Context.OperationKey switch | ||
| { | ||
| "DataLayer" => new TimeoutException(), | ||
| "ApplicationLayer" => new InvalidOperationException(), | ||
| _ => null // When the fault generator returns null the strategy won't inject any fault and it will just invoke the user's callback | ||
| }; | ||
|
|
||
| return new ValueTask<Exception?>(exception); | ||
| }, | ||
| Enabled = true, | ||
| InjectionRate = 0.1 | ||
| }; | ||
|
|
||
| // To get notifications when a fault is injected | ||
| var optionsOnFaultInjected = new FaultStrategyOptions | ||
| { | ||
| FaultGenerator = static args => new ValueTask<Exception?>(new InvalidOperationException("Dummy exception")), | ||
| Enabled = true, | ||
| InjectionRate = 0.1, | ||
| OnFaultInjected = static args => | ||
| { | ||
| Console.WriteLine("OnFaultInjected, Exception: {0}, Operation: {1}.", args.Fault.Message, args.Context.OperationKey); | ||
| return default; | ||
| } | ||
| }; | ||
|
|
||
| // Add a fault strategy with a FaultStrategyOptions instance to the pipeline | ||
| new ResiliencePipelineBuilder().AddChaosFault(optionsBasic); | ||
| new ResiliencePipelineBuilder<HttpResponseMessage>().AddChaosFault(optionsWithFaultGenerator); | ||
|
|
||
| // There are also a couple of handy overloads to inject the chaos easily. | ||
| new ResiliencePipelineBuilder().AddChaosFault(0.1, () => new InvalidOperationException("Dummy exception")); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| Example execution: | ||
|
|
||
| <!-- snippet: chaos-fault-execution --> | ||
| ```cs | ||
| var pipeline = new ResiliencePipelineBuilder() | ||
| .AddRetry(new RetryStrategyOptions | ||
| { | ||
| ShouldHandle = new PredicateBuilder().Handle<InvalidOperationException>(), | ||
| BackoffType = DelayBackoffType.Exponential, | ||
| UseJitter = true, // Adds a random factor to the delay | ||
| MaxRetryAttempts = 4, | ||
| Delay = TimeSpan.FromSeconds(3), | ||
| }) | ||
| .AddChaosFault(new FaultStrategyOptions // Monkey strategies are usually placed as the last ones in the pipeline | ||
| { | ||
| FaultGenerator = static args => new ValueTask<Exception?>(new InvalidOperationException("Dummy exception")), | ||
| Enabled = true, | ||
| InjectionRate = 0.1 | ||
| }) | ||
| .Build(); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| ## Defaults | ||
|
|
||
| | Property | Default Value | Description | | ||
| |-------------------|---------------|------------------------------------------------------| | ||
| | `OnFaultInjected` | `null` | Action executed when the fault is injected. | | ||
| | `FaultGenerator` | `null` | Generates the fault to inject for a given execution. | | ||
|
|
||
| ## Diagrams | ||
|
|
||
| ### Normal 🐵 sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant F as Fault | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>F: Calls ExecuteCore | ||
| activate F | ||
| F-->>F: Determines Injection<br/>Decision: 🐵 | ||
| deactivate F | ||
| F->>+D: Invokes | ||
| D->>-F: Returns result | ||
| F->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ### Chaos 🙈 sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant F as Fault | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>F: Calls ExecuteCore | ||
| activate F | ||
| F-->>F: Determines Injection<br/>Decision: 🙈 | ||
| F-->>F: Injects Fault | ||
| deactivate F | ||
| Note over D: The user's Callback is not invoked<br/>when a fault is injected | ||
| F->>P: Throws injected Fault | ||
| P->>C: Propagates Exception | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Chaos engineering with Simmy | ||
|
|
||
| [Simmy][simmy] is a major new addition to Polly library, adding a chaos engineering and fault-injection dimension to Polly, through the provision of strategies to selectively inject faults, latency, custom behavior or fake results. | ||
|
|
||
|  | ||
|
|
||
| ## Motivation | ||
|
|
||
| There are a lot of questions when it comes to chaos engineering and making sure that a system is actually ready to face the worst possible scenarios: | ||
|
|
||
| * Is my system resilient enough? | ||
| * Am I handling the right exceptions/scenarios? | ||
| * How will my system behave if X happens? | ||
| * How can I test without waiting for a handled (or even unhandled) exception to happen in my production environment? | ||
|
|
||
| Using Polly helps introduce resilience to a project, but we don't want to have to wait for expected or unexpected failures to test it out. A resilience could be wrongly implemented; testing the scenarios is not straightforward; and mocking failure of some dependencies (for example a cloud SaaS or PaaS service) is not always straightforward. | ||
|
|
||
| ### What is needed to simulate chaotic scenarios? | ||
|
|
||
| * A way to simulate failures of dependencies (any service dependency for example). | ||
| * Define when to fail based on some external factors - maybe global configuration or some rule. | ||
| * A way to revert easily, to control the blast radius. | ||
| * To be production grade, to run this in a production or near-production system with automation. | ||
|
|
||
| ## Chaos strategies (a.k.a Monkey strategies) | ||
|
|
||
| Chaos strategies (or Monkey strategies as we call them) are in essence a [Resilience strategy](../strategies/index.md#built-in-strategies), which means, as a *Resilience Strategy* is the minimum unit of resilience for Polly, a *Monkey Strategy* is the minimum unit of chaos for Simmy. | ||
|
|
||
| ### Built-in strategies | ||
|
|
||
| | Strategy | Reactive | What does the strategy do? | | ||
| |-------------------------|----------|----------------------------------------------------------------------| | ||
| | [Fault](fault.md) | No | Injects exceptions in your system. | | ||
| | [Result](result.md) | Yes | Substitute results to fake outcomes in your system. | | ||
| | [Latency](latency.md) | No | Injects latency into executions before the calls are made. | | ||
| | [Behavior](behavior.md) | No | Allows you to inject *any* extra behaviour, before a call is placed. | | ||
|
|
||
| ## Usage | ||
|
|
||
| It is usual to place the monkey strategy as the last strategy in the resilience pipeline. By placing the monkey strategies as last, they subvert the usual outbound call at the last minute, substituting their fault or adding extra latency, etc. The existing resilience strategies - further out in the `ResiliencePipeline` - still apply, so you can test how the Polly resilience strategies you have configured handle the chaos/faults injected by Simmy. | ||
|
|
||
| ## Common options across strategies | ||
|
|
||
| All the strategies' options implement the [`MonkeyStrategyOptions`](xref:Polly.Simmy.MonkeyStrategyOptions) class as it contains the basic configuration for every monkey strategy. | ||
|
|
||
| | Property | Default Value | Description | | ||
| |--------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `InjectionRate` | 0.001 ms | A decimal between 0 and 1 inclusive. The strategy will inject the chaos, randomly, that proportion of the time, e.g.: if 0.2, twenty percent of calls will be randomly affected; if 0.01, one percent of calls; if 1, all calls. | | ||
| | `InjectionRateGenerator` | `null` | Generates the injection rate for a given execution, which the value should be between [0, 1] (inclusive). | | ||
| | `Enabled` | `false` | Determines whether the strategy is enabled or not. | | ||
| | `EnabledGenerator` | `null` | the enable generator that indicates whether or not the chaos strategy is enabled for a given execution. | | ||
|
|
||
| [simmy]: https://github.com/Polly-Contrib/Simmy | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.