-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
[DO NOT MERGE] To verfiy build for Simmy docs PR #1894
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
664e4bb
expose simmy apis
vany0114 3dd5703
add missing docs
vany0114 928d250
Merge branch 'main' into simmy-expose-apis
vany0114 6c1eb6f
WIP:
vany0114 202bddc
WIP:
vany0114 5331538
Merge branch 'main' into simmy-docs
vany0114 6db206f
add latency monkey docs
vany0114 a60a1e0
add latency docs
vany0114 4b0a1ac
add outcome docs
vany0114 6bcbd52
Merge branch 'main' into simmy-docs
vany0114 d3ddc4b
fixes docs styling issues
vany0114 11b69e1
more docs styling issues
vany0114 a152434
Move code-snippets for Behavior and Fault
54fd269
Move code-snippets for Latency and Result
583d9d2
Merge branch 'App-vNext:main' into simmy-docs
peter-csala 680dd68
Fix spellcheck issues
1069fe7
Apply suggested changes to Behavior
55ccd95
Apply suggested changes to Fault
48fb694
Apply suggested changes to index
10aa7ec
Apply suggested changes to latency
ecbdd4b
Apply suggested changes to result
5d3fee5
Update Behavior code samples to align the code style
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| # Behavior monkey strategy | ||
|
|
||
| ## About | ||
|
|
||
| - **Options**: [`BehaviorStrategyOptions`](xref:Polly.Simmy.Behavior.BehaviorStrategyOptions) | ||
| - **Extensions**: `AddChaosBehavior` | ||
| - **Strategy Type**: Proactive | ||
|
|
||
| --- | ||
|
|
||
| The behavior chaos strategy is designed to inject custom behaviors into system operations right before such an operation is invoked. This strategy is flexible, allowing users to define specific behaviors such as altering the input, simulating resource exhaustion, putting the system in a given state before the actual operation is called, or other operational variations to simulate real-world scenarios. | ||
|
|
||
| ## Usage | ||
|
|
||
| <!-- snippet: chaos-behavior-usage --> | ||
| ```cs | ||
| // Behavior using the default options. | ||
| // See https://www.pollydocs.org/chaos/behavior#defaults for defaults. | ||
| var optionsDefault = new BehaviorStrategyOptions(); | ||
|
|
||
| // To use a custom function to generate the behavior to inject. | ||
| var optionsWithBehaviorGenerator = new BehaviorStrategyOptions | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.6 | ||
| }; | ||
|
|
||
| // To get notifications when a behavior is injected | ||
| var optionsOnBehaviorInjected = new BehaviorStrategyOptions | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.6, | ||
| OnBehaviorInjected = static args => | ||
| { | ||
| Console.WriteLine("OnBehaviorInjected, Operation: {0}.", args.Context.OperationKey); | ||
| return default; | ||
| } | ||
| }; | ||
|
|
||
| // Add a behavior strategy with a BehaviorStrategyOptions{<TResult>} instance to the pipeline | ||
| new ResiliencePipelineBuilder().AddChaosBehavior(optionsDefault); | ||
| new ResiliencePipelineBuilder<HttpStatusCode>().AddChaosBehavior(optionsWithBehaviorGenerator); | ||
|
|
||
| // There are also a handy overload to inject the chaos easily. | ||
| new ResiliencePipelineBuilder().AddChaosBehavior(0.6, RestartRedisVM); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| Example execution: | ||
|
|
||
| <!-- snippet: chaos-behavior-execution --> | ||
| ```cs | ||
| var pipeline = new ResiliencePipelineBuilder() | ||
| .AddChaosBehavior(new BehaviorStrategyOptions // Monkey strategies are usually placed innermost in the pipelines | ||
| { | ||
| BehaviorAction = static args => RestartRedisVM(), | ||
| Enabled = true, | ||
| InjectionRate = 0.6 | ||
| }) | ||
| .AddRetry(new RetryStrategyOptions | ||
| { | ||
| ShouldHandle = new PredicateBuilder().Handle<RedisConnectionException>(), | ||
| BackoffType = DelayBackoffType.Exponential, | ||
| UseJitter = true, // Adds a random factor to the delay | ||
| MaxRetryAttempts = 4, | ||
| Delay = TimeSpan.FromSeconds(3), | ||
| }) | ||
| .Build(); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| ## Defaults | ||
|
|
||
| | Property | Default Value | Description | | ||
| |----------------------|---------------|------------------------------------------------| | ||
| | `OnBehaviorInjected` | `null` | Action executed when the behavior is injected. | | ||
| | `BehaviorAction` | `null` | Custom behavior to be injected. | | ||
|
|
||
| ## Diagrams | ||
|
|
||
| ### Happy path sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant B as Behavior | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>B: Calls ExecuteCore | ||
| activate B | ||
| B-->>B: Determines Behavior Injection | ||
| deactivate B | ||
| B->>+D: Invokes | ||
| D->>-B: Returns result | ||
| B->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ### Unhappy path sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant B as Behavior | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>B: Calls ExecuteCore | ||
| activate B | ||
| B-->>B: Determines Behavior Injection | ||
| B-->>B: Inject Behavior | ||
| deactivate B | ||
| B->>+D: Invokes | ||
| D->>-B: Returns result | ||
| B->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ## Anti-patterns | ||
|
|
||
| ❌ Do not use behavior strategies to inject delays, use the latency monkey instead as the [`LatencyChaosStrategy`](latency.md) already correctly handles synchronous/asynchronous delay executions, cancellations, etc. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| # Fault monkey strategy | ||
|
|
||
| ## About | ||
|
|
||
| - **Options**: [`FaultStrategyOptions`](xref:Polly.Simmy.Fault.FaultStrategyOptions) | ||
| - **Extensions**: `AddChaosFault` | ||
| - **Strategy Type**: Proactive | ||
|
|
||
| --- | ||
|
|
||
| The fault chaos strategy is designed to introduce faults (exceptions) into the system, simulating real-world scenarios where operations might fail unexpectedly. It is configurable to inject specific types of exceptions or use custom logic to generate faults dynamically. | ||
|
|
||
| ## Usage | ||
|
|
||
| <!-- snippet: chaos-fault-usage --> | ||
| ```cs | ||
| // Fault using the default options. | ||
| // See https://www.pollydocs.org/chaos/fault#defaults for defaults. | ||
| var optionsDefault = new FaultStrategyOptions(); | ||
|
|
||
| // 60% of invocations will be randomly affected. | ||
| var basicOptions = new FaultStrategyOptions | ||
| { | ||
| Fault = new InvalidOperationException("Dummy exception"), | ||
| Enabled = true, | ||
| InjectionRate = 0.6 | ||
| }; | ||
|
|
||
| // To use a custom function to generate the fault to inject. | ||
| var optionsWithFaultGenerator = new FaultStrategyOptions | ||
| { | ||
| FaultGenerator = static args => | ||
| { | ||
| Exception? exception = args.Context.OperationKey switch | ||
| { | ||
| "DataLayer" => new TimeoutException(), | ||
| "ApplicationLayer" => new InvalidOperationException(), | ||
| _ => null // When the fault generator returns null the strategy won't inject any fault and it will just invoke the user's callback | ||
| }; | ||
|
|
||
| return new ValueTask<Exception?>(exception); | ||
| }, | ||
| Enabled = true, | ||
| InjectionRate = 0.6 | ||
| }; | ||
|
|
||
| // To get notifications when a fault is injected | ||
| var optionsOnFaultInjected = new FaultStrategyOptions | ||
| { | ||
| Fault = new InvalidOperationException("Dummy exception"), | ||
| Enabled = true, | ||
| InjectionRate = 0.6, | ||
| OnFaultInjected = static args => | ||
| { | ||
| Console.WriteLine("OnFaultInjected, Exception: {0}, Operation: {1}.", args.Fault.Message, args.Context.OperationKey); | ||
| return default; | ||
| } | ||
| }; | ||
|
|
||
| // Add a fault strategy with a FaultStrategyOptions{<TResult>} instance to the pipeline | ||
| new ResiliencePipelineBuilder().AddChaosFault(optionsDefault); | ||
| new ResiliencePipelineBuilder<HttpStatusCode>().AddChaosFault(optionsWithFaultGenerator); | ||
|
|
||
| // There are also a couple of handy overloads to inject the chaos easily. | ||
| new ResiliencePipelineBuilder().AddChaosFault(0.6, new InvalidOperationException("Dummy exception")); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| Example execution: | ||
|
|
||
| <!-- snippet: chaos-fault-execution --> | ||
| ```cs | ||
| var pipeline = new ResiliencePipelineBuilder() | ||
| .AddChaosFault(new FaultStrategyOptions // Monkey strategies are usually placed innermost in the pipelines | ||
| { | ||
| Fault = new InvalidOperationException("Dummy exception"), | ||
| Enabled = true, | ||
| InjectionRate = 0.6 | ||
| }) | ||
| .AddRetry(new RetryStrategyOptions | ||
| { | ||
| ShouldHandle = new PredicateBuilder().Handle<InvalidOperationException>(), | ||
| BackoffType = DelayBackoffType.Exponential, | ||
| UseJitter = true, // Adds a random factor to the delay | ||
| MaxRetryAttempts = 4, | ||
| Delay = TimeSpan.FromSeconds(3), | ||
| }) | ||
| .Build(); | ||
| ``` | ||
| <!-- endSnippet --> | ||
|
|
||
| ## Defaults | ||
|
|
||
| | Property | Default Value | Description | | ||
| |-------------------|---------------|------------------------------------------------------| | ||
| | `OnFaultInjected` | `null` | Action executed when the fault is injected. | | ||
| | `FaultGenerator` | `null` | Generates the fault to inject for a given execution. | | ||
| | `Fault` | `null` | The fault to inject. | | ||
|
|
||
| ## Diagrams | ||
|
|
||
| ### Happy path sequence diagram | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might make sense to explain why fault injection did not happen. Or rename the |
||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant F as Fault | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>F: Calls ExecuteCore | ||
| activate F | ||
| F-->>F: Determines Fault Injection | ||
| deactivate F | ||
| F->>+D: Invokes | ||
| D->>-F: Returns result | ||
| F->>P: Returns result | ||
| P->>C: Returns result | ||
| ``` | ||
|
|
||
| ### Unhappy path sequence diagram | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor C as Caller | ||
| participant P as Pipeline | ||
| participant F as Fault | ||
| participant D as DecoratedUserCallback | ||
|
|
||
| C->>P: Calls ExecuteAsync | ||
| P->>F: Calls ExecuteCore | ||
| activate F | ||
| F-->>F: Determines Fault Injection | ||
| F-->>F: Inject Fault | ||
| deactivate F | ||
| Note over D: The user's Callback is not invoked when a fault is injected | ||
peter-csala marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| F->>P: Throws injected Fault | ||
| P->>C: Propagates Exception | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Chaos engineering with Simmy | ||
|
|
||
| [Simmy][simmy] is a major new companion project adding a chaos-engineering and fault-injection dimension to Polly, through the provision of policies to selectively inject faults, latency, custom behavior or fake results. | ||
peter-csala marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| <img src="../../logos/Simmy-Logo.png" alt="Simmy"/> | ||
|
|
||
| ## Motivation | ||
|
|
||
| There are a lot of questions when it comes to chaos-engineering and making sure that a system is actually ready to face the worst possible scenarios: | ||
|
|
||
| * Is my system resilient enough? | ||
| * Am I handling the right exceptions/scenarios? | ||
| * How will my system behave if X happens? | ||
| * How can I test without waiting for a handled (or even unhandled) exception to happen in my production environment? | ||
|
|
||
| Using Polly helps introduce resilience to a project, but we don't want to have to wait for expected or unexpected failures to test it out. A resilience could be wrongly implemented; testing the scenarios is not straightforward; and mocking failure of some dependencies (for example a cloud SaaS or PaaS service) is not always straightforward. | ||
|
|
||
| **What is needed, to simulate chaotic scenarios in a production environment?** | ||
peter-csala marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| * A way to simulate failures of dependencies (any service dependency for example). | ||
| * Define when to fail based on some external factors - maybe global configuration or some rule. | ||
| * A way to revert easily, to control the blast radius. | ||
| * To be production grade, to run this in a production or near-production system with automation. | ||
|
|
||
| ## Chaos strategies (a.k.a Monkey strategies) | ||
|
|
||
| Chaos strategies (or Monkey strategies as we call them) are in essence a [Resilience strategy](../strategies/index.md#built-in-strategies), which means, a *Resilience Strategy* is the minimum unit of resilience for Polly, a *Monkey Strategy* is the minimum unit of chaos for Simmy. | ||
|
|
||
| ### Built-in strategies | ||
|
|
||
| | Strategy | Reactive | What does the policy do? | | ||
peter-csala marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| |-------------------------|----------|----------------------------------------------------------------------| | ||
| | [Fault](fault.md) | No | Injects exceptions in your system. | | ||
| | [Result](result.md) | Yes | Substitute results to fake outcomes in your system. | | ||
| | [Latency](latency.md) | No | Injects latency into executions before the calls are made. | | ||
| | [Behavior](behavior.md) | No | Allows you to inject *any* extra behaviour, before a call is placed. | | ||
|
|
||
| ## Usage | ||
|
|
||
| It is usual to place the monkey strategy innermost in a Resilience Pipeline. By placing the monkey strategies innermost, they subvert the usual outbound call at the last minute, substituting their fault or adding extra latency, etc. The existing resilience strategies - further out in the `ResiliencePipeline` - still apply, so you can test how the Polly resilience strategies you have configured handle the chaos/faults injected by Simmy. | ||
|
|
||
| ## Common options across strategies | ||
|
|
||
| All the strategies' options implement the [`MonkeyStrategyOptions`](xref:Polly.Simmy.MonkeyStrategyOptions) class as it contains the basic configuration for every monkey strategy. | ||
|
|
||
| | Property | Default Value | Description | | ||
| |--------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `InjectionRate` | 0.001 ms | A decimal between 0 and 1 inclusive. The strategy will inject the chaos, randomly, that proportion of the time, e.g.: if 0.2, twenty percent of calls will be randomly affected; if 0.01, one percent of calls; if 1, all calls. | | ||
| | `InjectionRateGenerator` | `null` | Generates the injection rate for a given execution, which the value should be between [0, 1] (inclusive). | | ||
| | `Enabled` | `false` | Determines whether the strategy is enabled or not. | | ||
| | `EnabledGenerator` | `null` | the enable generator that indicates whether or not the chaos strategy is enabled for a given execution. | | ||
|
|
||
| [simmy]: https://github.com/Polly-Contrib/Simmy | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.