Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ ndc
nuget
oss
pcl
paas
parallelize
pluralsight
pollydocs
Expand All @@ -47,6 +48,7 @@ rethrow
rethrows
retryable
reusability
saas
sdk
serializers
silverlight
Expand All @@ -60,6 +62,7 @@ telemetrylistener
testability
timingpolicy
ui
unhandled
uwp
waitandretry
wpf
Expand Down
6 changes: 0 additions & 6 deletions docs/advanced/simmy.md

This file was deleted.

126 changes: 126 additions & 0 deletions docs/chaos/behavior.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Behavior monkey strategy

## About

- **Options**: [`BehaviorStrategyOptions`](xref:Polly.Simmy.Behavior.BehaviorStrategyOptions)
- **Extensions**: `AddChaosBehavior`
- **Strategy Type**: Proactive

---

The behavior chaos strategy is designed to inject custom behaviors into system operations right before such an operation is invoked. This strategy is flexible, allowing users to define specific behaviors such as altering the input, simulating resource exhaustion, putting the system in a given state before the actual operation is called, or other operational variations to simulate real-world scenarios.

## Usage

<!-- snippet: chaos-behavior-usage -->
```cs
// Behavior using the default options.
// See https://www.pollydocs.org/chaos/behavior#defaults for defaults.
var optionsDefault = new BehaviorStrategyOptions();

// To use a custom function to generate the behavior to inject.
var optionsWithBehaviorGenerator = new BehaviorStrategyOptions
{
BehaviorAction = static args => RestartRedisVM(),
Enabled = true,
InjectionRate = 0.6
};

// To get notifications when a behavior is injected
var optionsOnBehaviorInjected = new BehaviorStrategyOptions
{
BehaviorAction = static args => RestartRedisVM(),
Enabled = true,
InjectionRate = 0.6,
OnBehaviorInjected = static args =>
{
Console.WriteLine("OnBehaviorInjected, Operation: {0}.", args.Context.OperationKey);
return default;
}
};

// Add a behavior strategy with a BehaviorStrategyOptions{<TResult>} instance to the pipeline
new ResiliencePipelineBuilder().AddChaosBehavior(optionsDefault);
new ResiliencePipelineBuilder<HttpStatusCode>().AddChaosBehavior(optionsWithBehaviorGenerator);

// There are also a handy overload to inject the chaos easily.
new ResiliencePipelineBuilder().AddChaosBehavior(0.6, RestartRedisVM);
```
<!-- endSnippet -->

Example execution:

<!-- snippet: chaos-behavior-execution -->
```cs
var pipeline = new ResiliencePipelineBuilder()
.AddChaosBehavior(new BehaviorStrategyOptions // Monkey strategies are usually placed innermost in the pipelines
{
BehaviorAction = static args => RestartRedisVM(),
Enabled = true,
InjectionRate = 0.6
})
.AddRetry(new RetryStrategyOptions
{
ShouldHandle = new PredicateBuilder().Handle<RedisConnectionException>(),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true, // Adds a random factor to the delay
MaxRetryAttempts = 4,
Delay = TimeSpan.FromSeconds(3),
})
.Build();
```
<!-- endSnippet -->

## Defaults

| Property | Default Value | Description |
|----------------------|---------------|------------------------------------------------|
| `OnBehaviorInjected` | `null` | Action executed when the behavior is injected. |
| `BehaviorAction` | `null` | Custom behavior to be injected. |

## Diagrams

### Happy path sequence diagram

```mermaid
sequenceDiagram
actor C as Caller
participant P as Pipeline
participant B as Behavior
participant D as DecoratedUserCallback

C->>P: Calls ExecuteAsync
P->>B: Calls ExecuteCore
activate B
B-->>B: Determines Behavior Injection
deactivate B
B->>+D: Invokes
D->>-B: Returns result
B->>P: Returns result
P->>C: Returns result
```

### Unhappy path sequence diagram

```mermaid
sequenceDiagram
actor C as Caller
participant P as Pipeline
participant B as Behavior
participant D as DecoratedUserCallback

C->>P: Calls ExecuteAsync
P->>B: Calls ExecuteCore
activate B
B-->>B: Determines Behavior Injection
B-->>B: Inject Behavior
deactivate B
B->>+D: Invokes
D->>-B: Returns result
B->>P: Returns result
P->>C: Returns result
```

## Anti-patterns

❌ Do not use behavior strategies to inject delays, use the latency monkey instead as the [`LatencyChaosStrategy`](latency.md) already correctly handles synchronous/asynchronous delay executions, cancellations, etc.
140 changes: 140 additions & 0 deletions docs/chaos/fault.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
# Fault monkey strategy

## About

- **Options**: [`FaultStrategyOptions`](xref:Polly.Simmy.Fault.FaultStrategyOptions)
- **Extensions**: `AddChaosFault`
- **Strategy Type**: Proactive

---

The fault chaos strategy is designed to introduce faults (exceptions) into the system, simulating real-world scenarios where operations might fail unexpectedly. It is configurable to inject specific types of exceptions or use custom logic to generate faults dynamically.

## Usage

<!-- snippet: chaos-fault-usage -->
```cs
// Fault using the default options.
// See https://www.pollydocs.org/chaos/fault#defaults for defaults.
var optionsDefault = new FaultStrategyOptions();

// 60% of invocations will be randomly affected.
var basicOptions = new FaultStrategyOptions
{
Fault = new InvalidOperationException("Dummy exception"),
Enabled = true,
InjectionRate = 0.6
};

// To use a custom function to generate the fault to inject.
var optionsWithFaultGenerator = new FaultStrategyOptions
{
FaultGenerator = static args =>
{
Exception? exception = args.Context.OperationKey switch
{
"DataLayer" => new TimeoutException(),
"ApplicationLayer" => new InvalidOperationException(),
_ => null // When the fault generator returns null the strategy won't inject any fault and it will just invoke the user's callback
};

return new ValueTask<Exception?>(exception);
},
Enabled = true,
InjectionRate = 0.6
};

// To get notifications when a fault is injected
var optionsOnFaultInjected = new FaultStrategyOptions
{
Fault = new InvalidOperationException("Dummy exception"),
Enabled = true,
InjectionRate = 0.6,
OnFaultInjected = static args =>
{
Console.WriteLine("OnFaultInjected, Exception: {0}, Operation: {1}.", args.Fault.Message, args.Context.OperationKey);
return default;
}
};

// Add a fault strategy with a FaultStrategyOptions{<TResult>} instance to the pipeline
new ResiliencePipelineBuilder().AddChaosFault(optionsDefault);
new ResiliencePipelineBuilder<HttpStatusCode>().AddChaosFault(optionsWithFaultGenerator);

// There are also a couple of handy overloads to inject the chaos easily.
new ResiliencePipelineBuilder().AddChaosFault(0.6, new InvalidOperationException("Dummy exception"));
```
<!-- endSnippet -->

Example execution:

<!-- snippet: chaos-fault-execution -->
```cs
var pipeline = new ResiliencePipelineBuilder()
.AddChaosFault(new FaultStrategyOptions // Monkey strategies are usually placed innermost in the pipelines
{
Fault = new InvalidOperationException("Dummy exception"),
Enabled = true,
InjectionRate = 0.6
})
.AddRetry(new RetryStrategyOptions
{
ShouldHandle = new PredicateBuilder().Handle<InvalidOperationException>(),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true, // Adds a random factor to the delay
MaxRetryAttempts = 4,
Delay = TimeSpan.FromSeconds(3),
})
.Build();
```
<!-- endSnippet -->

## Defaults

| Property | Default Value | Description |
|-------------------|---------------|------------------------------------------------------|
| `OnFaultInjected` | `null` | Action executed when the fault is injected. |
| `FaultGenerator` | `null` | Generates the fault to inject for a given execution. |
| `Fault` | `null` | The fault to inject. |

## Diagrams

### Happy path sequence diagram
Copy link
Contributor Author

@peter-csala peter-csala Jan 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might make sense to explain why fault injection did not happen.

Or rename the Happy path to No fault injected / No injection


```mermaid
sequenceDiagram
actor C as Caller
participant P as Pipeline
participant F as Fault
participant D as DecoratedUserCallback

C->>P: Calls ExecuteAsync
P->>F: Calls ExecuteCore
activate F
F-->>F: Determines Fault Injection
deactivate F
F->>+D: Invokes
D->>-F: Returns result
F->>P: Returns result
P->>C: Returns result
```

### Unhappy path sequence diagram

```mermaid
sequenceDiagram
actor C as Caller
participant P as Pipeline
participant F as Fault
participant D as DecoratedUserCallback

C->>P: Calls ExecuteAsync
P->>F: Calls ExecuteCore
activate F
F-->>F: Determines Fault Injection
F-->>F: Inject Fault
deactivate F
Note over D: The user's Callback is not invoked when a fault is injected
F->>P: Throws injected Fault
P->>C: Propagates Exception
```
53 changes: 53 additions & 0 deletions docs/chaos/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Chaos engineering with Simmy

[Simmy][simmy] is a major new companion project adding a chaos-engineering and fault-injection dimension to Polly, through the provision of policies to selectively inject faults, latency, custom behavior or fake results.

<img src="../../logos/Simmy-Logo.png" alt="Simmy"/>

## Motivation

There are a lot of questions when it comes to chaos-engineering and making sure that a system is actually ready to face the worst possible scenarios:

* Is my system resilient enough?
* Am I handling the right exceptions/scenarios?
* How will my system behave if X happens?
* How can I test without waiting for a handled (or even unhandled) exception to happen in my production environment?

Using Polly helps introduce resilience to a project, but we don't want to have to wait for expected or unexpected failures to test it out. A resilience could be wrongly implemented; testing the scenarios is not straightforward; and mocking failure of some dependencies (for example a cloud SaaS or PaaS service) is not always straightforward.

**What is needed, to simulate chaotic scenarios in a production environment?**

* A way to simulate failures of dependencies (any service dependency for example).
* Define when to fail based on some external factors - maybe global configuration or some rule.
* A way to revert easily, to control the blast radius.
* To be production grade, to run this in a production or near-production system with automation.

## Chaos strategies (a.k.a Monkey strategies)

Chaos strategies (or Monkey strategies as we call them) are in essence a [Resilience strategy](../strategies/index.md#built-in-strategies), which means, a *Resilience Strategy* is the minimum unit of resilience for Polly, a *Monkey Strategy* is the minimum unit of chaos for Simmy.

### Built-in strategies

| Strategy | Reactive | What does the policy do? |
|-------------------------|----------|----------------------------------------------------------------------|
| [Fault](fault.md) | No | Injects exceptions in your system. |
| [Result](result.md) | Yes | Substitute results to fake outcomes in your system. |
| [Latency](latency.md) | No | Injects latency into executions before the calls are made. |
| [Behavior](behavior.md) | No | Allows you to inject *any* extra behaviour, before a call is placed. |

## Usage

It is usual to place the monkey strategy innermost in a Resilience Pipeline. By placing the monkey strategies innermost, they subvert the usual outbound call at the last minute, substituting their fault or adding extra latency, etc. The existing resilience strategies - further out in the `ResiliencePipeline` - still apply, so you can test how the Polly resilience strategies you have configured handle the chaos/faults injected by Simmy.

## Common options across strategies

All the strategies' options implement the [`MonkeyStrategyOptions`](xref:Polly.Simmy.MonkeyStrategyOptions) class as it contains the basic configuration for every monkey strategy.

| Property | Default Value | Description |
|--------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `InjectionRate` | 0.001 ms | A decimal between 0 and 1 inclusive. The strategy will inject the chaos, randomly, that proportion of the time, e.g.: if 0.2, twenty percent of calls will be randomly affected; if 0.01, one percent of calls; if 1, all calls. |
| `InjectionRateGenerator` | `null` | Generates the injection rate for a given execution, which the value should be between [0, 1] (inclusive). |
| `Enabled` | `false` | Determines whether the strategy is enabled or not. |
| `EnabledGenerator` | `null` | the enable generator that indicates whether or not the chaos strategy is enabled for a given execution. |

[simmy]: https://github.com/Polly-Contrib/Simmy
Loading