Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,9 @@
Sample applications and infrastructure you can deploy and then break with Azure
Chaos Studio — a safe place to practice resilience testing end to end.

_(Scaffold — first samples land here soon.)_
| Sample | What it shows | Time |
|---|---|---|
| [`aks-zone-down-demo/`](aks-zone-down-demo/) | Deploy a sample retail app to a zone-redundant AKS cluster, take down an availability zone and watch the app break, then fix the deployment and rerun to prove it survives. | ~45 min |

Contributions welcome — one folder per sample, with a README covering what it
demonstrates, prerequisites, setup, and cleanup.
55 changes: 55 additions & 0 deletions samples/aks-zone-down-demo/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Agent guide: AKS zone-down demo

Context and ground rules for AI agents (GitHub Copilot, Claude, or similar)
running this demo on a user's behalf.

## What this demo is

A break-fix-prove resilience demo for Azure Chaos Studio Workspaces (public
preview). The arc matters:

1. Deploy the AKS store demo app with its **default single-replica**
deployments to a zone-redundant AKS cluster.
2. Run the **Compute Zone Down** scenario against the zone the `store-front`
pod runs in. **The storefront going down is the expected, desired outcome
of run 1** — it demonstrates the resilience gap. Do not treat the outage
as an error to fix mid-run, and do not scale the deployment before run 1;
that destroys the lesson.
3. Apply the fix (replicas + zone topology spread constraint, in README.md
step 5), rerun the same scenario, and show the storefront surviving.

## Ground rules

- Run only in a subscription and resource group set aside for demos. Never
target production resources.
- `deploy.sh` tags its resource group `chaos-demo=aks-zone-down-demo`, and
`cleanup.sh` refuses to delete a group without that tag. Don't bypass the
guard.
- Confirm with the user before starting a scenario run and before any
deletion.
- Chaos Studio Workspaces are in public preview; regions and behavior can
change. If something doesn't match these instructions, prefer the live
docs (links at the end) over improvising.

## Steps

1. `./deploy.sh` — env overrides: `RESOURCE_GROUP`, `LOCATION`,
`CLUSTER_NAME`, `MANIFEST_URL`. It prints the storefront URL, the
cluster's infrastructure resource group (`MC_*`), and the zone the
`store-front` pod landed in. Verify the storefront loads before
proceeding.
2. Create the workspace and run the scenario per README.md ("Run the demo").
You can do this in the Azure portal with the user, or with the `az chaos`
CLI extension and this repo's Copilot CLI plugin. Permissions: the Reader
banner on the workspace, and the **Fix Permissions** action on the
scenario configuration page when validation reports missing RBAC.
3. Between runs, apply the fix exactly as written in README.md step 5, and
confirm the replicas spread across nodes before rerunning.
4. `./cleanup.sh` when finished (`FORCE=1` skips the confirmation prompt).

## References

- Full walkthrough: <https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app>
- Manage workspaces and scenarios with the CLI: <https://learn.microsoft.com/azure/chaos-studio/chaos-studio-manage-cli>
- Least-privilege roles: <https://learn.microsoft.com/azure/chaos-studio/chaos-studio-workspaces-least-privilege-roles>
- Copilot CLI plugin: [`../../copilot-cli-plugin/`](../../copilot-cli-plugin/)
142 changes: 142 additions & 0 deletions samples/aks-zone-down-demo/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# AKS zone-down demo

A ~45-minute end-to-end demo of Azure Chaos Studio Workspaces with a
break-fix-prove arc: deploy a sample retail app to a zone-redundant AKS
cluster, take down an availability zone and watch the storefront **actually go
down**, then fix the deployment, rerun the same scenario, and watch it survive.

A demo where something visibly breaks teaches more than one where nothing
happens — and the sample app's default single-replica deployment provides the
breakage for free. Good as a first hands-on experience, a customer demo, or a
workshop exercise. It reuses the
[AKS store demo](https://github.com/Azure-Samples/aks-store-demo) sample
application (public container images, no registry or build steps).

The full written walkthrough lives on Microsoft Learn:
[Tutorial: Deploy a sample application and test its zone resilience with Chaos Studio](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app).
This folder adds scripts that automate the setup half, so a demo starts at the
interesting part.

## What the demo shows

1. A zone-redundant AKS cluster running a storefront app that *isn't* zone
redundant: every component is a single replica.
2. A Chaos Studio **Workspace** that discovers the cluster's node
infrastructure.
3. **Run 1 — break it.** The **Compute Zone Down** scenario shuts down the
node in the storefront's zone. The store goes unreachable for several
minutes (Kubernetes waits ~5 minutes by default before rescheduling pods
off an unreachable node). That downtime is the finding.
4. **The fix.** Scale the front end to one replica per zone — one command.
5. **Run 2 — prove it.** Same scenario, same zone. A node still dies, but the
store keeps serving from the surviving zones.
6. Two **Scenario reports** that both say `Succeeded` — the teaching moment
that a run succeeding measures the disruption delivered, not app health.
The before/after difference lives in your monitoring.

## Prerequisites

- An Azure subscription with permission to create resource groups, AKS
clusters, and Chaos Studio workspaces.
- Azure CLI and `kubectl` — [Azure Cloud Shell](https://learn.microsoft.com/azure/cloud-shell/overview)
has both preinstalled.
- The `Microsoft.Chaos` resource provider
[registered](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-quickstart-azure-portal#register-the-chaos-studio-resource-provider)
in the subscription.

## Set up (before the demo)

Cluster creation takes a few minutes, so run this ahead of time:

```bash
./deploy.sh
```

The script creates a resource group (`chaos-demo-rg` in `eastus2` by default —
override with the `RESOURCE_GROUP`, `LOCATION`, and `CLUSTER_NAME` environment
variables), creates a 3-node AKS cluster spread across zones 1–3, deploys the
store app with its default single-replica deployments, and prints:

- the storefront URL — open it and confirm the store loads
- the cluster's **infrastructure resource group** (`MC_...`) — the workspace
scope for the next step
- the **zone the storefront pod is running in** — the zone to target

## Run the demo

Follow the walkthrough from the
[workspace step onward](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app#create-a-workspace-scoped-to-the-infrastructure-resource-group).
In short:

1. In the portal, create a **Workspace** scoped to the infrastructure resource
group printed by `deploy.sh`, with a system-assigned identity. Discovery
finds the node VM scale set. If the workspace shows a banner that the
identity is missing read permissions on the scope, select **Assign the
Reader role over the Workspace Scope**. (Creating role assignments needs
Owner or User Access Administrator on that resource group.)
2. Open the **Compute Zone Down** scenario, target the zone `deploy.sh`
printed (the number after the region name), and save the configuration.
If validation flags missing permissions, select **Fix Permissions** to
grant the identity the recommended built-in roles. Strict least-privilege
shop? Build a [custom role from the validation output](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-workspaces-least-privilege-roles)
instead.
3. Set up the view before injecting anything — split the screen: the
storefront in a browser, the cluster's **Monitoring → Metrics** blade
charting CPU per node, and a terminal running:

```bash
kubectl get pods -o wide -w
```

4. **Run 1:** select **Run** (5–10 minutes). The node goes `NotReady`, its
metrics flatline, and the storefront stops loading. Let the downtime sink
in — note how long it lasts.
5. **The fix:** one replica per zone, and a spread constraint so the
scheduler guarantees the "per zone" part (replicas alone can co-locate):

```bash
kubectl patch deployment store-front --patch '{"spec":{"replicas":3,"template":{"spec":{"topologySpreadConstraints":[{"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app":"store-front"}}}]}}}}'
kubectl get pods -l app=store-front -o wide # confirm one per node
```

6. **Run 2:** rerun the same scenario against the same zone. The storefront
keeps serving while the node dies. Refresh it liberally.
7. Open **Run history** → **Generate report** for both runs. Both say
`Succeeded`; the report proves the disruption delivered, while the app's
fate shows up in the metrics chart and the browser. That pairing is the
product pitch.

## Prefer to drive it with Copilot?

This repo ships a [Copilot CLI plugin and MCP server](../../copilot-cli-plugin/)
that can create workspaces, configure scenarios, run them, and analyze the
results conversationally. With the plugin set up, try a prompt like:

> Deploy the aks-zone-down-demo sample from the chaos-studio repo, then help
> me run the Compute Zone Down scenario against the zone the storefront is
> running in. When the run finishes, summarize the scenario report.

[`AGENTS.md`](AGENTS.md) in this folder gives coding agents the context and
ground rules they need to run the demo end to end.

> [!NOTE]
> Chaos Studio Workspaces are in public preview. Run this demo in a
> subscription and cluster set aside for testing, not production.

## Clean up

```bash
./cleanup.sh
```

Deletes the resource group, which removes the cluster, the app, the
infrastructure resource group, and the workspace.

## Going further

- [Workspaces overview](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-workspaces-overview)
and [Scenarios](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-scenarios)
- [Test workload resiliency on AKS](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-aks-guidance) —
the caveats and interpretation guidance behind this demo
- [`copilot-cli-plugin/`](../../copilot-cli-plugin/) — drive the same setup
conversationally from GitHub Copilot
35 changes: 35 additions & 0 deletions samples/aks-zone-down-demo/cleanup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Removes everything the AKS zone-down demo created. Deleting the resource
# group also removes the cluster's MC_* infrastructure resource group and the
# workspace (if you created it in the same resource group, per the tutorial).
set -euo pipefail

RESOURCE_GROUP="${RESOURCE_GROUP:-chaos-demo-rg}"
CLUSTER_NAME="${CLUSTER_NAME:-chaos-demo-aks}"

# Only delete groups that deploy.sh created (identified by its tag), so a
# pre-existing group with the same name can't be wiped by accident.
TAG="$(az group show --name "$RESOURCE_GROUP" --query 'tags."chaos-demo"' -o tsv 2>/dev/null || true)"
if [ "$TAG" != "aks-zone-down-demo" ]; then
echo "Refusing to delete '$RESOURCE_GROUP': it doesn't carry the chaos-demo=aks-zone-down-demo tag,"
echo "so it wasn't created by deploy.sh. If you're sure, delete it yourself:"
echo " az group delete --name $RESOURCE_GROUP"
exit 1
fi

if [ "${FORCE:-0}" != "1" ]; then
read -r -p "Delete resource group '$RESOURCE_GROUP' and everything in it? Type the group name to confirm: " CONFIRM
if [ "$CONFIRM" != "$RESOURCE_GROUP" ]; then
echo "Aborted."
exit 1
fi
fi

echo "==> Deleting resource group '$RESOURCE_GROUP'"
az group delete --name "$RESOURCE_GROUP" --yes --no-wait

echo "==> Removing kubectl context"
kubectl config delete-context "$CLUSTER_NAME" 2>/dev/null || true
kubectl config delete-cluster "$CLUSTER_NAME" 2>/dev/null || true

echo "Deletion is running in the background (az group delete --no-wait)."
67 changes: 67 additions & 0 deletions samples/aks-zone-down-demo/deploy.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Deploys the AKS zone-down demo: a zone-redundant AKS cluster running the
# AKS store demo sample app. See README.md in this folder.
set -euo pipefail

RESOURCE_GROUP="${RESOURCE_GROUP:-chaos-demo-rg}"
LOCATION="${LOCATION:-eastus2}"
CLUSTER_NAME="${CLUSTER_NAME:-chaos-demo-aks}"
# Pinned to a release so the demo doesn't drift with upstream main; override if needed.
MANIFEST_URL="${MANIFEST_URL:-https://raw.githubusercontent.com/Azure-Samples/aks-store-demo/2.2.0/aks-store-quickstart.yaml}"

echo "==> Creating resource group '$RESOURCE_GROUP' in $LOCATION"
az group create --name "$RESOURCE_GROUP" --location "$LOCATION" \
--tags chaos-demo=aks-zone-down-demo --output none

echo "==> Creating AKS cluster '$CLUSTER_NAME' (3 nodes across zones 1-3; takes a few minutes)"
az aks create \
--resource-group "$RESOURCE_GROUP" \
--name "$CLUSTER_NAME" \
--node-count 3 \
--zones 1 2 3 \
--generate-ssh-keys \
--output none

echo "==> Connecting kubectl"
az aks get-credentials --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --overwrite-existing

echo "==> Deploying the AKS store demo app (default single-replica deployments -- that's the point)"
kubectl apply -f "$MANIFEST_URL"

echo "==> Waiting for the storefront rollout"
kubectl rollout status deployment/store-front --timeout=300s

echo "==> Waiting for the storefront public IP (can take a couple of minutes)"
STORE_IP=""
for _ in $(seq 1 60); do
STORE_IP="$(kubectl get service store-front -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)"
[ -n "$STORE_IP" ] && break
sleep 10
done

NODE_RG="$(az aks show --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --query nodeResourceGroup -o tsv)"

STORE_NODE="$(kubectl get pods -l app=store-front -o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true)"
STORE_ZONE=""
if [ -n "$STORE_NODE" ]; then
STORE_ZONE="$(kubectl get node "$STORE_NODE" -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}' 2>/dev/null || true)"
fi

echo
echo "Done."
echo
if [ -n "$STORE_IP" ]; then
echo " Storefront: http://$STORE_IP"
else
echo " Storefront IP still pending -- check with: kubectl get service store-front"
fi
echo " Infrastructure resource group: $NODE_RG"
if [ -n "$STORE_ZONE" ]; then
echo " store-front pod zone: $STORE_ZONE <- target this zone (the number after the region) to break the app"
else
echo " store-front pod zone: check with: kubectl get pods -l app=store-front -o wide"
fi
echo
echo "Next: create a Chaos Studio Workspace scoped to '$NODE_RG' and run the"
echo "Compute Zone Down scenario against the store-front zone:"
echo "https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app#create-a-workspace-scoped-to-the-infrastructure-resource-group"
Loading