diff --git a/samples/README.md b/samples/README.md index edf6dae..2b9dbed 100644 --- a/samples/README.md +++ b/samples/README.md @@ -3,4 +3,9 @@ Sample applications and infrastructure you can deploy and then break with Azure Chaos Studio — a safe place to practice resilience testing end to end. -_(Scaffold — first samples land here soon.)_ +| Sample | What it shows | Time | +|---|---|---| +| [`aks-zone-down-demo/`](aks-zone-down-demo/) | Deploy a sample retail app to a zone-redundant AKS cluster, take down an availability zone and watch the app break, then fix the deployment and rerun to prove it survives. | ~45 min | + +Contributions welcome — one folder per sample, with a README covering what it +demonstrates, prerequisites, setup, and cleanup. diff --git a/samples/aks-zone-down-demo/AGENTS.md b/samples/aks-zone-down-demo/AGENTS.md new file mode 100644 index 0000000..cffb9a4 --- /dev/null +++ b/samples/aks-zone-down-demo/AGENTS.md @@ -0,0 +1,55 @@ +# Agent guide: AKS zone-down demo + +Context and ground rules for AI agents (GitHub Copilot, Claude, or similar) +running this demo on a user's behalf. + +## What this demo is + +A break-fix-prove resilience demo for Azure Chaos Studio Workspaces (public +preview). The arc matters: + +1. Deploy the AKS store demo app with its **default single-replica** + deployments to a zone-redundant AKS cluster. +2. Run the **Compute Zone Down** scenario against the zone the `store-front` + pod runs in. **The storefront going down is the expected, desired outcome + of run 1** — it demonstrates the resilience gap. Do not treat the outage + as an error to fix mid-run, and do not scale the deployment before run 1; + that destroys the lesson. +3. Apply the fix (replicas + zone topology spread constraint, in README.md + step 5), rerun the same scenario, and show the storefront surviving. + +## Ground rules + +- Run only in a subscription and resource group set aside for demos. Never + target production resources. +- `deploy.sh` tags its resource group `chaos-demo=aks-zone-down-demo`, and + `cleanup.sh` refuses to delete a group without that tag. Don't bypass the + guard. +- Confirm with the user before starting a scenario run and before any + deletion. +- Chaos Studio Workspaces are in public preview; regions and behavior can + change. If something doesn't match these instructions, prefer the live + docs (links at the end) over improvising. + +## Steps + +1. `./deploy.sh` — env overrides: `RESOURCE_GROUP`, `LOCATION`, + `CLUSTER_NAME`, `MANIFEST_URL`. It prints the storefront URL, the + cluster's infrastructure resource group (`MC_*`), and the zone the + `store-front` pod landed in. Verify the storefront loads before + proceeding. +2. Create the workspace and run the scenario per README.md ("Run the demo"). + You can do this in the Azure portal with the user, or with the `az chaos` + CLI extension and this repo's Copilot CLI plugin. Permissions: the Reader + banner on the workspace, and the **Fix Permissions** action on the + scenario configuration page when validation reports missing RBAC. +3. Between runs, apply the fix exactly as written in README.md step 5, and + confirm the replicas spread across nodes before rerunning. +4. `./cleanup.sh` when finished (`FORCE=1` skips the confirmation prompt). + +## References + +- Full walkthrough: +- Manage workspaces and scenarios with the CLI: +- Least-privilege roles: +- Copilot CLI plugin: [`../../copilot-cli-plugin/`](../../copilot-cli-plugin/) diff --git a/samples/aks-zone-down-demo/README.md b/samples/aks-zone-down-demo/README.md new file mode 100644 index 0000000..61e8891 --- /dev/null +++ b/samples/aks-zone-down-demo/README.md @@ -0,0 +1,142 @@ +# AKS zone-down demo + +A ~45-minute end-to-end demo of Azure Chaos Studio Workspaces with a +break-fix-prove arc: deploy a sample retail app to a zone-redundant AKS +cluster, take down an availability zone and watch the storefront **actually go +down**, then fix the deployment, rerun the same scenario, and watch it survive. + +A demo where something visibly breaks teaches more than one where nothing +happens — and the sample app's default single-replica deployment provides the +breakage for free. Good as a first hands-on experience, a customer demo, or a +workshop exercise. It reuses the +[AKS store demo](https://github.com/Azure-Samples/aks-store-demo) sample +application (public container images, no registry or build steps). + +The full written walkthrough lives on Microsoft Learn: +[Tutorial: Deploy a sample application and test its zone resilience with Chaos Studio](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app). +This folder adds scripts that automate the setup half, so a demo starts at the +interesting part. + +## What the demo shows + +1. A zone-redundant AKS cluster running a storefront app that *isn't* zone + redundant: every component is a single replica. +2. A Chaos Studio **Workspace** that discovers the cluster's node + infrastructure. +3. **Run 1 — break it.** The **Compute Zone Down** scenario shuts down the + node in the storefront's zone. The store goes unreachable for several + minutes (Kubernetes waits ~5 minutes by default before rescheduling pods + off an unreachable node). That downtime is the finding. +4. **The fix.** Scale the front end to one replica per zone — one command. +5. **Run 2 — prove it.** Same scenario, same zone. A node still dies, but the + store keeps serving from the surviving zones. +6. Two **Scenario reports** that both say `Succeeded` — the teaching moment + that a run succeeding measures the disruption delivered, not app health. + The before/after difference lives in your monitoring. + +## Prerequisites + +- An Azure subscription with permission to create resource groups, AKS + clusters, and Chaos Studio workspaces. +- Azure CLI and `kubectl` — [Azure Cloud Shell](https://learn.microsoft.com/azure/cloud-shell/overview) + has both preinstalled. +- The `Microsoft.Chaos` resource provider + [registered](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-quickstart-azure-portal#register-the-chaos-studio-resource-provider) + in the subscription. + +## Set up (before the demo) + +Cluster creation takes a few minutes, so run this ahead of time: + +```bash +./deploy.sh +``` + +The script creates a resource group (`chaos-demo-rg` in `eastus2` by default — +override with the `RESOURCE_GROUP`, `LOCATION`, and `CLUSTER_NAME` environment +variables), creates a 3-node AKS cluster spread across zones 1–3, deploys the +store app with its default single-replica deployments, and prints: + +- the storefront URL — open it and confirm the store loads +- the cluster's **infrastructure resource group** (`MC_...`) — the workspace + scope for the next step +- the **zone the storefront pod is running in** — the zone to target + +## Run the demo + +Follow the walkthrough from the +[workspace step onward](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app#create-a-workspace-scoped-to-the-infrastructure-resource-group). +In short: + +1. In the portal, create a **Workspace** scoped to the infrastructure resource + group printed by `deploy.sh`, with a system-assigned identity. Discovery + finds the node VM scale set. If the workspace shows a banner that the + identity is missing read permissions on the scope, select **Assign the + Reader role over the Workspace Scope**. (Creating role assignments needs + Owner or User Access Administrator on that resource group.) +2. Open the **Compute Zone Down** scenario, target the zone `deploy.sh` + printed (the number after the region name), and save the configuration. + If validation flags missing permissions, select **Fix Permissions** to + grant the identity the recommended built-in roles. Strict least-privilege + shop? Build a [custom role from the validation output](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-workspaces-least-privilege-roles) + instead. +3. Set up the view before injecting anything — split the screen: the + storefront in a browser, the cluster's **Monitoring → Metrics** blade + charting CPU per node, and a terminal running: + + ```bash + kubectl get pods -o wide -w + ``` + +4. **Run 1:** select **Run** (5–10 minutes). The node goes `NotReady`, its + metrics flatline, and the storefront stops loading. Let the downtime sink + in — note how long it lasts. +5. **The fix:** one replica per zone, and a spread constraint so the + scheduler guarantees the "per zone" part (replicas alone can co-locate): + + ```bash + kubectl patch deployment store-front --patch '{"spec":{"replicas":3,"template":{"spec":{"topologySpreadConstraints":[{"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app":"store-front"}}}]}}}}' + kubectl get pods -l app=store-front -o wide # confirm one per node + ``` + +6. **Run 2:** rerun the same scenario against the same zone. The storefront + keeps serving while the node dies. Refresh it liberally. +7. Open **Run history** → **Generate report** for both runs. Both say + `Succeeded`; the report proves the disruption delivered, while the app's + fate shows up in the metrics chart and the browser. That pairing is the + product pitch. + +## Prefer to drive it with Copilot? + +This repo ships a [Copilot CLI plugin and MCP server](../../copilot-cli-plugin/) +that can create workspaces, configure scenarios, run them, and analyze the +results conversationally. With the plugin set up, try a prompt like: + +> Deploy the aks-zone-down-demo sample from the chaos-studio repo, then help +> me run the Compute Zone Down scenario against the zone the storefront is +> running in. When the run finishes, summarize the scenario report. + +[`AGENTS.md`](AGENTS.md) in this folder gives coding agents the context and +ground rules they need to run the demo end to end. + +> [!NOTE] +> Chaos Studio Workspaces are in public preview. Run this demo in a +> subscription and cluster set aside for testing, not production. + +## Clean up + +```bash +./cleanup.sh +``` + +Deletes the resource group, which removes the cluster, the app, the +infrastructure resource group, and the workspace. + +## Going further + +- [Workspaces overview](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-workspaces-overview) + and [Scenarios](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-scenarios) +- [Test workload resiliency on AKS](https://learn.microsoft.com/azure/chaos-studio/chaos-studio-aks-guidance) — + the caveats and interpretation guidance behind this demo +- [`copilot-cli-plugin/`](../../copilot-cli-plugin/) — drive the same setup + conversationally from GitHub Copilot diff --git a/samples/aks-zone-down-demo/cleanup.sh b/samples/aks-zone-down-demo/cleanup.sh new file mode 100755 index 0000000..3824762 --- /dev/null +++ b/samples/aks-zone-down-demo/cleanup.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Removes everything the AKS zone-down demo created. Deleting the resource +# group also removes the cluster's MC_* infrastructure resource group and the +# workspace (if you created it in the same resource group, per the tutorial). +set -euo pipefail + +RESOURCE_GROUP="${RESOURCE_GROUP:-chaos-demo-rg}" +CLUSTER_NAME="${CLUSTER_NAME:-chaos-demo-aks}" + +# Only delete groups that deploy.sh created (identified by its tag), so a +# pre-existing group with the same name can't be wiped by accident. +TAG="$(az group show --name "$RESOURCE_GROUP" --query 'tags."chaos-demo"' -o tsv 2>/dev/null || true)" +if [ "$TAG" != "aks-zone-down-demo" ]; then + echo "Refusing to delete '$RESOURCE_GROUP': it doesn't carry the chaos-demo=aks-zone-down-demo tag," + echo "so it wasn't created by deploy.sh. If you're sure, delete it yourself:" + echo " az group delete --name $RESOURCE_GROUP" + exit 1 +fi + +if [ "${FORCE:-0}" != "1" ]; then + read -r -p "Delete resource group '$RESOURCE_GROUP' and everything in it? Type the group name to confirm: " CONFIRM + if [ "$CONFIRM" != "$RESOURCE_GROUP" ]; then + echo "Aborted." + exit 1 + fi +fi + +echo "==> Deleting resource group '$RESOURCE_GROUP'" +az group delete --name "$RESOURCE_GROUP" --yes --no-wait + +echo "==> Removing kubectl context" +kubectl config delete-context "$CLUSTER_NAME" 2>/dev/null || true +kubectl config delete-cluster "$CLUSTER_NAME" 2>/dev/null || true + +echo "Deletion is running in the background (az group delete --no-wait)." diff --git a/samples/aks-zone-down-demo/deploy.sh b/samples/aks-zone-down-demo/deploy.sh new file mode 100755 index 0000000..710228d --- /dev/null +++ b/samples/aks-zone-down-demo/deploy.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Deploys the AKS zone-down demo: a zone-redundant AKS cluster running the +# AKS store demo sample app. See README.md in this folder. +set -euo pipefail + +RESOURCE_GROUP="${RESOURCE_GROUP:-chaos-demo-rg}" +LOCATION="${LOCATION:-eastus2}" +CLUSTER_NAME="${CLUSTER_NAME:-chaos-demo-aks}" +# Pinned to a release so the demo doesn't drift with upstream main; override if needed. +MANIFEST_URL="${MANIFEST_URL:-https://raw.githubusercontent.com/Azure-Samples/aks-store-demo/2.2.0/aks-store-quickstart.yaml}" + +echo "==> Creating resource group '$RESOURCE_GROUP' in $LOCATION" +az group create --name "$RESOURCE_GROUP" --location "$LOCATION" \ + --tags chaos-demo=aks-zone-down-demo --output none + +echo "==> Creating AKS cluster '$CLUSTER_NAME' (3 nodes across zones 1-3; takes a few minutes)" +az aks create \ + --resource-group "$RESOURCE_GROUP" \ + --name "$CLUSTER_NAME" \ + --node-count 3 \ + --zones 1 2 3 \ + --generate-ssh-keys \ + --output none + +echo "==> Connecting kubectl" +az aks get-credentials --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --overwrite-existing + +echo "==> Deploying the AKS store demo app (default single-replica deployments -- that's the point)" +kubectl apply -f "$MANIFEST_URL" + +echo "==> Waiting for the storefront rollout" +kubectl rollout status deployment/store-front --timeout=300s + +echo "==> Waiting for the storefront public IP (can take a couple of minutes)" +STORE_IP="" +for _ in $(seq 1 60); do + STORE_IP="$(kubectl get service store-front -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)" + [ -n "$STORE_IP" ] && break + sleep 10 +done + +NODE_RG="$(az aks show --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --query nodeResourceGroup -o tsv)" + +STORE_NODE="$(kubectl get pods -l app=store-front -o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true)" +STORE_ZONE="" +if [ -n "$STORE_NODE" ]; then + STORE_ZONE="$(kubectl get node "$STORE_NODE" -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}' 2>/dev/null || true)" +fi + +echo +echo "Done." +echo +if [ -n "$STORE_IP" ]; then + echo " Storefront: http://$STORE_IP" +else + echo " Storefront IP still pending -- check with: kubectl get service store-front" +fi +echo " Infrastructure resource group: $NODE_RG" +if [ -n "$STORE_ZONE" ]; then + echo " store-front pod zone: $STORE_ZONE <- target this zone (the number after the region) to break the app" +else + echo " store-front pod zone: check with: kubectl get pods -l app=store-front -o wide" +fi +echo +echo "Next: create a Chaos Studio Workspace scoped to '$NODE_RG' and run the" +echo "Compute Zone Down scenario against the store-front zone:" +echo "https://learn.microsoft.com/azure/chaos-studio/chaos-studio-tutorial-sample-app#create-a-workspace-scoped-to-the-infrastructure-resource-group"