diff --git a/tools/stress-cluster/cluster/README.md b/tools/stress-cluster/cluster/README.md index 6d5112367bd..110e0a36bd3 100644 --- a/tools/stress-cluster/cluster/README.md +++ b/tools/stress-cluster/cluster/README.md @@ -7,6 +7,7 @@ Table of Contents * [Prod Cluster](#prod-cluster) * [Local Cluster](#local-cluster) * [Deploying Stress Test Addons](#deploying-stress-test-addons) +* [Rotating Cluster Secrets](#rotating-cluster-secrets) * [Development](#development) * [Bicep templates](#bicep-templates) * [Helm templates](#helm-templates) @@ -125,6 +126,45 @@ Steps for deploying the stress test addons helm chart: 1. Run `kubectl get pods -n examples -w` to monitor the status of each pod and look for Running/Completed and make sure there are no errors. 1. Update all the stress tests' Chart.yaml files across the other repos in the same manner. +# Rotating Cluster Secrets + +Each stress cluster provisions one app/service principal with permissions to deploy resources to a subscription. This is used for stress tests that define bicep templates for live resources. + +The secret is initialized in the `rg-stress-secrets-` resource group in the subscription. There will be a keyvault named `stress-secrets-` and will have one secret named `public`. This secret takes the format of a .env file like: + +``` +AZURE_CLIENT_SECRET= +AZURE_TENANT_ID= +AZURE_CLIENT_ID= +AZURE_SUBSCRIPTION_ID= +AZURE_CLIENT_OID= +STRESS_CLUSTER_RESOURCE_GROUP= +``` + +During cluster buildout (`provision.ps1`), this is all initialized automatically, however sometimes this secret needs to be rotated on-demand (for expiration or security reasons). + +To rotate the secret, find the underlying app registration for the cluster. This will match the `AZURE_CLIENT_ID` of the secret, or you can search in Azure Portal for `stress-provisioner-`. Navigate to the application/app registration page, and click `Certificates & secrets` on the left side. Click `New client secret`, set expiration to 12 months and name/describe it `rbac`. When the secret is created, you will be able to copy the value. + +Next, run the following to get the existing .env file secret for the stress cluster: + +``` +az keyvault secret show --vault-name stress-secrets- -n public -o tsv --query value > stress-secret +``` + +Update the file, replacing the `AZURE_CLIENT_SECRET` value with the new secret value, then run: + +``` +az keyvault secret set --vault-name stress-secrets- -n public -f ./stress-secret +``` + +To verify the rotation is complete, do a test run of the deployment example. From the root of `azure-sdk-tools`: + +``` +eng/common/scripts/stress-testing/deploy-stress-tests.ps1 -Environment -SearchDirectory ./tools/stress-cluster/chaos/examples/stress-deployment-example +``` + +Then monitor the stress deployment and make sure the resources deployed successfully in the `init-azure-deployer` init container. + # Development ## Bicep templates