Skip to content

Add stress secret rotation instructions #7934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 21, 2024
Merged

Add stress secret rotation instructions #7934

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions tools/stress-cluster/cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Table of Contents
* [Prod Cluster](#prod-cluster)
* [Local Cluster](#local-cluster)
* [Deploying Stress Test Addons](#deploying-stress-test-addons)
* [Rotating Cluster Secrets](#rotating-cluster-secrets)
* [Development](#development)
* [Bicep templates](#bicep-templates)
* [Helm templates](#helm-templates)
Expand Down Expand Up @@ -125,6 +126,45 @@ Steps for deploying the stress test addons helm chart:
1. Run `kubectl get pods -n examples -w` to monitor the status of each pod and look for Running/Completed and make sure there are no errors.
1. Update all the stress tests' Chart.yaml files across the other repos in the same manner.

# Rotating Cluster Secrets

Each stress cluster provisions one app/service principal with permissions to deploy resources to a subscription. This is used for stress tests that define bicep templates for live resources.

The secret is initialized in the `rg-stress-secrets-<env>` resource group in the subscription. There will be a keyvault named `stress-secrets-<env>` and will have one secret named `public`. This secret takes the format of a .env file like:

```
AZURE_CLIENT_SECRET=<secret>
AZURE_TENANT_ID=<tenant id>
AZURE_CLIENT_ID=<client id>
AZURE_SUBSCRIPTION_ID=<sub id>
AZURE_CLIENT_OID=<oid>
STRESS_CLUSTER_RESOURCE_GROUP=<rg>
```

During cluster buildout (`provision.ps1`), this is all initialized automatically, however sometimes this secret needs to be rotated on-demand (for expiration or security reasons).

To rotate the secret, find the underlying app registration for the cluster. This will match the `AZURE_CLIENT_ID` of the secret, or you can search in Azure Portal for `stress-provisioner-<env>`. Navigate to the application/app registration page, and click `Certificates & secrets` on the left side. Click `New client secret`, set expiration to 12 months and name/describe it `rbac`. When the secret is created, you will be able to copy the value.

Next, run the following to get the existing .env file secret for the stress cluster:

```
az keyvault secret show --vault-name stress-secrets-<env> -n public -o tsv --query value > stress-secret
```

Update the file, replacing the `AZURE_CLIENT_SECRET` value with the new secret value, then run:

```
az keyvault secret set --vault-name stress-secrets-<env> -n public -f ./stress-secret
```

To verify the rotation is complete, do a test run of the deployment example. From the root of `azure-sdk-tools`:

```
eng/common/scripts/stress-testing/deploy-stress-tests.ps1 -Environment <env> -SearchDirectory ./tools/stress-cluster/chaos/examples/stress-deployment-example
```

Then monitor the stress deployment and make sure the resources deployed successfully in the `init-azure-deployer` init container.

# Development

## Bicep templates
Expand Down